Ive had people favourited on my phone that I had no intention on favouriting. It was essentially a "butt favourite". Didnt even realise until much later, and undid it as soon as I realised.
I consider myself fairly techy, and its the kind of thing that has happened to me, so maybe it isnt that suspicious.
I would go with the tech glitch scenario. Because who, in their right mind, goes to a different phone, hours later, to answer a text they dont have in front of them?
I find it much more plausible that he replied straight away, the number hadnt changed yet, but was in the process of, it got stuck in the ether of text messages, then the number ported, he replied to the others, and, eventually, later, the first message was processed and sent out.
I also dont understand why you wouldnt just ASK him why you got a message from a different number. Who cares if he knows youre "on to him"? Surely thats the point, if anything nefarious is going on???? You want him to know you know something is up! And confront him!