You paid for 4 seats, and, I presume, had 4 allocated to you? So that seat is yours.
The different question that you have not asked here, and the crux of the matter, is WYBU to not give up this seat so that the couple with an infant could use it, despite the fact that you have paid for it.
For me it depends on how they asked, how long the flight was ect.
The couple with the child made the choice not to buy a ticket for their baby, and knew they would have to have them on their lap. They saw an opportunity in the empty seat and want to use it.
I wouldn't say you would have been massively unreasonable not to give it up - depending on how much it would inconvenience you, and any other reasons you might have had.
However, I'd like to think giving it up would have been the kind and charitable thing to do, and something that I hope I would consider. We're all people who struggle with things at times, and it's nice to think we would help each other out.
However - if the couple were rude about it any way, or acting entitled about it, then this makes me see red, so I would say definitely YANBU for not giving up the seat.