Children are not stupid. Lovebombing a child who you don't particularly like will come off as insincere and he will pick up on it.
At the end of the day OP, he's 4. He's still so tiny and he still has so much to learn. I've just come out the other side of this with DD and it was a learning curve for both of us.
But you're the adult. You're the one who he is looking to, to model the right behaviour, to love him, to help him, to understand him.
He needs you to teach him, not write him off as a not very nice kid.
As a PP has said, he doesn't misbehave when he is with the childminder. So it isn't intrinsic behaviour. It's something to do with the surroundings/conditions that he's in when he starts playing up.
It's up to you and DH to pick those apart and work out why he's acting like that and try to prevent/minimise it in future.
I don't mean having the world revolve around him either, but even giving him small (closed) choices so he feels slightly in control of his day.
Things like; 5 minutes of playing and then we're going out, Do you want to wear the red or blue top today? Shall we go shopping first or shall we see the ducks first?
Things that don't inconvenience you, but that help him feel like he is in control of himself.
Imagine if you had a day of lounging around in pjs, watching boxsets planned and someone came in and told you that you needed to get dressed to go out right now. That someone had also chosen your clothes, where you were going, what time you were going and how long you were going for.
The person then tells you that there's no exceptions. It would frustrate you to no end, but that's everyday for a pre-schooler - and they're expected to tolerate it, no buts.
This goes for everything in his life.
He didn't ask to be born, you made that choice. Only to now be saying that your life was better without him. That's not his fault.
I do think some sort of counselling, talking or play therapy may be of use to you - maybe even a parenting course, but harbouring resentment for a child not even old enough to start school yet is not on.