Just to reitterate what the others have said really. When my DD1 was 4 / 5 I was almost despairing of our relationship. I had also gone back to work when she was 6 months and not had an easy time in her babyhood (I had a much easier time with her siblings). Anyway, by this stage her behaviour was appalling and she seemed to hate me. Sometimes it seemed that Dh and I did nothing but shout at her and I had really begun to 'dislike' her almost. Of course this makes me feel like weeping in retrospect. At the time I also had a demanding toddler and a new baby and she had just started at Reception. It's not surprising that she was crying out for attention and I was too exhausted and snappy to give it.
Anyway, to cut a long story short on the advice of many, I started to try and take her out to tea or something on her own, for about half and hour / 45 mins just once every 10 days / 2 weeks or so which was about all I could manage at the time and initially although it didn't exactly help her general behaviour, during those few snatched moments of just us she was a different (perfect) little girl and I remembered what her 'real' character was. Over time I managed to build on that and change my behaviour and attitude to her (i.e. when her behaviour was difficult I tried not to react in the obvious way but instead to break the cycle, give her a hug / tickle her, make a joke etc). I also tried to make a conscious effort regularly to give her a hug or a positive comment for no reason - just because - rather than when something specific had happened. It was all v hard but a combination of finding small slots of time where we could be on our own and consciously analysing and breaking the pattern of our conflicting behaviour worked wonders over time.
She is now 7 and a v high maintenance drama queen (there may at times still be much shouting and argument) but our relationship is rock solid. So it is not too late to repair things between you, but make and effort as it won't necessarily happen without some input on your part.
while I think it is a good idea to be pro-active about spending time with him, it is worth pointing out that at around 4 years old, dcs go through a phase where they identify strongly with the parent of same gender, so some daddy bonding is normal now too.
I'd agree with heather1. My DD is a bit like this with her Daddy - she is terribly Mummy centred. The best thing I can do for them is to make sure they spend time together without me. Find an activity that you both enjoy, but DS especially likes and spend some time exclusively together and let your son know you love him. Your DP does need to support you though and maybe explain to your son that he needs to be kind to Mummy.
I was worried about my relationship with DS1 he is 8. So now we have an afternoon a week which is "our afternoon" we are in a country where he does not have school one afternoon a week so this is our aftenoon and after lunch, and homework we usually have 1 hour where we do something together e.g. a puzzle, go for a walk, do something he is interested in. Depending on your schedual could you give him some time each week or every couple of days when it is just the two of you and emphasise how much you are looking forward to it, ask what shall we do together, its going to be so much fun? Me and DS1 still clash, espically over homework, but our relationship has really improved since we started in August. Also 4 can be a tricky age they want to be in control a lot and very contrary. There can be no doubt that he loves you as you are his Mum. But maybe it could be worth trying to make this time together away from the house in a neutral location without your DP so that he cant go to Daddy for help e.g. swimming. Hope this helps