Oh goodness, @totallydonetoddlermum
There is a lot to unscramble here.
Firstly, your son loves you. You are his anchor, and his everything. He isn't old enough to know any different. You're sick of people saying he is all cuddly with his dad because he sees him relatively little - but it's true.
My very long experience tells me that we all have phases with children which we find easier, and those which are complete hell. I loved mine as babies, obviously - but I found them a bit boring (so shoot me now). I absolutely loved every second of them being toddlers. The bit from 5-11 was fine, but unmemorable. The teenage years, with one in particular, have been an unmitigated nightmare. I have gone to bed many, many times crying because everything she does suggests that she loathes me. It is like wading through mud and treacle combined. I hate every minute of it. Yet other people love the teenage years (and, it's true, I only have one who has made me question why I bother remaining alive - but even then, I can't say that teenagers have given me joy in the way that toddlers did).
I think parenting is a very, very long haul. If you can get through this bit, it will seem like a bad dream once you have a school-aged child. Your DS will probably be a dream teenager whom you thoroughly enjoy.
I know this sounds trite, but it isn't meant to. It's just that parenting can be really, really hard.