That’s so sad OP 
Me and my husband have a 6 year old and a three year old and we offer our children very different things in terms of interacting with them.
I tend to do the indoor things like play board games, painting, play-doh, playing with their cars and dinosaurs with them, playing card games with them, I do the homework, the bedtime stories and tuck them into bed etc - all the interactive stuff.
My husband does all the outdoorsy stuff with them...he takes them to sports lessons, he takes them to watch football/rugby games, takes them to Theme Parks, he takes them to park, walks through the forest etc - he even takes them abroad three times a year on his own - something I would NEVER do, the thought fills me with complete dread 
Despite my husband having a very active role on their life, they still have an obvious preference for me - I generally just think young children do feel a different kind of closeness to their mums.
However - your poor boy, the fact he said that about his dad is truly heartbreaking. Does his dad know how his son feels?
We all need time for ourselves but when you have children we have to accept that their needs come first and children so much need to feel they are important to their parents and feel like they are loved.
I’m sorry you’re dealing with this OP, it must be quite upsetting to deal with 