He's too young for you, emotionally.
-He's 24 and lives at home with his dad. (Why doesn't he get his own place?)
-He has a job ( you say a 'business' but I'd guess that means he is self-employed in a trade maybe?)
-He has a 2 year old, but only sees her at weekends (when you step in and help amuse her.) So he was a father at 22 and has split with the mother of his DD.
This may be harsh but why would a red blooded male prefer to sleep at his dad's house when you are offering him a bed and sex every night?
And a meal on the table after work?
And helping with his childcare at weekends?
You- it's not reasonable to say you treat his child as your own daughter. This is a new relationship. She is a young child and you are not her substitute mum. She already has a mum.
It doesn't look as if you are a 'family' as it is far too soon for that. This man is still living at home, but coming to your house for sex, food and an alternative to his Dad's house.
I don't think his being with his friends on a Friday night is the issue.
The real issue is he's not emotionally in a relationship.
Not to the extent you think he is.
And if his ex tried to clip his wings and control him, I'm inclined to think he's in the right here, because FGS he was only 22-23 then!
Many lads are coming out of uni at that age and nowhere near ready to settle down to a cosy night in and warm slippers.
Sorry but you need to extract yourself. It's not so much the age gap as the life experience gap and different expectations.
This is not going to work out long term.
Sorry 💐