Let me start by saying that I really am not having a go at you, OP, because I know it might come across that way on the screen. But I need to draw your attention to some of the things that you have said, and ask you to think about them.
"I've never kicked up a fuss about because I just wanted an easy life but if I'm honest, it's like having three kids to look after."
People who won't kick up a fuss at shitty behaviour NEVER have an easy life. NEVER. If there are no consequences to shitty behaviour, there is no reason to change, so by not kicking up a fuss you are effectively inviting more shitty behaviour. If you truly want a quiet life, maybe consider treating him like that third child. With a child, bad behaviour has consequences.
"I've always felt bad for putting my foot down and telling him that I canthave DSD because im busy. I feel guilty that he won't get to see her because of me."
Total bollocks, and again inviting more shitty behaviour. You are "never consulted when he agrees to have her" - this has to stop. There is absolutely no reason why you cannot be involved in family decisions. An easy life involves planning ahead, DSD's stays should be planned and agreed between all three adults. If only two are involved (him and ex) then only those two should be doing the work created from the decision. Put your foot down, that's what it's for. If he (initially doesn't see her), that is down to his poor planning and not involving you at the right (early) stage.
"He's very old fashioned and is lovely and hardworking but lazy around the house and with the kids. He says things in the heat of the moment as we all do but I'm sick of backing down."
He is not 'lovely', and 'old-fashioned' is a self-deluding euphemism for 'entitled mysogynist'. Lovely people do not treat their partner as a cross between staff and a domestic appliance. He is lazy because he can be, because you pick up the slack. You are ENABLING him, and you need to stop. He says things not because it is the heat of the moment, but because he knows you will not call him on it and he knows it is effective in manipulating you.
Look at yourself from the outside. He's got you putting up with shit and guilty about his access to DSD whilst he lives the life of Riley. 'Hardworking'? We are ALL hardworking, it's sort of the default way to live. That's like praising him for putting one foot in front of the other 
.
If you truly want an easy life, you need to toughen up, because how you are handling this situation right now, it is just going to get worse. Your acceptance of his shit just invites him to ramp it up. He doesn't even have to be a bad person to respond that way; he's 'rewarded' for it by your compliance, of course he'll keep doing it. Basic human psychology/nature.
So, you don't like something? You say so, and you don't accept it.
- He tells you DSD's coming tomorrow for a week and that interferes with your plans? You respond that that doesn't work for you and he'll have to rearrange it.
- He leaves pissy tights in DD's shoes? You tell him to pick them up and clean her shoes and ask if he'd leave pissy socks inside his own shoes.
- He squeals complains he's tired and been working hard? You respond that so are you and you've been working harder caring for his children, so STFU and go bath his daughter.
If you do what you've always done, you'll get what you've always got.
And if you won't do it for yourself, consider the welfare of your daughter. Do you want her to grow up in a household where the woman is clearly subordinate? Is that what you want her to learn is her fate, and all she deserves anyway? Think hard, OP. What do you want, both for yourself and for her? And remember - putting up with shit will get you the absolute opposide of an easy life.