"Apart from these reactions he has never treated me badly, he is kind hearted, come from a good family, he is financially stable, has good morals etc where as my dad lived in a council house, never bought it, spent all his money down the pub and never left me anything, used to charge me rent for living there"
You call his abuse of you his reactions?.
All of this re him is irrelevant because of the ways in which he has and continues to treat you and in turn your DD who is absorbing all this from her dad and you like a sponge.
His father and he not charging you rent was not the kindness you thought it was either; you could have shown a rent book then but your status is not even that of a lodger. He could actually throw you and your DD out at any time. Your position is that vulnerable.
Your dad showed you a really poor male relationship example and with this man you have found someone not too dissimilar to your dad. You were not right to cheat but I feel you actually cheated because you were that unhappy with him back then and perhaps therefore fell for a person who actually showed you some kindness and interest. Something that your man does not do even now and he has already told you he no longer loves you. I do not think he ever did actually; you were just useful to him.
Your own lack of boundaries and low relationship as a result of that in your childhood made you ideal for someone like this man to sink his claws into; he decided to target you accordingly.
I tell you now that your DD would prefer a smaller home anyway because her home currently is clearly not a happy one. Material things mean nothing to such young children, she wants a happy mother!!.
You as her mother are miserable and she will pick up on your inherent unhappiness too. As for him getting angry over a towel not being in the bathroom, that is a clear overreaction on his part but is an ideal thing to beat you further about the head with.