OP, I have been in a relationship that is similar to what you describe. It is emotional abuse. He is undermining you, subtly (and not so subtly) undermining your confidence, and making you question yourself. He does not accept you for who you are. However much you change yourself, it will not be enough for him. He doesn't sound as though he loves you, but that he wants to control you. Lots of the things you are saying are waving massive red flags at me - you have set times for contact (wtf is that all about?? He just wants you when and where he wants you, and not otherwise?), he said you were using "false advertising" (my ex said that about me, those exact words, but about a facet of my appearance. It is aimed at undermining your self esteem and confidence, and it is aimed to kick you exactly where it hurts). I made excuses for my ex for a long time - "oh he's tired", "he's having a hard time at work", "he's just not that sociable" etc etc. No, actually he was just an abusive arsehole.
I compare my relationship to a slot machine - you put in coins (or invest emotionally), and every so often, you get a reward out, which is just enough to keep you putting in more emotional 'coins'. Over time it will get worse, but you become addicted.
You say you feel fragile about breaking up, but it sounds as though you know in your heart it is the right thing to do. Why are you feeling fragile? Is it because he will persuade you not to do it? My ex did this numerous times, so be prepared for that.
I thank goodness you aren't living with him. I was, and it was a nightmare to escape from. But I will be forever glad that I was strong enough to do it, even though it was shitty at the time.
Also, think of your DD - what made a big difference for me was when I began to see that my ex's abuse of me was having an impact on my DD. He had started using bullying tactics on her, and was undermining her confidence too - humiliating her and making her feel small.
If your gut is saying it isn't right, please listen to it. Why do you stick with a relationship that is "strict and lonely"? - you don't. You deserve a relationship that is open and honest, and loving and kind.
Good luck to you OP.