OP, I'm a bit confused here. Either this is or is not a problem for you. You say it is, then go on to excuse his behaviour and say it's not often, And he's really perfect in every other way. Few people are all bad, and he may be an otherwise pleasant enough person- but he is not a good partner to you at present and that is reason enough to leave.
It does sound there may be reasons for him acting this way- but that is all they are- they aren't excuses. And it isn't ok to make you feel upset and hurt, or to accuse you of something you haven't done or to make you feel guilty. You aren't responsible for whatever happened in the past and aren't responsible for his feelings of insecurity.
The other issue you have here OP is that if this is simply all about his low self-esteem and so on, rather than attempts to be controlling/abusive- you do realise that you can't fix this? You'll never be able to be supportive/loving enough that he feels secure. You'll never be able to prove to his satisfaction that you really are trustworthy- there will always be a "but" and a "what if" or a "well maybe you haven't done it yet, but you will/you're tempted" etc. Your partner is the only one who can fix this- but it doesn't sound like he really sees there is a problem- or rather, he thinks the problem is that you are having an affair and so he is justified in his behaviour.
He may not have said "please leave your job, I hate that you work with so many men" and he may look after your DD to let you go and so on. And yet you are still very aware that he does not like the fact you work in a male dominated environment and would rather you left. So he doesn't have to actually have said "leave your job" to make you well aware of his feelings on the matter.
I'd say what he is doing, regardless of the underlying cause, is very manipulative- whether it is consciously done or not. Manipulation is one way of gaining control- it's not as controlling as stopping you going out or making you phone every 10 minutes or something. But it is a subtle form of control nonetheless.
I wouldn't personally put up with it as I would find being told that I am untrustworthy, a liar, deceitful and selfish (which is what I would feel a partner who constantly accused me of cheating is essentially saying, IMHO) would be intolerable. I would resent the attempts at manipulation. And I suspect I would lose any respect I had for them because they were being so unreasonable. Whatever the reasons for their behaviour.
I think it's a bad example to set a child too, if I'm honest.
Overall, only you can decide what you are willing to put up with. If you want to work on it, then you need to decide your boundaries/"red lines" and sit down and have a serious chat about how things have to change and have a tangible plan about how that is going to be achieved rather than accepting empty promises. But if you make the threat to leave if his behaviour doesn't change, you need to mean it.