I think you are right, where you said you need to get better at setting boundaries. I think people who aren't confident of their own rights also tend to attract self-centered and/or abusive people.
I'm not one to suggest ending an established relationship at the first misstep or obnoxious habit. However, I would be open to the possibility that this "might" not be a person that it's possible to have a healthy relationship with.
From what you've said, he seems threatened by your work role and wanting to make you feel smaller than you are because of his insecurities, rather than being supportive of you. He also seems overly self-centered, at your expense.
However, if you have allowed him to take this position then I could see how you might want to try changing the dynamic before taking a more drastic step. It's also possible that his health problems contribute to bringing out a childish and unreasonable side of him. If so, maybe try something like this, as a step between letting him continue to mistreat you and ending the relationship:
- Tell him he is not to come over any more during your work hours.
- Tell him he is not to make any more disparaging remarks about your job.
Then you will need to be prepared to back up your words. Spoiled adults, like spoiled children, don't like to be told no and are, in my experience, likely to test you. So I'd be expecting it.
If he comes over during work hours anyway, don't let him in. If one of your children lets him in, tell him he needs to leave. If he doesn't respect any of that, after you get him out of your house, don't speak to him for a week. This is to demonstrate to him that things have truly changed.
If he makes another insulting comment about your work role, don't speak to him for a week, to show him that things have change.
But if you need to do the above, consider if someone who needs to be "trained" like a dog to treat you with basic respect is the kind of guy you want to partner up with. Good luck and please keep us posted.