Of course it taints the 'good times'. Once you see that a person has a nasty, vicious, contemptuous, dismissive, selfish, arrogant or disgusting side to them, you can't unsee it.
Even when you're sitting in a nice restaurant, eating a delicious meal and he's chatting away about his week at work or a new album he's been listening to, on the inside you know that what you see right now is not always what you get.
It puts you on edge about what might happen when you go home after dinner, or whether the mood might change if you say the wrong thing. You might feel uncomfortable about the cost of what you order if he's been making fun of what you earn. You might simply feel resentful that he can behave nicely in public and horribly behind closed doors.
I spent way too long in a relationship where if I was honest with myself, it was obvious that my partner actually didn't like me all that much, or think all that much of me, or care about my feelings. Of course he SAID he loved me when he was walking out the door to work or after I'd made a nice meal. But when I needed his help, or didn't like the way he was speaking to me, or had a different opinion to him about something he considered himself an expert in, his real view of me became perfectly clear.
In all honesty, your partner mocking you and making fun of you in an unkind way only needs to happen once for you to decide to end the relationship. People who care about each other simply don't do that kind of shit. I didn't even like my ex-partner very much by the end of the relationship, but even then I didn't do anything like that to him, because I'm not abusive.
Leave. It won't get better, as a previous poster astutely said.