I have said there's things I just don't like, even in fantasy, like for example one time I said I didn't want to have sex with other men (while he watched), this was very early on and I was trying to hold back a bit. But he still pushed it, talking about things that I had said I didn't like. But usually after another talk it stopped... but he finds ways around it a bit.
NO NO NO! This is an attempt at sexual coercion and a definite dumping offence.
Another thing I've noticed though is he's not that interested in hearing my fantasies... It's not a red flag but I thought that was strange. My ex always preferred hearing mine.
It is a red flag- he's trying to be the one getting everything his own way sexually and in control of it all. It's probably part of the kink for him, but is very unpleasant.
he kind of introduced the sub,/dom dynamic to me and I did really like a lot if it at the beginning.
That may be another example of him being pushy. Certainly the 'three holes' bit sounds worse in that case.
if it is me being oversensitive
No, he's trying to pressurize you. I'm in therapy for my ex's sexually coercive behaviour, constantly barraging me with sexual nagging. I felt uncomfortable but felt trapped, as he'd implied (he was a FWB) that if I didn't do stuff he wanted the friendship would end. It is actually making me throw up, thinking of the anxiety/horror of being trapped etc. You will have trauma if you stick around in this relationship.
it's more important to me that other people are happy than if I am.
You need to stop this. It leads us to be easy to manipulate. Being a 'people pleaser' in a relationship, if you mix it with BDSM can lead to a pretty bad dynamic in my recent experience, as the BDSM element intensifies any power imbalance in the relationship. You might well end up doing stuff you don't want to do. 
Some people play their BDSM in a hardcore way, where the dom is allowed/expected to push boundaries. But I don't think most people would play that way.
I'd work on your people-pleasing and maybe low self esteem. Why does everyone deserve to be put before you? You should be equally important to you as the other in any kind of relationship- as you have a duty to care for yourself.
I won't lie to you though, I'm going to do the work on how to set boundaries and listening to myself and meeting my own needs (as well as others), because something about being a submissive is very appealing to me.
I know most BDSM isn't pathological, but for me I think a lot of wanting a man I could look upto was due to my childhood- my father was weak and I had no respect for him etc. There may be some childhood issue that is leading you to want a relationship where there is an imbalance of power.
There are loads of things you could look into before you do any BDSM.
My personal experience would be, if someone can handle it, this stuff is best if it stays in the bedroom. So, in bed you might want to be treated a certain way, but it doesn't spill over into everyday life or mimic an everyday pattern. I was in a Total Power Exchange (the dom is in control of a lot of the sub's life etc) relationship for a month or so a few years ago. It soon got boring to me to be set tasks or whatever- I was crap at walking in heels, so he wanted me to walk in them in the house for, I think it was 15mins a day. As it was also a long distance relationship, he had very little control of what I did, I didn't say anything I just stopped doing it. 
He also ignored my safeword once when beating me.
he didn't show much interest in me at all at times, I have just felt like someone who was there to receive him.
He is a user who is/was trying to sexually pressurize you.
I just walked into a world I didn't know anything about and just assumed he would be good at it and be leading me in the right direction. It could be worse I suppose.
I suggest joining Fetlife, not to pull, just because I think you can learn a lot about BDSM on there. There's also a BDSM wiki which is interesting to read. www.bdsmwiki.info/Main_Page
Join Fetlife IMO, and don't let any man make you come off it. My ex tried to say it was a dating site and so I shouldn't have joined. This was part of his control. I didn't say anything (that's what people pleasers are like, and it can get us in a lot of trouble. But I rightly thought to myself that that's not a dating site, or at least doesn't have to be used as one. It's a way of getting different perspectives. If a bloke tries to control what you read online about BDSM or whatever, or is that jealous, they should be expunged from your life.
Best wishes and tread carefully. xxxxx