I dislike him when he's with his friends he's loud and obnoxious. Honestly I am having a whole re think of this relationship.
IMHO there are a few intertwined issues here, and him telling you he was with a sex worker at 17 isn’t the main issue: it’s a symptom.
It sounds like, as you’ve got to know him, it’s come to light he hasn’t been entirely honest with you about who he is. And you (quite understandably) don’t trust him. It’s not really any of your concern who he had sex with before you knew him, as long as he’s had safe sex with you. But…
We had talked about things like this before and he would always say I'd never pay for sex that's desperate.
This means he’s lied to you. That’s a betrayal of trust. It makes it more likely he’s been deceptive about other things. Including, perhaps, that he’s been with a sex worker since he’s been with you, as you’ve suspected. You should always listen to your instinct. It’s a powerful thing.
Is your relationship together good? Is he a good partner to you? How does he feel about his friends? Does he condone or like their attitude, including them being with sex workers? Would his preference be to hang around with them going forward? Or does he mainly see them out of habit and loyalty, because they’re old friends?
This is the crux. If he likes hanging around with men like that, and condones their behaviour, it says a lot about him, and sounds like he’s not the man for you. However, if he’s always been surrounded by men like this (seems the case, if the brother encouraged him to be with a sex worker at 17), then maybe he’s weak and goes along with it. Maybe he’s now matured and changed and they haven’t. Maybe he always felt a bit uncomfortable.
He needs to honestly and openly explain to you who he is and what his values are. He’s a father. How would he feel about men using his DD for sex? How would he feel about his DS being with a sex worker? You’re entitled to know this if you have DC together. It’s fundamental values. Does he feel the loud and obnoxious man he is when with these friends is the real him?
Once you know that, you can decide if that’s the sort of man you want to remain in a relationship with. Don’t stay in a relationship with someone you don’t share values with, trust or even feel comfortable him for the sake of your kids. Even co-parenting will be hard if you have very different views on how to behave. So you’d have to discuss and agree what you both teach your kids about values. So your kids aren’t exposed to this.