I believe the main reason men end unhappy marriages with an affair is the fact that in times past the default of the court was to give residency of the children primarily to the children. With fathers getting EOW if they were lucky.
This was the situation with my DH when he was in his first marriage. The marriage was horrendously miserable and filled with arguments. This is not my opinion- it is his children's view. They were simply not compatible as partners or as people to parent jointly.
He didn't want to leave his children. He wanted to leave his wife.
When we met (we had an affair for nearly six weeks before he did) I encouraged him to seek legal advice as he was sure the children would have been denied contact with him whatever the reason for him leaving. As this proved to be. His wife did not know about me for at least a year after he left.
In this time he was able to secure a Custody order (a long time ago before CAOs) and even then it had to go to enforcement.
Women who leave a marriage more often than not do not have to make the choice between leaving a bad marriage and losing day to day contact with their kids.
Although 50/50 care is now much more common it is also uncommon for it to be agreed between divorcing parents unless the split is a genuine joint decision. 50/50 in bitter divorces is almost always by court order which takes time.
In our case all our children - both his and mine agree that their mum and I are much happier married to different people.. and enforcing contact in those early days was absolutely essential . It has left the children much happier to have happy parents.
OP - your partner needs to fill in a C100 on the HMCTS website and apply for a child arrangement order. He can then have his child on the times set by the court . What he does with his child on his time - and who he allows his child to see - will be ENTIRELY his business and no one else's. It is his child's right to have an 'effective relationship with his father and if that means he see you then that's normal.
It costs £215 for a CAO and most people do it themselves. No lawyer required. Unless he is a danger to the child then he will DEFINITELY yet an order . I would not want to be with a man who didn't do this for his child.