How old is your baby? And how long were you together before you had your child? And how were things then?
I'm asking because I think having a child can put strain on even the strongest of relationships, so if you don't have a lot of experience of dealing with life together before the baby comes along, it can be even harder to adjust to the new challenges as a couple. That doesn't mean it couldn't work. But it would take more time and effort - which it does sound like you have been trying to put in. The trouble is, you would both have to really want it.....
Would you consider relationship counselling? Would he?
It does sound like your grounds for being suspicious in both cases have been quite slight - would you say you have a tendency to be jealous or insecure? That's not a criticism - it's certainly something I have struggled with myself, I think a lot of people so. But again, it is something counselling can help with.....
Tnat would be my first step if I were you. Obviously it would be ideal on many ways if you manage to deal with your differences and stay together - but I would also say, from experience, that it's possible to coparent really well without being a couple, providing you both strive to keep things amicable with the child's needs always paramount for both of you.
One thing I did notice is that you say "we like each other". Not love. If I was counselling you both that is the first thing I would pick up on and explore.
Is there love between you? Has there ever been? And what would things need to look like for you both to feel love for the other.
If there is no love, has never been, and you don't feel love would be possible, better to separate now and find a wag to support each other as coparenrs who both love your shared child.