Arguments could arise for lots of reasons, many of which could be do to do with how you raise your child - which you haven't discussed. How to parent, who's going to be contributing what financially, what hour plans for childcare are, who's doing night wakings and feeding, when or if to use nursery...?
If you move 2 hours away from your home and in with a man you barely know, away from your family and friends, with no knowledge of the area, how can you not see that would be an enormous strain on the already-scant relationship you have? You're very likely going to be lonely, hormonal, isolated from friends and family, feeling homesick and dependent on him, while navigating your potentially rekindled relationship.
I know that sounds doom and gloom, but it's enormously stressful for strong, secure couples who have a settled home life and know where their local baby groups are and have friends to pop in for a brew on the days you need support and a listening ear, and relatives to come round and do your washing up while you're exhausted, who've already planned their finances and how to raise their child well in advance. To do all of that with someone you don't know very well, whilst trying to rekindle a relationship, in an area you don't know, is so obviously fraught with pitfalls and potential resentment and difficulties.
Don't get your hopes up - he may be a wonderful man and it may be fantastic. But it is more likely to be very, very hard and you need to be prepared for him to be pretty angry about you not telling him.