I would consult a family solicitor. It doesn't have to end in court, but a third party mediator might help you both to sort out a more formal agreement that you both have to agree to.
I wouldn't have felt comfortable about leaving mine for any length of time at that age - my middle daughter would only settle for me for a very long time and screamed until she was brought back to me. My mum (who knew her really well) tried to test this and took her off for a walk in the pram screaming. She came back after an hour with a still screaming baby.
I think it's important to involve the father as much as possible, and it sounds like you've had a really sensible approach so far in terms of letting him help with nights and having access.
It sounds like you're a bit scared of him and allowing him to call the shots. I think you could do with some help to reverse that - could you manage to get some counselling?
Also, if you do decide to take the baby to where his family are and stay locally, you need to make sure the father is footing the bill. Don't be out of pocket because he has made these decisions.
Finally, you definitely need to try and get a financial agreement in place. Your baby might not need much now, but formula and nappies add up, and he will constantly need new clothes. I think there are calculations on the CSA website to give an indication of what he should be paying - do you know roughly how much he earns? Unfortunately children only ever seem to get more expensive...
If you are planning to go back to work at some point, there's also childcare costs to consider - these should also be split.
I think some family solicitors give a free initial consultation - go with a clear list of questions that you need answers to and they will help you find a way forward that's best for the baby.