I don't think asking to see his child 4 days out of 8 on his days off if you don't work is an unreasonable starting point. But there are then the practicalities to consider and I suspect you've both been trying to discuss arrangements when worked up, angry and both unwilling to compromise.
It sounds like you've tried living amicably under the same roof as co-parents, and that hasn't worked out, so you need somewhere to stay. How are you planning to fund this? If your only options are homeless or move in with your family 90 minutes away, it's a bit of a no-brainer.
Could moving in with your family be a temporary option until you find work in your ex's town that allow you to rent somewhere nearer to him? If it's just a temporary option, he might even be more willing to help you financially to avoid you moving away. Or, if he's unwilling to help you financially, perhaps he can accept you moving in the context of it not being a permanent move.
What sort of work did you do before getting pregnant? What sort of work do you intend to do now? If the answer is a normal Monday to Friday job, then you have two different sets of patterns, and if your ex's 4 days out of 8 always fall on his free days, then you're not always going to get your child on your free days, which isn't fair to you. There is going to have a a compromise where you both find things a little unfair.
I suspect your ex hasn't thought about what happens when you also have a working pattern. If you need to work to afford to live, then, then your likely shift pattern needs to be brought into the discussion too. After all, if he doesn't want to support you financially in the long-term, you need a solution that allows you both to work independently.
If 90 minutes for your ex to travel to see your kids isn't a big deal, then it surely isn't a big deal for your family to travel 90 minutes, especially when your family can potentially bunk with you rather than have to do a 3-hour round trip in one day.
I think you both need to calmly consider the facts and agree to discuss properly when you're both ready to think about how to make co-parenting work. Of course, if your relationship has broken down to the point where that kind of conversation just isn't possible, then mediation it is.
PS I'm sorry this is so hard. Although people are advocating a solution that gives more to your ex than you want, it's not because people are against you. It's just the sad inevitabilities of a relationship breakdown away from home.