Leaving aside all the stuff about kids, there are plenty of other considerations.
If he moves into your house, the power balance is going to change, very much in your favour. Assuming he won't be owning any part of it, you will be able to chuck him out at any time.
What is his current living situation? Is it his flat, does he rent it? Is it HA?
What financial arrangements do you have in mind for after he moves in? Will the benefits you get, if any, be affected? If so, who is going to make up the shortfall? Is he going to pay rent, or just contribute to bills? Are you going to split food bills 50-50, or will you pay more because of your three other kids? What about other costs (school trips, uniform etc). Will you keep separate finances, or will all money become family money?
If you have discussed all this, then great. Honestly, though, you don't come across as someone who would have discussed practicalities before jumping in. But anyway, they are some practical reasons why people might be reluctant to move in.
Sorry to hear about your miscarriage.