How did you reach the decision you'd move to where he lives (rather than him moving closer to you or you both moving to a new location all together)? How isolated does moving to where he is make you from family and friends?
Giving up a job without a new job to go to or considerable savings to live on/a well developed plan to set up your own business reads as risky in the current economic climate (even allowing for him being both willing and able to support you at present and for a considered assessment that there are jobs in your field in his area) - what happens if he's made redundant next month? Then you have no ongoing income between you other than JSA.
In the longer term (assuming you move into a house you have no legal interest in ie. he owns it or he is the sole tenant), don't pay his mortgage, rent, any DIY or other decorating or maintenance costs as you will get nothing for them. Pay your fair share of gas/electricity/water/landline/internet/TV licence/Council Tax costs. Consider what you want to do by way of contents insurance - this might be worth paying for jointly (though any buildings insurance must be his alone). In your own budget, prioritise (a) repaying debt, (b) establishing savings, (c) transport - car MOT + maintenance, road tax, petrol and/or money for train tickets, so you can maintain links with family/friends in other areas of the country.
Having left one job, it will take you some time to acquire rights to SMP in any new job - six months employment, completed by the week 15 weeks before your due date. If you are serious about having children with him soon, in your employment search look for employers offering contractual maternity pay above the SMP level and who are good at flexible working arrangements.