I don't know if you should go back, or how your relationship with your H has been. But it does seem that you are carrying a lot of trauma and have expectations about what a relationship should be. And it is possible that those expectations are affecting how you are in a relationship - this or any other you may be in.
"All I’ve ever wanted was a proper family, I didn’t get that growing up" - that is a lot to carry. That also makes you want and expect some "proper family" ideal that may be impossible to attain and maintain.
"I left him for numerous reasons, he never made me feel attractive, important, didn’t prioritise me, his work has always been his man priority"-- here is where I think it is very clear that what you want/expect/need is maybe not something a relationship and family is meant to be.
You say you wanted a family - yet your reasons for leaving are about YOU and your self esteem. It is not on your family - or your H - to make you feel attractive and prop up your self-esteem. I'd also go as far as to say that how you feel about your attractiveness is NOT the main reason to have a family, and certainly not after you have children - where I believe that the priority do shift. Or at least - should. We bring kids to this world - and sort of make a commitment to prioritise them while they are young - to raise and nurture them. And if you are struggling with your self esteem - you need to work on it on your own - maybe with a counsellor?
You want and expect your H to be your cheerleader and prop you up. But you don't feel attractive and important on your own - no one can constantly be there to cheer you. Maybe your H is inattentive - I am not there. But I am not sure it's realistic to expect any man would to be there to do this for you on a long term basis.
As to your H working all the time - it is, of course, hard to say much without knowing more about your circumstances. My exH also used to work all the time - and sure it bothered me too. But at the same time - I knew the demands of the sport of work he was doing. And he worked to provide for our family. So - as in any partnership we did our respective bits.
I'll conclude saying again that I don't know if you need to try again with your H.
It is clear in your OP that you have a lot of pent up issues - some of them specific to your H, but many more are specific to you and what you want/need/expect from a relationship. Personally - I think you could benefit from talking to a professional who can help you sort out how you feel and self-reflect on what you expect from this or ANY relationship.