Of course you are not being unreasonable to pursue a career you want and you would enjoy, but surely you can still job hunt whilst you are in a job that has more hours and move from there into your dream role? It could be a long time before your dream role comes up (although obviously I hope it wont be!). From what I've seen in the sector at the moment, the main recruitment is going on in specialist areas of support, rather than in the general advice/picking up the phone roles.
Where do you live? That can make a big difference - London = lots of HE institutions, so more opportunities, whereas in other cities you tend to only have a choice of 2 different institutions so opportunities are more limited. Do you know how far you are prepared to travel for work?
I would advise thinking carefully about the exact type of role you want in student support - it is a pretty varied field - and then look at how you can tailor your experience and qualifications to suit that type of role. For example, my student support dept has specialist teams that deal with disability and wellbeing, immigration issues, finance issues, counselling, crisis work, faith support (chaplains and so on), careers, etc etc. Do you know which of these areas you want to work in, or is just student support in general you are interested in? You will have more chance of success I think if you target your role.
If you are interested in student advice, have you tried looking at paid student union roles? A lot of student unions have paid staff in student advice positions.
Would letting your DH know about these sorts of considerations make him realise that this is not a wishy-washy pie in the sky idea, but something you are committed to, and something he should support you in.