Your daughter has only just started school, so I don't think that's such an issue. It'd be different if it were an exam year. I think moving the family in the summer is a good idea - I have friends who moved back home this summer. At least it's where we grew up, but the children had never lived there - they had been living about 2.5 hours away, but saw the grandparents in the holidays. They seem to be settling in well, though. They did wait till the summer hols though, which is a sort of transitory time of year anyway, school-wise.
I think the main issue is going to be your husband and how flexible his current job will be, and how easy it will be for him to find another job - which might be easy or may not - it will depend on the field he works in and where you are in the country, I guess.
But none of this is insurmountable. Just make sure if you've any particular questions directly related to the job, be it hours, flexibility or whatever, if it hasn't come up during the interview, ask them when they ask at the end if there's anything you want to know. Interviews are two-way things. Make sure it's the job you want and expect it to be.
Your husband is right - you don't know until you try, and until you've been interviewed and offered it, you have no decision to make anyway. If nothing else, it's always useful to get interview practice, and if you don't get it or don't take it, it should give you a clearer idea about what you do for the next interview.
Good luck!