im sorry if this doesnt make you feel much better but you could be in a worse position, my dh has to work every hour god sends and we only bring in 14k a year inc. our tax credits etc. so we are struggling, i can understand you're concerned and worried but things could be worse, at the end of the day even if you downsized and your dh was a SAHD it wouldnt be the end of the world and would give him his break, you could then pick yourselves back up.
not sure what im really saying except that it could be far worse and you need to sit down together and put things into perspective, no you dont want to downsize but it would probably be best (i dont want to rent but we just couldnt afford a mortgage) he probably doesnt want to hand financial responsibility to you but again its probably best, you need to try and approach him when hes in a good mood and have a proper conversation about it.
all of you would be better if you were watching the pennies and happy rather than flush with cash and miserable as sh*t