I'd be very worried.
It wouldn't take a lot for your property empire based on interest only mortgages to come crashing down.
Interst rates are historically low and the rents you are receiving should more than cover the mortgage payements. But some of us remember the late 1980's when they peaked at 15%. On Black Wednesday they rose from 12% to 15% in one day.
Have you considered how much your interst only mortgage payments would be at 15% or even half of that. At what level of interest rate does your rents start to be outweighed by your mortgage payments?
You probably think it's unlikely that they would reach those levels again and you're probably safe in assuming that if they do rise they will do so gradually. However when they do start to rise (as they must) you'll be just another Buy to let landlord caught in the stampede of those desperate to sell because the rents are exceeded by the mortgage payments.
You've been just as irresponsible as your DH. These debts are in your name - not his. If interest rates rise and house prices fall then you will be in negative equity forking out money to meet the shortfall in your tenants rent or being forced to sell at a loss.
There are people in this world who are risk-takers and there are those who are cautious, weigh up the risks before they take on what could be life-changing committments, and have an exit strategy. You've just sailed along in your DH's wake hoping, as he tells you, everything will be OK. But we are in uncharted waters in terms of the banking crisis, Euro crisis etc. No one can predict how thiings will pan out over the next decade. You could be millionaires - I doubt it.
I don't know how you rein in your husband but you should start by reining yourself in and having enough savings to extract yourself and your DC from your DH's affairs if it all goes belly up - because from what you've said here, he has no Plan B if his Ponzi scheme hits the buffers.
I have a strange feeling that this is a wind-up by some buy-to-let landlords having a laugh at our reactions to this.