YABU. I fly a lot for work and I wouldnt dream of doing this.
if there is an emergency or you had to evacuate the plane you wouldnt have the option of going to help your children, they would have to fend for themselves. you would be completely reliant on the goodwill of unknown strangers for their welfare.
You have no control over who they are sitting with, you might find them sat next to an obnoxious drunk. I have in the past been asked if I would move to sit next to two UM as they were on a night flight and the cabin crew werent at all happy about them being sat all night next to the rather odd man they had been seated beside. I did move in this case, but why should I end up supervising other peoples kids just because I am on my own and look like someones mum??
I have also seen a fight kick off on a plane before and it was very scary.
Tell your H not to be a selfish arse and book a holiday you can afford