You are in a very vulnerable position, OP. You paid a significant chunk of capital into the house, I'm assuming your name is on the deeds? You've effectively got no recent credit history because you're not managing a current account so at minimum you should have one (poss one of the overdraft-less ones like the Co-op do).
And you're given a card once a week to do a shop. How demeaning is that? I'd tell him if he wants to control the card he can do the sodding shop as well.
Is the extent of your DH's treat money just enough to to go out and about, and have a hot chocolate in Costa and get in the bus and mooch about a bit.? If it is, fine, but my guess he has significantly more fun money than you do.
Your DH doesn't pay extra tax if you claim CB. He pays back the equivalent of what you have been given in CB. At a household level it works out the same. And is better for you because again it gives you a level of income of your own and a financial presence. Frankly if the marriage goes tits up you will need to be claiming this.
So far you've not given any evidence of financial mismanagement on your part. If you use a joint account your DH will be able to access all the transactions online and keep an eye on things if there actually is any reason to be concerned about your spending habits.
I find it very disturbing that your DH closed the joint account so soon after your dad's death (and when he must have known how much you were due to inherit?)
The current situation does not sit right with you, and I think that's for a good reason. Having to go and ask for money is not acceptable to you, and that's the bottom line as far a the discussion goes. Would he like it? You bought a house for him but you don't make him ask to live in it.