Just don't tell them! If it works for you, and you are very very sure it's not about hiim financially controlling you in an abusive way, it's only your business, not theirs.
However those people telling you to be more responsible do have a point.
You're an adult and should be able to control your own money in a responsible way.
I've seen lots of posts on here where women lose respect for partners that act in a childish way (whether to do with money, housework, childcare etc). Be very careful that's not you. Abdicating responsibility for basic life skills can be very wearing for the person picking up the slack.
Also think - what would happen if you suddenly had to take over the finances of the household - say if your husband had to work abroad for a year?
You could take small steps to take responsibility back - for instance, can you do the money transfer for the housekeeping each Friday, and give him back the card immediately afterwards? If that's successful, can you then (for special purchases such as birthdays) take the card for a day, ONLY do that transaction, and then put it back in a mutually agreed place where the card should live?
I keep a spreadsheet each month (in work, v easy to check a couple of times a day) where I have all my incomings and outgoings. Every week or so I check my bank balance, I note what direct debits haven't been paid yet, so I know how much needs to be in the account to meet these, and any extra is for luxuries/outings etc. I'm not naturally good with money, but I find the more monitoring I do, the better I get. The point is, I made myself better at managing my finances, and now I don't have to gird my loins (mentally) to check the level of my overdraft every month. Plus it's my own skill, so I never have to rely on anyone 'policing' my spending.