You aren't. The reason people are talking about him being abusive is because if he was the one making you feel as though you're being inconsiderate, that would be so out of line/unreasonable that it would be abuse.
Since it's coming from you, you need to give yourself a break!
Imagine a small company of four people: One makes a product, one researches the best and new ways to make the product and feeds this back to the maker, one advertises and sells the product, the fourth person cleans the office, does the accounts, orders and keeps track of materials, makes the tea.
The salesperson is the one bringing all the money in, but they don't take home the largest salary. The money that the salesperson brings in is used to pay for materials, the electricity bill, the advertising costs etc and then whatever is left over is profit, to be divided between the four people as their take home pay.
It would not be fair for the salesperson to take everything, and say well - research person, you need to go online and sell your ideas to bring in some money. Making person, you need to keep some product back and sell it. It's not fair that I am the only one bringing in money here. This would be stupid because the person who is good at sales should be doing all of the selling, it would stretch the other people too thin if they were doing their jobs and also thinking about how directly they could bring money into the company.
A family should be the same IMO. You have two people who directly bring in different amounts of money (perhaps one of them all of it, perhaps just different amounts) but once it's in the family, it's just all one pot. It's used for the family expenses (housing, heating, childcare, food, cars toys clothes, savings etc) and the rest is either divided out between a small amount of pocket money for older children and an amount of spending money for adults, or put into a communal pot for general spending.
If you are having issues budgeting as a family I recommend YNAB, it's not free but it's paid for itself in our family and it's helped me understand DH's wage and everything seems less confusing.