Look Polka, you said in your OP that this gets worse with every holiday, so we were led to assume that you actually lose it every holiday you have with your ds.
It's easier when they are at school, but then the holidays come and suddenly there is no getting away from their excess energy and demands for attention. What disturbed me though was not that you lost your temper with him, or shouted at him even if it was every day, it was the words you used towards him. Those words have come from somewhere and they are designed to have a real impact, to get a reaction.
That's what you need to look at.
Do you remember getting any homework in primary school yourself? No, because it didn't happen. This is a relatively new thing and I'm not sure I agree with it. It just puts pressure onto parents and children. They sit down and are told to be quiet and behave for 6 hours a day, 5 days a week and then when they get home they have to do yet more work??? Playing is all part of the learning process. They are learning hand-eye co-ordination, they are learning about role play, about sharing, about the world around them and I think that's much more important than reinforcing what they have already learnt in school.
Not that this makes it easier for you. But I will guarantee that in your son's class there will be a few children who don't do any homework because there is actually a large percentage of parents who refuse to take part in homework whilst their children are so young.
So, let the homework slip. What's the worst that is going to happen? Some parents even have an agreement with the teachers to do either no homework or just a small part like reading once a week.
Now, to tackle the holidays when your dh doesn't help out. Could you arrange playdates? Trips to the library, museums and so on? There are always activities around holidays for younger kids and if you plan something for every day you will find yourself more in control and able therefore, to control your temper better.
When he is at school, could you not do a job then? Are the school looking for Teaching Assistants? Lunchtime Assistants? Playground Assistants? It will give you a little money and get you out of the house and mixing with others.
If you can afford private CBT then great, but you do need to act on this as you did say that your behaviour gets worse every holiday, so this is not a one-off is it?
I was harsh with you because you seemed to be more interested in telling us what a horrible child he was than telling us what you were going to do about it. Now you are trying damage limitation but this has got to be quite bad hasn't it? Be honest.
I am thinking about the welfare of your little boy at such a vulnerable age so apologises if my posts are not solely focused on you OP.