This is not a worrying problem but it would help if you support him. There are many things you can do. Keep the learning period very short and positive. Let him eat one of the sweets you are using.
What's one/two/, three more or less than (a number under 10, then 20). Do this with buttons, little marshmallows, craft gems, sweets - let your child get used to seeing 4/5/6/7 things in different arrangements so they get a feel for what a number looks like. It's not a symbol on a page. It's an amount.
Counting up in 5s then 10s. You can get the most wonderful penguin number square for this. Or you can get chalk and draw a number line to 100 on the pavement. Have your child run from 0 to 20/70/etc but they can't run past the number or they lose. Try and make it a race but don't let an older child beat them every time. Older siblings mess this stuff up if they come along and effortlessly win it. Try to move from this into saying "Run to the number that is equal to three fives" etc. This is the beginning of times tables. There are lots of fun songs on YouTube if he's interested.
Number bonds. Have two amounts of something adding up to 10 behind your back. They pick a hand, then guess how much is in the other hand. You reveal it and they take a sweet/raisin if they're correct. If they're consistently wrong try saying "Nearly! That was so tricky and you thought carefully about it!". Also have them think of the number bonds as friends. Make up stories about 7 and 3 and have them draw the illustrations. Try making salt dough decorations of the numbers and put the friends together around the house (with eyes and hats, obviously).
More number line work - just make number lines together. Cut up the numbers and ask for help finding them and sticking them together. Obviously better if you find this stuff very tricky and are terribly grateful for his help, don't know how you would have managed without him, must tell Daddy what a huge help he was etc. Incidentally Daddy should do some of this too because statistically that is incredibly beneficial to your child's progress.
Take screens away so there are no competing distractions. Keep it short and frequent. Children are not born with the hardwiring for this in the way that they are hardwired for language although some children take to it as if they were. They have to learn it through practice.
Shop or cafe: Take turns playing the different roles. Label different items together. Stick to pennies at first but if counting in 5s and 10s is going well by all means use bigger amounts. At this stage you can have a day when prices have gone up or down and everything in the store/cafe is 1p more or less than labelled.
Get a white board. Play a game where he gets a point if he can shout out the number you're writing before you've finished writing it. If he doesn't, you get a point. Whoever gets to ten first wins the game.
Don't make writing the focus of maths. Any written work he does should be to show his learning. You don't learn maths with a pencil in your hand.
Don't be afraid to give each number under 10 a colour and a song if he is that way inclined. You can then play number bonds by saying "purple (ie 7) and he says orange (3). Give him as much ownership as possible. Let him choose the numbers. If he's imaginative. Let him imagine a different kingdom for each number. Eg. 3 is a land of dragons that have breakfast three times a day, say good morning three times, sneeze three times etc. The possibilities are endless and there is a lot of opportunity to relate warmly to each other and enjoy shared laughter. Any artwork around this put it up and comment on it to cement the idea that grown ups take this seriously. Maybe Granny could send a card with a freddo bar because she is so proud of how much effort he is putting into helping you with the numbers.
Some children love Cuisenaire rods and numicon. As a home educating parent, I spent a fortune on them and barely used them. But definitely worth a look.
Children have an inbuilt knowledge of halves, quarters and thirds depending on how many siblings they have. They know their rights. I find that stories around theft are most interesting. So you were walking along with four sweets and half of them were stolen. How many were stolen and how many were left. Or you had six beautiful multicoloured fish and half of them went down the plughole so you went to the drain and needing to track down how many fish. To make this a step harder, you could produce two Oreos and say that the dog ate half of them and this is all that are left. How many Oreos were in the packet this morning before the dog got at them? You can also have a lot of fun with coloured water and measuring jugs.
Number blocks are helpful for showing the difference between odd and even numbers. You can show that even numbers are always paired while odd numbers have pairs and then one over. (Numicon shows this too). So you can then fit the odd numbers together with their unpaired extra number to make an even number. And you can fit 2 numbers together to make an even number. But an odd and an even number will still have one left over. This will be helpful to know later.
3D solids (don't call them shapes because shapes are 2D and only have sides) are really useful to start becoming familiar with. Words like cylinder, face, edge, pyramid are all easy to discuss now rather than trying to cram the knowledge in later. Also don't be afraid of using Venn and Carroll diagrams. I know they're daunting to look at but they are fun to draw out. Distinguishing a curved face from a flat side is part of maths. A curved face means that the solid will roll - eg a sphere or cylinder but not cube or pyramid. Collect things around the house and put them in your diagram. Huge sheets of cheap white paper are your friend. Photograph everything.
Practice being able to describe one thing in relation to another using appropriate language. Arrange coloured pipe cleaners in order of increasing or decreasing length. You can ask him to make a sentence telling you about the length of the blue stick compared to the red stick (all the more fun if he has collected these sticks and personally dipped them in paint). You're looking for sentences like "The yellow stick is longer/shorter than the blue stick" etc and do the same thing for heavier than and lighter than.
Talk about how solids don't change shape but liquids will take on the shape of their container and gases just float away. Useful later. Discuss when it would be fun to be a gas and when you'd rather be a solid.
Take every opportunity to let your child play the teacher and especially teach Daddy or other supportive adults. Your child is not learning to hit bench marks. They are learning how to engage, how to explore with curiosity, how to enjoy learning and how to enjoy a different and precious facet of their relationship with you. The more you are prepared to make learning a way of life and respect your child's task to grow and explore, the more engaged and confident they will become about their learning. The fact that they will sit down and find the homework sheets a doddle after all this is irrelevant. Worksheets are an opportunity to show what you know or a springboard into learning.
The white rose website has helpful videos. Happy learning!