My 4yo does this, in fact I came to this board trying to see if this was "gifted". Last time I saw something about another 4yo who could do a lot more so thought it was more ordinary.
My 4yo (born end of Jan so not in reception till Sept) absolutely loves numberblocks, I think this is what sparked his interest! And the mathlink cubes- he recreates full episodes of it with great joy. He knows his times tables, adds and subtracts in his head, multiplies, divides and does fractions, knows odd/even, primes, sequences of squares and cubes, number bonds to 20 and lots other things he's learnt by looking over his siblings shoulders when they do mathletics for their homework. He is in nursery and is the only kid there doing this. They don't teach them much beyond recognising and writing numbers 1-10, he does it himself.
He seems to have a love of maths and numbers and I think he's got a talent for it. He will be starting reception in September and I am pretty sure he will find the curriculum very easy though showing what he knows in writing will be new.
His older brother was similarly gifted in maths at this age, and is autistic and it has lead me to wonder whether the hyper numeracy is a possible sign my 4yo is also neurodiverse (sometimes he seems to bring conversations back to numbers a lot, often out of context, but he's 4, they all say a lot of random things so I don't know if it's an ND trait) so far I haven't spotted the other signs that I did with my older son by this age. His fine motor skills are fine (in line with age, not terrible like my other son) and he has no sensory struggles (older son hyposensitive and sensory seeking) and uses more gestures and eye contact. Has a bestie in his class and is sociable. Time will tell though.
I think your son (and mine) would be considered to have a high aptitude for maths at this age.
I have noticed that the maths skills leveled out a bit, so the lead that my older boy had over his twin sister and peers has narrowed a lot as they've gotten older, what once stood out as streets ahead in nursery now looks more like being a bit ahead (both twins just got max scores in their y2 maths SATs) for example. He remains strong at maths and absolutely loves it, but it isn't genius.
Since they like maths, I let them have access to fun resources rather than trying to push them on through the curriculum, there are loads of nice fun maths games (dice games, board games, online games) and other things like chess, coding. But I actually spend more time trying to round out other areas, like getting my older one to stop doing everything mentally and show his workings or show alternative methods which he gets annoyed by! I know in early years sometimes an exceptional memory and good mental arithmetic might not necessarily translate into strength at problem solving or harder maths as they get older. We work more on fine motor or writing because that's where there's an asymmetry of skill. Maybe if my older son had the maths skills without the challenges he faces as part of being autistic I might have wanted to "develop" it, I don't know. My 4yo is the youngest of 4, and myself and my husband work full time so I haven't the time or inclination to try to take it further, I hope I am not doing him a disservice in this, just don't know how on earth I would fit it in.
I think at 4 whatever you do, keep the joy alive. They learn so much faster and so much more deeply through play and self discovery.