I would assume genes play some role.
I have always excelled at Maths, as has my DH.
Both of us did well at Maths challenge when we were at school.
ALL of our parents (my childrens grand parents) are also talented at Maths, and their careers reflected this.
Even all 4 of my grandparents were good at maths and used maths (without calculators but with their brains) for their careers, and the 2 of DHs grandparents who I had met were also highly intelligent people who were well educated and im sure would have been pretty good at Maths. (I have no reason to believe the 2 i didnt meet were not academically able either)
My children have oxbridge educated parents, aunts / uncles and grandparents, and its fair to say DD1 also is aiming for Oxford.
Both of my children have got Gold / best in school / best in year for Maths challenge (at a top independant school, where they do not focus on or practise maths challenge) and got through to kangeroo. DD1 was 1 mark from olympiad.
Yes, they have both had a "rich childhood upbringing", but im sure their genes have helped too! Neither have had any extra private tuition or 11+ tuition etc. (but DD1, the only one who did 11+ sailed through hers with no tuition, but chose to stay at her current school, and declined her place in one of the best selective state schools around.... because her older sister was staying at their current school we didnt enter DD2 into the 11+ so cant comment on her performance as she didnt take it.)