OP I'm not going to tell you not to worry because as a Mum that's part of the job description
but I hope my experience might make you feel somewhat less anxious about the future in the sense that his current behaviour isn't an automatic predictor of the future.
My son since being a toddler had always been quite single minded. He spent his 3rd birthday party totally wrapped up in a present and ignored all his friends by way of example.
At primary school he was figgity and mildly disruptive - all the time. He struggled with writing particularly and was at least 2 years behind. On the flip side at maths he was very good. If he liked something then he worked at it, if he didn't his disengagement was total.
At home he was a different person. Sitting in parents evenings trying to reconcile my lovely, bright, articulate, boy with the trouble making, disruptive, figgity pupil teachers described was very hard.
He rarely got merits (or his schools equivalent) and every school play or parents assembly involved me and DH steeling ourselves for tomfoolery and embarrassment - in spite of threats (followed through on) for poor behaviour or treats for good.
In year 5 his teacher called us in. She wanted him referred to a child psychologist with a view that there was something "wrong". ADHD and potentially autism was mentioned.
Obviously we were very concerned but felt strongly (having done our research) and talking to my mum (ex primary teacher) that he wasn't displaying consistently at home and in school traits that fit any specific diagnosis.
As such (and this might not be right for you) we declined any intervention. Quite frankly we felt the root cause was simply a cross between some personality traits and immaturity.
Come year 6 and with a new teacher, plus a strictly applied behavioural reward scheme at home we started to see some big changes.
His writing started to improve as did his overall classroom behaviour. His maths which had always been good got even better and by the end of the year he was just
below expectations in English an top of the school in maths.
We also sat through his last class leaving play for the first time with a video we were proud to watch where he behaved impeccably.
Now in his 2nd year of Grammar he is flourishing. He's top set in English (something I never expected to see) and taking maths top set with yr 9.
Quite frankly 5 years ago I wouldn't have thought this possible. However in his case he simply just "grew up" and whilst he is still stubborn, struggles not to fidget and apt to play class clown at times it isn't part of an all consuming pattern of behaviour.
Telling him you were upset was the right thing to do and how he will learn.
However remember he is still young and there is plenty of time for him to grow and mature. 