I find that students who do well in maths challenge tend to think about the problems differently - they look for what has to be solved and how they might find it, what possibilities there are, what patterns they can find in things, etc, and only then try to work out what maths need doing. They sometimes spot the more elegant, inteeresting, faster solutions as a result. Other students tend to look at a question and try to identify what bit of maths they have been taught that they are supposed to do, or how they might be able to solve something in a very long-winded, maybe brute-force, style answer, which can work but might take longer (I do that myself when I see some of the problems, knowing that they could potentially be solved with an algebraic method, for example, but then looking at the solutions and realising that there is a much simpler way).
I think the students who do well on and enjoy maths challenges are often the sort that do enjoy A-level, if they can fill in the missing gaps that are stopping them doing well on their GCSE papers., whereas people who do well on GCSE but don't necessarily enjoy the sort of speculation about methods that you have to do when you see a challenge question, might not enjoy carrying on with maths as much.
If a student isn't doing as well on GCSE exams as expected, it's worth trying out some practice papers and going through the mark scheme to work out why he is only getting 7s or 8s, to spot the weak areas, but also to spot things to do with exam technique - being able to remember all the different methods he has to do something like factoring, for example, or what possible topics there might be associated with triangles? Does he know how to properly simplify algebra? (I find a lot of pupils are not very confident with things like when they can cancel things from top and bottom of a fraction, or they forget how to square something in brackets, or they miss out putting brackets in altogether etc). Does he tend to do the question correctly, but lose marks on the actual calculations? Going through a past paper in detail and working out whether his mistakes are things like doing something mathematically wrong, or just not helpful for a particular question, fixing any mathemtical errors, and working on exam strategy if he just does random steps that aren't actually helpful - making a little plan of how to solve a multi-step problem first, for example, so that he knows what he needs to find and what he will do with it when he finishes that step.