"Also, globally, the UK does incredibly badly at maths and science on all the objective comparisons. This is a tragedy and has to be due to something."
I think there are many, many factors which lead to this, for maths (can't speak for science).
First off, it isn't cool to like maths, and maths leads to boring jobs like accountancy rather than cool jobs like (weirdly popular dream job with my students) marine biologist. Parents will happily take their children to museums for culture or make them watch nature documentaries, but talk to them about maths?
Secondly, it is socially acceptable to be rubbish at maths. So many people say 'I was rubbish at maths at school' to me at parents' evening in front of their kids, or 'No point in asking me to help with his homework'.
Thirdly, we will never compete with Asia educationally while their schools are backed up by exceptionally academically driven parents - good old Tiger Mother and our schools are plagued by parents who complain when homework interferes with extra-curricular activities and don't back up the school with sanctions for poor behaviour. There's a thread on here somewhere with parents complaining that schools expect them to teach their kids too.
Fourthly, our curriculum teaches and tests maths in discrete little units. The kids do areas of triangles and do a test on it. They do Pythagoras and have another test. They know that a question is a Pythagoras question because that's what they've just learned and it's a question that says 'here's a triangle, find the length of x'. They are used to little 2 mark questions where the question tells them exactly what they are expected to do to work out the answer. This is why I confidently predict that on the maths paper my foundation kids sat in November, none of them will have got the surprising 4 mark question which had a picture of a triangle with the hypotenuse and one of the sides and asked them to work out the area. Because it was an area question, it won't have occurred to any of them they they might need Pythagoras.
The 2 mark scaffolded questions are how the exams have been for some time now (the exam boards are to blame for this I think), although there are now measures being taken to fix this with the new specification - hence Edexcel's effort with the triangle question. I will be very interested to see what happens with the grade boundaries - if they want to make the exams more rigorous and challenging, then they must expect a dip in results if the change is to have value.
The way we chug through the curriculum is down in part to the drive to get as many students a C as possible - and in the past, like I said, there has been no need to prepare them for in depth questions so we haven't (league tables are king, teach to the test).
The brighter students for whom a C is guaranteed still chug through, instead of using the extra time to develop their deep mathematical thinking skills, we tack on another GCSE in statistics. Then they get to A-level and flounder because the questions don't tell them exactly what to do.
With the addition of functional maths and longer questions to the exam paper, it will take a while for the changes to bed down, but I know that in my teaching, I don't have time to take a week or 2 to do some in-depth problem solving because I've got to get them through percentages and fractions before the next test. And I'm also reluctant to do the in-depth work because a) I haven't been trained how to do it well and b) the kids are generally scared of any question they can't answer immediately, and give up, causing behaviour problems.
Fifthly, the primary school maths curriculum appears to be full of stuff that could probably wait until secondary school and short on the repetition repetition repetition and more repetition of basic numerical methods that we want them to have as second nature. When they get to Y7, the amount that can't remember how to do long multiplication or even how to borrow correctly when doing subtraction is quite frustrating. And those who only know 'chunking' and not the 'bus stop' method of division then come unstuck when we start dividing decimals. Not blaming primary school teachers, they are as bound by tests and league tables as we are in secondary.
Anyway, that was quite long. Sorry about that, although I could probably add much more :)