I am about to change area. I qualified into private client in a top 50 regional firm back in the days of the economic boom. I then moved to London when I was 18 months PQE and switched to pensions, joining a magic circle firm.
At that time, they would more or less take anyone with a pulse to do pensions. There was so much work and virtually no experienced candidates.
I stayed there for a few years before moving back to the north west into a regional firm to do pensions, then more or less immediately got pregnant and am shortly due back after mat leave.
I have just been offered a job back in private client, which I am enormously relieved about. I never enjoyed pensions.
Anyway, I guess what you can take from this is that it is possible, but you will need a much better recruiter and you will have to be patient and realistic. What area are you trying to move into? Is it very popular? Is there a similar area that you could use as a stepping stone (e.g. if you want to do comm lit, could you do fraud/insurance or something where there is high demand?)
What do you do now and why do you want to switch?
I was quite fortunate that the two areas I have done are quite similar in many ways (technical, non-contentious, lots of drafting etc) - you may struggle more if you want to switch to an oversubscribed area of law that is utterly different from what you do now, but honestly if you have good academics and experience in a top 40 firm plus relevant experience in the area you want to switch to, the key thing you need is an excellent recruiter who can get you in front of an interview panel.
The recruiter I used (both times) is someone I know quite well through networking events etc. Think it helped that she had met me in person and felt able to 'sell' me. Have you got a recruiter who is taking you seriously? It may be worth picking one and meeting them face to face and keeping in touch with them regularly.
I am taking a pay cut and demotion to switch areas again (got a pay rise last time due to moving to a magic!) and this was a concern for the firm I am moving to, but I was able to discuss this with them at interview and deal with it.
I don't think that switching is impossible, but you do need someone on your side to get an interview. What area are you in? I know of one recruiter in London who helped a friend of mine switch practice areas when no other recruiter would even put her forward for roles in the area she wanted, and the one I used (and really rate) is based in Manchester. Happy to pass on the details if you're interested. As I said, though, you would be best advised to go and meet them face to face.