I was a nursery nurse for years, until I decided I needed something else. My reasons were similar, a need to get out of the structure and team of the nursery, but also because I was stuck in minimum wage jobs, living at home with no chance of getting out.
I looked into nannying but found firstly it was practically unheard of in Cornwall, where I'm from. No one can afford a nanny, work is scarce and low paid. So I first looked at area, London is obviously the biggest place for nannying so I looked for work there. I set up profiles on nannyjob, greatcare, childcare.co.uk, and gumtree and tried agencies. None were willing to take me on though, for the same reason you mention.
In the end I was contacted by a British family in Brussels who needed an au pair but were having difficulties. In the end I compromised, a took a short term position with them for between nanny and au pair wages doing a mixed job, not quite either. This gave me the most important thing, a reference. Honestly the best way to get work is experience and references. If you don't yet have the experience, get as many refs as you can! From babysitting, volunteering, anything.
Things to consider- nannying is a bit more lonely than in a nursery. I know people who prefer the team work. I don't! I make friends with nannies and parents at groups and libraries and parks.
The hours tend to be long, as a lot of nanny jobs cover not only the time the parents work but also their commute. But that depends on the job, there are part time roles.
Some parents try to take advantage as there is no HR, no management team etc. I know nannies who've been told they have to be self employed, which is very rarely possible. Or paid cash in hand.
You may come across the Ofsted registration being touted as a good thing, that nannies must have. Not true. The only benefit is to the employer, as they can use childcare vouchers. It's not like in schools and nurseries, they don't inspect your practice or anything. It's just for the vouchers and doesn't imply you are good, bad, whatever, unlike childminding.