Most important - definitely check references. Don't assume anything. Ask the other mums lots and lots of questions.
Ask exactly what they do with a child of your age group every day.
Are they self-motivating or will you have to find classes/activities?
What food do they eat? Do they understand good nutrition? Will they be feeding your LO McDonalds every day?
Ask how many times they have been off sick in the last year.
Ask if they have any holidays planned.
What would they do in an emergency? Do they have a first aid certificate? Eg - your child hit their head, they seem ok, but what would you do? What signs would you look out for? Do you know the emergency number to dial?
Do they know other nannies for playdates?
Get a feel if this job is a means to an end (eg for travel) or if they genuinely love children.
If they are live in check their living habits - clean/tidy - boyfriends - eating habits / cigarettes/alcohol.
Do they have a drivers licence? No uk residents are difficult to insure - would they change their licence for a British one?