Hi Meadowquark,
Please do talk to him first... The reason why I say 'please do' and feel strongly about it, is that I have an au pair agency and I have a number of au pairs saying that they are looking for another family because the one they found through au pair world has given them notice (or even kick them out to move out the next day). They cry because they don't know what they did wrong and suddenly, they are facing being on the street or returning home without having an idea what they did wrong, and feeling bitter about their dream country.
If you talk to him, you will give him a chance to improve and engage more with your boys.
Some years back I was an au pair myself and couldn't understand the language much, which made me nervous about how I should engage with the children. Coming here is a change for the au pair, different culture, different way of running the house, there is lots to observe and adjust. Plus the language. I can see that many au pairs feel very tired the first couple of weeks and so was I. Unless you experience being an au pair, it is hard to imagine the pressure. Even though that many au pairs try to smile and be fine, inside there is loads they have to deal with and adjust to. And this may/may not impact on their daily performance.
On the other hand, and to be objective, I think it is possible to observe (within the first week already) if he shows any sign of trying to connect with your boys. Does he smile at them, or is he generally shy (with you as well)? Encourage him to do activities with the boys, organise it for them and let them enjoy it. Help them connect.
I can understand if you see in him your husband's behavioural patterns, it is harder to give him a chance, because it would be easier for you just to let him go.
On a separate note... I have a friend working in HR and they apply long procedures of warnings, documented chats and more warnings, before they let someone go. In their case, they maybe take too long doing it by the book, but in general I do believe in talking to people first and giving them a chance.
I would suggest that you talk to him, give him one week to improve and if he doesn't, give him the agreed two week notice. That will give you a comfortable time to start looking for a replacement, should you need one.
I hope this helps. Sorry for the long message... If you wanted to read more tips about helping an au pair settle in, here is an article I wrote: www.youplusagency.com/#!blogger-feed/c1x46/post/8452779528314570030
Good luck!
Katerina