There's an awful lot of misinformation on this thread (although I understand that the other posters are well-meaning in offering their experience).
The legal situation as I understand it is as follows:
Passport - IIRC any person with parental responsibility can apply for a child's passport. Given the ages of your children and assuming their father is named on the birth certificate, that means he could apply for a passport for them as well as you. This means that you do not need your children's father to give consent when applying for the children's passports and as others have said, it is quite usual to put "unknown" on some questions even where the parents are together. If you have any questions, ring the Passport Advice line on 0300 222 0000 - IME they're quite helpful.
Holidays out of the jurisdiction of England and Wales - any party holding parental responsibility must give consent before the child is removed from the jurisdiction. That means, assuming the father has PR, you must obtain his consent before leaving the UK preferably in writing and notarised by a solicitor so that you don't have any problems either leaving the UK or entering France. You've said you have no court orders but if you had a Residence Order then you could remove the children for up to 28 days without his permission. Obviously you'd need to take a copy of the Order (sealed by the court) with you on holiday.
In your shoes, I would assume that written consent is needed. Write to him at his last known address enclosing details of the holiday including flights, accommodation etc. and stressing your intention to return. Send it by first class post but keep a copy of the letter. If you do not receive a written reply within 28 days, I would then make an application to court for a Specific Issue Order and ask the Court to grant you permission to remove the children for the purposes of the Holiday. I might add a request for a Residence Order in order to avoid any confusion in the future although the Court may be unwilling to grant this. Once you receive the papers from the court giving you a hearing date, you serve him by first class post (again at his last known address).
If all goes well, you show up to court, he doesn't and the Order will most likely be granted in his absence. The court will want to ensure you have taken reasonable steps to locate him and of course the history of his relationshiop with the children. If you have any evidence of him running off and showing no interest in their well-being all the better.
Worst case scenario is that he receives the papers, acknowledges service and then shows up to Court to "fight" the application. In your case, I think it likely that you will succeed simply because it's a short holiday with specific departure and return dates as well as the fact that you're leaving the child at home. If you're employed, take proof of employment as it demonstrates a clear intention to return. Ditto if you're a homeowner/on a long term lease.
What he may do is issue a cross application for Contact or Residence and this is something which I think you would be better to face in the short-term rather than long-term. The only other alternative is for you to cancel the holiday and never leave England and Wales with the children until they're adults. My advice would be to face this problem head on and get the matter into court as soon as possible.
Disclaimer - I'm not currently practising family law so my knowledge may be a little out of date. Hopefully Mumblechum, MOS, Collaborate, Babybarrister or one of the other excellent legal MNers will be along shortly to add to/correct anything I am saying. The best thing you could do is visit a solicitor asap to get the ball rolling. You could do the whole thing yourself as I suggested above but getting advice on your position from someone in "real life" who can see the whole picture would be a good starting point.