I have a very similar situation to the OP, my ex basically turns up late, cancels at the last minute, refuses to have them if they have runny noses (its "too difficult when they are ill", and recently has refused to take the youngest as he is potty training and that is "too much hassle") 
Not living in the UK, I dont have the option of going to mediation (I wonder if I would do this? It's basically would be like giving them an ultimatum wouldnt it? 'shape up or dont bother at all')
However when I first split up with my ex, I was given (IMO) a great bit of advice from a counsellor I saw at the time, she said to realise that I cant make ex be a perfect parent, and shouldnt try to, I shouldnt "cover" for him (ie says things like "I am sure daddy is stuck in traffic that's why he is late"etc) and that one day my kids would realise for themselves what he is like. Obviously that wont be nice, and I am prepared to be there for them, but it seems right that they should get an honest picture and that none of us should have false expectations. He is late, untrustworthy, unreliable etc, and when he demonstrates it I dont cover for him, just comfort my kids and let them know that its normal to feel disappointed or angry if someone treats you like that.
I sincerely hope they wont grow up treating people like this (both boys)
..and at the risk of spinning this thread off on a tangent about sexism/men against women/ my opinion is that it is not mothers who make boys turn into sexist men, it is other men who tell boys/men that it is OK to disrespect women.