I wouldn't say that you can't go in because you have to drop your DD at school. I don't think we women should explain childcare issues - particularly to male colleagues. It can weaken our position. Just say that you're afraid it's not possible.
Whilst he (she?) probably shouldn't have called you at home on a Sunday evening (especially as he doesn't officially have your number, which means it's not a generally accepted practice), it was also very unprofessional of you to yell at him. What sort of working relationship do you have, that this would be remotely appropriate behaviour?
You should probably explain (by way of a carefully-worded 'apology') but try to avoid going into detail about the nightmare with DD and bedtime. Just say he caught you at a bad moment and you assumed it was urgent.