We've had two good ones this weekend, both in the same conversation.
First one was:
me: 'You have been treating your brother like dirt today and its totally unacceptable'
ds1: 'Huh? I don't have an opnion on dirt, I don't treat it like anything."
me: "
Yes, well its actually just a phrase people use to mean treating people badly or like they aren't worth anything."
ds1: "But dirt is valuable, without it we wouldn't be able to grow any food."
me: "Erm .... yes ... but I think it means dirt as in grime - dust etc."
ds1: "Oh, well its a stupid phrase - it doesn't make sense."
me: "Right, well, er .... how about you just go and be nicer to your brother"
ds1 "Ok, but I wasn't being nasty.
me: 
Second one was during yet another argument:
me: "ds1, please don't split hairs"
ds1: "I'm not splitting hairs, hairs are too thin to split."
me:
ds1: "Well, that's just silly, why not just say what you mean."
me: "Yes you're right, ds we should say what we mean, but in the meantime, if someone ever says 'don't split hairs' to you again - that's what they mean ok?"
ds1 "OK - but people are weird."
me: "Don't I know it!"
ds1: "What does that mean?
me: "Never mind!!"
.... and I didn't think he was very literal! 
Actually, he seems to be getting more literal as he gets older - but I think in actual fact its just that instead of going away confused, these days he challenges what he doesn't understand.