...and I found several emails between him and Z - pretty innocuous I have to say eg - he sent some youtube clips to watch/listen to, she told him what she and her friend had for their meal out.
But these aren't actually innocuous, are they?
They'd be innocuous if they were email exchanges between him and Old Friend X who he's known for some time. They'd be innocuous if they were between him and you. They are not innocuous when they're between Music Teacher and Mother of Pupil. These two people do not have those sorts of exchanges, unless they're also otherwise good friends. Think about it.
I don't doubt that nothing (other than a bit of pining and mooning and taking up too much head space) has gone on yet. Which is why he thinks he is in the right, and that any umbrage you take is out of line.
The point you need to make to him is that his behaviour right now - while it might not have crossed any lines - has all the hallmarks of pre-affair, emotional affair, whatever you want to call it.
And it is THIS that you are unhappy with, and are entitled to be unhappy with it.
They might not have crossed a line yet, but their behaviour has been carried out by a zillion pre-adulterers before them. And if he continues with it, then it's an almost inevitable downhill slide.
What does he, therefore, think he should about it all now?