So, he wants you to tell him what ended the relationship BUT he wants this to happen HIS way, not your way....
You could write a list, with the reasons listed in bullet form, and send it to him. You are under no obligation to put yourself in a situation where you are uncomfortable and feeling under pressure, tongue-tied, inarticulate or unable to express yourself because of fear or nerves or hurting him. So I would recommend a simple list if you choose this option, and include his unreasonable demand that you tell him face to face why you left down near the bottom.
I think your idea to get it all out in Relate is a good one, because you would have a counsellor there to direct and referee the meeting, and give you a feeling of security. Your ex seems sadly unaware of how his demeanour affects you if he thinks shooting down a very reasonable and sensible suggestion without so much as a minute's reflection, and insisting on having this out on his terms is OK.
If you ever have doubts about whether you've done the right thing, try to remember this telling little incident you have described -- he is apparently sad that he has lost his partner, his child and soon his house too, yet he continues to want things his way. He is still trying to bully you even though bullying has resulted in him losing everything he says he holds dear.
Some of this recent communication of his is a pathetic and very poorly executed attempt to get you back so he won't lose the house for the immediate future, imo. He is also getting used to his new circumstances as you observe, and no doubt he is missing things like meals and laundry, and having someone to play mind games with, plus all that undiluted and unconditional affection from your LO of course he feels sorry for himself, and he will try to get all that back. You have stood up to him and he is trying to undermine your resolve with his demand for an explanation this is a knee-jerk response from him. He just wants a chance to see you face to face and intimidate you or put on some act that will tug at your heart-strings (Valentines Day is coming up, after all...) -- either way he wants things back as they were, on his terms.
It's a terrific sign that you don't miss him, and I believe the more time elapses the less you'll feel guilty too, especially as you see a change coming about in your LO as he begins to feel more secure. Hold your ground on the Relate issue, or write him a brief letter. Include in your remarks that your decision is final, and hopefully he will get the message.