I think your alterations are reasonable and he doesn't lose out much time in total, just in frequency. I agree completely with your counsellor that any collection/pick up times should be agreed and stuck to. I also think that the collecting and returning should be shared equally between the two of you. i.e. you drop off and he return. When you do collect or drop them off, you have the power to make it as short as possible. Having said that, if you would would prefer to do both the dropping off and collecting then do what works best for you.
If you are taking them to him, then when he opens the door, just tell the kids that you will see them soon and to have a nice time, give them a kiss and leave. If he is bringing them to your house, you don't have to invite him in. You can just open the door, say hello to your children, and say to him see you on -- and bye. If you are collecting them, you say Hi to your children, ask them to say bye to their father and off you go.
After all, the contact arrangemts are for the children and should work to their needs. You know them best and you will find the strenth to get the control back. Any discussions that you need to have about the children can be done via email and never in front of the kids. If he starts to try to have a conversation with you about the children, you just say please send me an email about it. Nice and poilte and off you go. If he starts to make conversation about something else, you just say - Got to go then, bye.
I hope that you make the changes you need in your life, nobody should be ignored and you are most definitely not irrelavant.