It is a tough decision, a real leap of faith. There are so many issues and concerns and different advice...
So just a few more thoughts. Firstly, and honestly, how involved do you want his Dad to be? A bit of a visitor, able to cope with a few hours here and there, but with an entirely separate lefr, as it is now? Or an actual parent, someone your DS can rely on and turn to whatever happens?
Clearly, bf raises difficulties. As you say, you won't be bf forever. So you need a plan. Even if you want to go on bf until DS starts school you could express. It wouldn't be impossible.
So back to square one. How much does the fact that ex has now moved in with someone else impact on DS? Will he mind? Is it logical that or beneficial to DS that Daddy is required to move out of his home in order to have the first and second visit 'alone' ? Well, probably not, and you certainly don't sound remotely like you are bothered by the fact that your ex has a new person in his life. All credit to you.
Does your child need protecting from his Daddy? Would he be horribly, irreparably traumatised by being put to bed by someone else? Will this change dramatically over time? Could you prepare him for this by allowing his Granma to put him to bed? If you were suddenly to need to go into hospital for an operation and you were going to have to be in for a few nights, how long would you put the operation off because your Ds's routine demanded that you, and only you, put him to bed?
It's all a question of priorities. If you were still together, DS would have had the benefit of overnight contact with his Daddy from the moment you came home from hospital. What is he gaining from not having that level of involvement with both his parents?