Women are delivering nutritionally balanced food that does not contain unknown chemicals and delivers enough, usually, for that infant's full requirements. They are feeding their child, that is their number one priority. To be fair, if a woman was using their child as a sexual encounter as the focus of their breastfeeding, then I would be very concerned with that as well.
They might find it stimulating. However, I remember being so fucking tired from getting up on demand that there was no way that I was even thinking erotic thoughts. Or maybe that slips people's mind, particularly if they had a straight forward delivery and one of those magic children that sleep through early. I mean the mastitis and the painful exploding breasts (again, not something a male will get because they produce so little milk).... yeah... that was really erotic stuff.
But hey ho! A rare few women find it 'erotic' so therefore it is perfectly fine for any male to use an infant for their own needs.
I am not sure people ever think this through.