We're in the same position - except that it's egg donation that we need - and, like you, have decided against adoption.
For me, it's several things:
the encouragement of "open" adoptions, I just don't want to have to have contact with the birth family, it would make me feel like my child wasn't really mine, particularly as my child would likely (assuming that we'd be unlikely to be given a baby) have memories of them
the fact that absolutely everyone would know that our child wasn't biologically ours from the start - of course, we would tell our child about the egg donation, but it would then be their choice who else to tell, it wouldn't be something that everyone knew by default. I like the idea of it being something that is private but that the child can reveal to people that they trust on their own schedule
the fact that an adopted child would have potentially quite a lot of formative experiences that we were no part of - of course, any child has lots of experiences away from you but, with an adopted child, potentially we're talking about years that you just don't know much about
on a related note, most adopted children have had some really bad experiences in their childhood, attachment issues/behavioural issues, often disabilities of various sorts. My DH and I aren't well equipped to parent that kind of child - we both work full-time in demanding jobs and intend to continue with that. I think, with a baby that we conceive and raise from birth, that should be fine but a child that we adopted would likely need more focussed time and attention from us.
I don't feel that, as someone with fertility issues, I have more of a 'duty' to adopt than anyone else. In fact, in some ways, I think people who are already parents are probably much better equipped to do so. I would consider, later in life, when I've started to wind down my career doing fostering or other work to support children in care but adoption isn't for me.