Hello 
You've got quite a long while to think about it, because agencies are very unlikely to consider you until your youngest child is at least 3, and with many agencies it will be 4, some it will be 5. No agency will take on a couple whose youngest child is 1 because approval doesn't actually take very long (can be ~6 months or a few months more) and they need a 2 year age gap between your youngest child and any new child. Enigma is very unusual IME with being accepted with a younger BC, I rarely hear of other adopters in the same situation.
By the time you do come to apply, agencies will be looking for a couple who are both very committed to adoption, so your DH wanting to start ahead of when you are really ready to committ and feel comfortable with adoption, is also a non starter.
So you have time to think and research as much as possible about what adoption is like now, and what it might mean for you as a family, and hopefully a couple of years and more information will give you more clarity and enable you to work out whether adoption is something that you want.
I would say that concerns about the effect on your existing children are shared by everyone who is adopting after BC, or adopting after adopting. I never stopped having thoughts or concerns about that, I got to a point where I felt it would probably work out and I felt comfortable with going ahead. And my existing kid/s were comfortable with going ahead too, they were older and it was a family decision to adopt again.
What is concering you about birth family ties? Is it your BC's feelings, or you wondering if an AC would love their birth siblings more than their adoptive siblings? My 3 children have birth siblings as well as each other, and they have varying degrees of contact, ranging from nothing to letters, to visiting each other a few times a year, and my DD2 is living with one of her sisters right now. It really depends on the child and their siblings, there's no way to predict how a relationship will turn out. My DS is closest to DD1, and they have no blood connection at all. DS and DD2 are half siblings by birth and they love each other very much but their relationship is fraught with difficulties. DD1 counts all of her siblings as equally her siblings no matter how they became that, on the other hand DS says he only has two siblings, DD1 and DD2, and doesn't count his other birth siblings. IME it's not about a birth or adoptive connection, it's largely about how their personalities fit together! And about growing up together. My 3 all love each other fully as their brother and sisters, regardless of whether they are related by birth. That doesn't mean they get on with each other, but there you go!
I would say that the most children available for adoption have additional needs of some description, which are very often emotional or behavioural needs, because of what they've been through. I think to adopt you should be comfortable with the idea of parenting a child with some additional needs and also with adjusting your parenting style as necessary because of those needs. Some children also have diagnosed special needs. As the others said, you can choose whether or not to go forwards with specific special needs. But there is obviously uncertainty, especially with the youngest children.