I'd agree with others, that there is lots of risk. The further you get from young, white, healthy baby the less 'in demand' a child will be (and yes, I have heard girls are more saught after and there are more boys in the system).
So being 'less in demand' makes a child harder to place. But it doesn't mean more risks and necessarily more issues.
Things to factor in 'for risk' to me would be lots of moves in foster care, parents with mental health issues that may be hereditary, potential foetal alcohol, autism in family etc.
But of course people have wonderfully and successfully parented children with some/all these 'risk' factors.
My birth dd is most likely somewhere on the autistic spectrum. If I had known when expecting her I may have been nervous but I love her to bits, as you would expect!
Personally, I'd look at what risk factors you and your partner can handle and go with that.
LAs will place their easier to place children first and any voluntary agencies etc would then be able to try and place other children and (if I have understood this rightly) any agency outside the LA who places a child would be paid by that LA.
So your SW is correct, Linkmaker and any organisations outside an LA will most likely have harder to place children. But if you are willing to consider older children, sibling groups or different/mixed heritage children you may find a perfect match that way.
We were happy to consider a mixed heritage child but her social workers didn't want us for that particular child. So it is not as simple as being open to this child or another. I know at least one couple keen to adopt a sibling group but they were only approved for one.
Anyway, all the very best.