Hmm this will depend from agency to agency. But in my experience it has been the other way round - SWs have tended to use the expression 'just a baby' and implying that things would be easy, minimising any potential problems. As in, 'we'll cut introductions short but don't worry, it's just a baby' or 'a month further delay and a potential further move aren't really concerning, it's just a baby'.
Accordingly I think some SWs/agencies may have a tendency to dismiss problems that children may have who have been adopted very young, whilst acknowledging that older children may be severely traumatised; and hence if anything, would look for a stronger support network for prospective adopters of older children.
But more importantly, they will be looking for experience with children of the age range you are interested in. If you have no experience whatsoever of caring for babies, but e.g. work at a primary school and hence have lots of experience of that age; I think you'd find it hard to have a baby placed with you. Your assessing agency would probably require you to gain some relevant experience first. Vice versa, if you are a nursery nurse and have lots of experience of babies, but have never really had much to do with school age children, you'd probably wouldn't be easily considered for 4+ year olds/would be asked to acquire some relevant experience first.
Going back to the support network, this is something many prospective adopters seem to worry about (we did too) but I've never heard of anyone being rejected on those grounds. At most, I can imagine that if your support network isn't strong, this will influence the assessing SW's judgement of what degree of additional needs you might be able to cope with. But though some local support is valuable; if you have close friends who you can offload to if needed and who support you emotionally, but just don't happen to live nearby - this counts too!