My son had just turned 3 when our first foster child arrived. I foster children older than him, although princesspeppa is right in that it's generally advised that foster children are younger than birth children.
Fostering has a massive impact on birth children, impacts everything really...daily routines, atmosphere in the house, how you spend your time, divided attentions, emotional impact when the FC leaves, etc. etc. etc. I don't think you can really underestimate the impact. But, for me the positives of it outweigh any negatives. I think it's brought out some really good qualities in my son and shown me that he's an incredibly adaptable, accepting and welcoming child. It also occasionally brings out some not so good qualities...he's bitten a FC before, refused to share him books with one (shared all his toys, but was quite territorial about his books!)...different with each child we've had but I'm very proud of how he's just getting on with being part of a fostering family and really just accepting these children into his life. I think this may be an advantage of starting when he was so young?
Personally, I think it would have been more difficult for him if I'd fostered babies. He's the baby of the family, and a baby FC would have taken more attention away from him and their needs would have been very similar (nappies, bathtime, help eating, etc.) and I think that would be more problematic than having older children whose needs are quite different to his. Less competition/jealousy that way, I feel.
FCs' behaviour towards him varies from ignoring to being downright mean to tolerating to being lovely to him...impossible to predict. My son's quite robust, not too sensitive, I think that helps massively.
It's a lot to take on, and my son's well-being is obviously the most important thing to me, so you've lots to think about and consider when deciding whether it's right for your family.
If I were you, I'd go along to your LA's information sessions for both fostering and adoption. You can go now even if you know you won't be doing it for a few years, just to get a feel, listen to some carers/adopters...might answer some of your questions.