OP YANBU.
It's not an emergency or a one off favour you might one day need returned , it's a three day commitment every week to someone else's child and someone else's work schedule.
What happens on the mornings they suddenly need to leave very early for, because they have an early meeting miles away, or the extra day they have been asked to work that week because someone else is off?
You'll suddenly find yourself expected to have the child at your house for breakfast two hours before school starts rather than waiting at the gate for you to collect as you walk by.
Or if they are running late one morning when your children need to be there early?
It doesn't matter how many other people have arrangements like this or how well it might work for them, you don't want to do it and it's not fair to expect you to take on a commitment you will come to resent, just because it works well for someone else. Especially when there's a fourth child lurking in the background for the following year.
DS is coming to the end of his first year in F1. One of the mothers of another boy in his class made a big point of getting friendly with us and at first it was nice to have someone as a back-up if there was a problem.
I brought her DS home for lunch one day when she was ill, she offered to pick mine up when I thought I might be late back from an appointment.
Then she started to shout to me as we were passing her house, asking me to take him in with DS. It wasn't a big problem at first but it became a daily thing and soon she was just sending him out and I wouldn't even see her.
The teachers started to comment to me about it as they wanted to have a word with her. And it was a bit of a hassle on wet days to get two children out of wellies and waterproofs in an overcrowded cloakroom, put everything on the right pegs and shelves, find diaries and drinks and snacks and get them both in on time, just because she didn't fancy coming out in the rain. And then later on I'd get a text "can you pick X up and drop him off on your way passed?" So it would be the other way then, get two boys, find waterproofs and wellies and drinks and snack boxes and get them ready and out.
And her son is hard work. He's one of those who will run off or misbehave and then say "you can't tell me off, you can't tell me what to do." Or worse "I can hear you talking but I don't know what you're saying, it's just blah blah blah to me." 
Then she got a part-time job and just asked outright "Which days are you going to have X after school? His grandma has said she can do Friday so if you do Tuesday and Thursday I will ask Friend 1 to do Monday and Friend 2 to have him on Wednesday. Can you walk him home at 2pm?"
She was a bit shocked when people were not happy about being told they were now her unpaid regular childcare and it all fell through after a week or so.
I had to be quite blunt and say that having him twice a week every week was not going to work for us and that I would take him occasionally but would have to let her know when it was possible rather than commit to a regular day.
DS has other friends he likes to invite home and having this one boy twice a week and working one of the other days myself meant that it was hard to invite anyone else over to play for a start.
Stick to your guns OP. Say no and don't be pressured into giving reasons or excuses or doing it for just one day a week or anything like that.
If they have a younger child they must already have some sort of childcare in place, I'm sure they will be able to organise a childminder or breakfast club for the older one.