There's nothing illegal in what you are doing.
The actual legislation says you may leave your house
(b)to take exercise either alone or with other members of their household
That is all, it says nothing about how many times or for how long.
The Guidelines say 'once a day'
one form of exercise a day, for example a run, walk, or cycle - alone or with members of your household
However there are exemptions for various needs
What you're doing us would be 'permitted'
However, I wouldn't take them out any more than they absolutely need to go out. I'd keep the baby at home (plenty of fresh air via open windows) and only take the toddler out when you feel they absolutely need to go out. But that's about their/your exposure, not the 'rules'.