Secondary schools are oversubscribed currently due to the high birth rates from 2007 onwards. However they fell a lot in more recent years so over subscription unlikely to be as much of a problem for your kids who are still young.
So from your point of view, secondary private school admission generally doesn't depend on catchment (but each school is different so this might not be universally true) and they have done away with priority based on waiting list. This is good news as you should be able to figure out which schools you want to apply to when the time comes rather than worrying about it now.
You need to think more about primary admission, which does depend on your address. You apply directly to the school, there is not centralised admission. In Dublin, if your child is born after about March or April they typically start the September after their 5th birthday, otherwise when they are 4. (legally can go any time between 4 and 6, no one "defers" it is purely parental choice whether to start at 4 or 5, but primary may prioritise 5 year olds when allocating places)
Most kids go to state primary schools although there are a few private ones.
If you want a private secondary school, your best bet is to live in south Dublin as this is where most is them are located. However the suburbs you mentioned are all north Dublin so your private school choice is more limited.
If you want to live in Castleknock - look at Kings Hospital. If you live in Clontarf - Belvedere, St Michaels, Loreto on the Green. (Many/most girls in Clontarf go to Holy Faith Clontarf which is a state school).
Malahide/Sutton/Howth then Sutton Park or commute into Dublin City centre for Belvedere/Loreto or further to any of the south Dublin schools. As the PP said, many Malahide/Sutton/Howth kids go to state schools which are all pretty good.
If I were you, I'd just focus on where you want to live, while keeping an eye on the primary school admission process.
Good luck with it all!