Active discussions

Meet the Other Phone. A phone that grows with your child.

Meet the Other Phone.
A phone that grows with your child.

Buy now

Please or to access all these features

Talk
Chat

Join the discussion and chat with other Mumsnetters about everyday life, relationships and parenting.

Original poster

Anyone work in risk & DR looking at Heathrow today

33 replies

toooldforbrat · 21/03/2025 10:41

Properly funded DR is never necessary til it is!

i see some insurers suggesting they won’t pay due it being an unforeseen event, but it’s got to have been very known there was reliance on 1 substation & a consequent decision not to invest.

OP posts:
MidnightMillie · 21/03/2025 10:46

What does DR stand for?

Sorry, Google's not helping me.

BestIsWest · 21/03/2025 10:47

Disaster Recovery.

sunbum · 21/03/2025 10:50

yep, i work in IT on large datacentre storage and backup projects. I find it, frankly, unbelievable that the whole of LHR does not have dual, or even triple power supplies, never mind generator backup. All their systems will be houses in datacentres somewhere amd there are no DC without dual power supplies. I dont believe we are being told the full story tbh. I think you'd have to engage in planned sabotage at more than one site to bring this about.

sunbum · 21/03/2025 10:51

and yes, exactly, why hasnt DR kicked in?

MidnightMillie · 21/03/2025 11:05

Thank you @BestIsWest 👍

Littleoakhorn · 21/03/2025 11:05

A single point of failure for an airport? Seems unbelievable.

InWalksBarberalla · 21/03/2025 11:11

I suspect that there were maintenance works being undertaken in a related part of the system and hence this substation was more heavily loaded and redundancy was reduced. Possibly someone stuffed up some switching or something. The amount of forced outages that occur during planned outages is quite large - things seem to tick along quite well until people start fixing things.

wonderstuff · 21/03/2025 11:12

Seems odd that they’ve been able to restore power to the local houses but not Heathrow? Also that they decided so early to close for the whole day. Ed Miliband said there’s never been a fire at a substation like this. Times report Counter terrorism police investigating.. Definitely more to this than being reported right? Feels like Russias style?

Original poster
toooldforbrat · 21/03/2025 12:41

sunbum · 21/03/2025 10:50

yep, i work in IT on large datacentre storage and backup projects. I find it, frankly, unbelievable that the whole of LHR does not have dual, or even triple power supplies, never mind generator backup. All their systems will be houses in datacentres somewhere amd there are no DC without dual power supplies. I dont believe we are being told the full story tbh. I think you'd have to engage in planned sabotage at more than one site to bring this about.

Same here, frequently audit data centres , and look at power supply resiliency and back up generators.

i recall Heathrow losing a data centre many years ago with an energy outage due to an incorrect power supply being switched off and UPS not kicking in.

the reason will be interesting.

OP posts:
Mrsbloggz · 21/03/2025 13:01

Littleoakhorn · 21/03/2025 11:05

A single point of failure for an airport? Seems unbelievable.

I agree!

TwoLeftSocksWithHoles · 21/03/2025 13:05

@sunbum yes I agree,

I notice there have been a few incidents recently (e.g. Virgin, BA, Lloyds, HSBC) where they blame an 'upgrade' as introducing a fault and knocking out their entire systems across the world.

I cannot believe that they would implement upgrades that can have such serious effects on their business (and the country's economy) without rigorous prior testing.
I suppose alternatively they may not be declaring the true cause?

wherearemypastnames · 21/03/2025 13:07

The substation and a back up generator both failed ?

Clearinguptheclutter · 21/03/2025 13:16

wherearemypastnames · 21/03/2025 13:07

The substation and a back up generator both failed ?

Just reading that

ejther something very fishy indeed or daft decision to put them in the same place

wherearemypastnames · 21/03/2025 13:20

Or just poor routine maintenance

Fuuuuuckit · 21/03/2025 13:24

Says on bbc news that counter-terrorist police are part of the investigation. Might not just be the fact that power is out in one generating station.

Believe me, they wouldn't shut Heathrow without bloody good reason. Shit if you're stuck in it though.

wherearemypastnames · 21/03/2025 13:40

sounds like they didn’t have big enough supplies of diesel / big enough generators for anything but a short term outage .. cost saving

DazzlingCuckoos · 21/03/2025 13:48

Apparently the backup generator was on the same site.

It also caught fire...!

Going back to the old days of backup tapes for servers, it's like taking a backup of your data and then keeping it next to the server that's backed up!

HowardTJMoon · 21/03/2025 13:57

Even if the gennies kept the datacentres, control tower and radar up that's not enough to keep the airport as a whole open. There's the (truly vast!) baggage handling system, the check-in desks and security scanners, escalators, lifts, jet bridges, interior lighting, digital signage, tannoys etc etc etc. You need all those and a thousand other things up and running before you can safely have thousands of members of the public wandering around terminals that big.

JumpingPumpkin · 21/03/2025 15:50

Yeah, you’d think they’d have considered something like a fire taking the power down. Even just to keep say 20% still working.

sunbum · 21/03/2025 15:57

HowardTJMoon · 21/03/2025 13:57

Even if the gennies kept the datacentres, control tower and radar up that's not enough to keep the airport as a whole open. There's the (truly vast!) baggage handling system, the check-in desks and security scanners, escalators, lifts, jet bridges, interior lighting, digital signage, tannoys etc etc etc. You need all those and a thousand other things up and running before you can safely have thousands of members of the public wandering around terminals that big.

Yep, and all would have been designed with dual power supplies connected to at least 2 different sub stations. Something more than a fire at one sub station gone wrong here.

FatherFrosty · 21/03/2025 16:01

It appears there was the substation, then a backup to that (on the same site) and then a generator (unless I’ve missed it, they’ve not said if it’s national grid or Heathrow’s)
so two backups.

They all failed

HowardTJMoon · 21/03/2025 16:21

sunbum · 21/03/2025 15:57

Yep, and all would have been designed with dual power supplies connected to at least 2 different sub stations. Something more than a fire at one sub station gone wrong here.

Show quote history

There was an incident 10 or 15 years ago where a chunk of the London underground went dark because although it was powered from two different sub stations, one failed and then the other tripped out due to the sudden extra load...

Having feeds from two sub stations gives some extra protection but it's not a complete answer unless you can supply dual power to every single piece of equipment. That's feasible for a datacentre but not an entire airport.

HowardTJMoon · 21/03/2025 16:25

JumpingPumpkin · 21/03/2025 15:50

Yeah, you’d think they’d have considered something like a fire taking the power down. Even just to keep say 20% still working.

Which 20% would be enough to keep the airport open? It's not simply a matter of getting the aircraft off the ground and back down again. You have to make sure that the thousands of staff and tens of thousands of the public can move around in safety, be properly security checked, and that their luggage goes with them on the right aircraft at the right time. If they couldn't guarantee every single one of those criteria then I can't see how they could keep the airport open.

Myengagementring · 21/03/2025 16:27

TwoLeftSocksWithHoles · 21/03/2025 13:05

@sunbum yes I agree,

I notice there have been a few incidents recently (e.g. Virgin, BA, Lloyds, HSBC) where they blame an 'upgrade' as introducing a fault and knocking out their entire systems across the world.

I cannot believe that they would implement upgrades that can have such serious effects on their business (and the country's economy) without rigorous prior testing.
I suppose alternatively they may not be declaring the true cause?

I work for one of those banks and upgrades do cause issues even with rigorous testing. There can be an unknown environmental factor at play, it could all be find until customers use it and it doesn't like being at load, sometimes production isn't 100% like for like with the test system and then an issue appears that didn't in testing.

TheDandyKhakiDuck · 21/03/2025 16:28

TwoLeftSocksWithHoles · 21/03/2025 13:05

@sunbum yes I agree,

I notice there have been a few incidents recently (e.g. Virgin, BA, Lloyds, HSBC) where they blame an 'upgrade' as introducing a fault and knocking out their entire systems across the world.

I cannot believe that they would implement upgrades that can have such serious effects on their business (and the country's economy) without rigorous prior testing.
I suppose alternatively they may not be declaring the true cause?

It happens… TSB got a big fine from the FCA for huge problems after upgrading a platform in the last couple of years