Anyone work in risk & DR looking at Heathrow today

[toooldforbrat]

Properly funded DR is never necessary til it is!

i see some insurers suggesting they won’t pay due it being an unforeseen event, but it’s got to have been very known there was reliance on 1 substation & a consequent decision not to invest.

I've worked in IT since the late 80s. The amount of problems that have their root cause in an update that's gone wrong is staggering. Modern computer systems rely on interactions between multiple different services running on numerous virtualised hosts across complicated networks. They're fundamentally fragile and often being run by overworked staff with inconsistent funding and a much bigger push from management to rollout new features than to fix technical debt.

Just yesterday we lost incoming mail because Microsoft pushed an update to its hosted Exchange service and they did an oopsie. And let's not forget CrowdStrike last year.

Absolutely what @HowardTJMoon said and the amount of time something is running on out of support software and can’t be upgraded because it depends on something else that uses a package that can’t be rewritten because no one knows how or there isn’t the resource because everyone is working on the funky new stuff and everyone who knew how to write that particular language on that database is retired. So many inter-dependencies.

Completely agree on the big IT systems , as a consultant for the last 30 years worked on many.

lack of investment is always the biggest factor - no one wants to spend money on properly resiliency ( except once when it was mandated by the regulators) proper modernisation and skilled staff.

It would need to be a huge generator. Think turbine hall at Battersea. Supposedly back up but all located on the same site because that's where the connections are. The substation uses a transformer that uses 25000 litres of cooling oil and the Fire Brigade were battling a fire fed by that. Water, high voltage electricity and oil are not a good mix.

That's reassuring, as that FCA must have established that the TSB did not apply sound financial principles to their banking approach and so are culpable.

It will be interesting to see if the FCA fine the others similarly, unless it transpires that the cause was not a result of the bank's negligence.

@TwoLeftSocksWithHoles yep it was something to do with a lack of oversight controls over the outsourced company in TSB’s case. Obv the FCA is a financial services regulator so not relevant with the Heathrow situation, but it just shows things can go incredibly wrong even with big credible companies who have risk and audit functions.

Is it because the owners arnt apparently in the UK and perhaps they are less invested in it excuse the strange pun?

The FCA fined TSB because the issues took 8 months to be fully resolved for all customers, some customers were locked out of systems and some were given confidential information on other customers. The FCA discovered the project didn't have the the correct governance in place and they didn't have adequate risk management systems. It is worth noting it wasn't an upgrade, they were moving all their customers from an old legacy platform to a new one, still shockingly bad performance from them which they paid for in fines, compensation and they lost a lot of customers.