This global Microsoft Outage

[SSpratt]

https://www.bbc.co.uk/news/live/cnk4jdwp49et

Any worries? It is chaos out there by the look of the news.

My experience is that I’m not able to work today and had trouble using my debit card this morning. The transaction eventually went through but it’s not showing on my account.

Although in NZ it was Friday evening so not ideal for those who have to sort these issues out, whereas for others it was Friday morning

I agree - my exDH, who works in IT software, was keeping an eye on things for their clients. It didn't seem to be bothering him however, in fact I think he was sort of enjoying the drama!

People who work in that field have to be prepared to work at all sorts of hours.

Exactly. In all the discussion of this by normies, there seems to be this idea that backups are an instant-restoration-of-service wildcard that can be deployed at any time.

A full backup system will take hours, sometimes even days to be fully deployed - especially when there are many interdependent systems affected. And, in case anyone hasn't thought it through that far...as soon as they completed the build procedure, including ensuring that all the security software was up to date, they would reboot and it would instantly show the same problem again (prior to the discovery of the actual cause of the issue).

Backups are not a silver bullet for configuration issues. They're a last resort for data loss, which was not the issue here.

Now, a hot spare...that's something that could potentially have been useful, if every company involved had a hot spare of their entire infrastructure...but again, that would've also been affected, because the security software would necessarily have to be kept up to date.

It was always a question of when not if. More and more hospital records are becoming technological with no back up. We are being held to ransom by computer use.

Yes and knowing this organisations should also be factoring how they fix problems.

It will be interesting to see who is slowest to rectify problems. They should be doing reviews into a lack of recovery coverage. That's as important as protection. If it's been overlooked heads should roll for that too.

Agreeing @ntmdino and @PetuniaT

What CrowdStrike are going to need to answer now to all clients and to their certification bodies is how a company that apparently has FedRAMP, ISO and SOC approved Secure Software Development and Deployment policies and procedures in place managed to have this happen.

There was a lack of governance somewhere in the process, whether that was caused by resource, competency or poor adherence to process is something they need to identify and address.

This is a hugely costly issue for them, they will incur fines from clients with strict SLAs and service penalties, they'll have clients terminate as a result of breach of contract, reputationally this will be hard to recover from. There are concerns about their outsourcing practices so they will likely need to quickly revert to in housing those tasks to show willing and build confidence. There is a significant risk this could be financially damaging to the point of no return for them.

It's entirely possible that one consequence of this incident is that we see moves towards regulating the antivirus and security industries - in particular, a framework which mandates liability for companies who push out uncontrolled changes like this.

Yes, that means the cost of security software massively increases, and it will probably mean a consolidation of players in the market, but it also means there will be some sort of peace-of-mind to offset the world's sudden horrifying realisation that, in the name of security, everybody's given all of these companies a window of root access into the very core of their infrastructure.

If the US mandates any kind of effective strict liability for errors like this I'll eat my own knees. Depending on who wins the election there's a small chance they will put forward something but even if it does make it through congress and the senate, and it probably wouldn't, it'd be so watered-down as to be useless. There's really no appetite for legislation like this.

The EU, on the other hand, just might. I don't think it's very likely but it's the only entity I can see who's got a big enough hammer and the potential will to wield it.

Well tried again this morning and I still can’t log on after not being able to work on Friday. I work in payroll and lots of Bacs deadlines today for people to be paid on the 25th, really panicking that my clients won’t get paid if none of my colleagues can pick up my work…..

Does anyone know when all will be back to normal?

It's a really annoying bug because a fix can't be rolled out all at once, IT teams are having to fix every computer one by one.

You will need to make contact with your IT helpdesk if they haven't been able to communicate a plan to you. If you highlight the payroll deadlines they may be able to bump you up the fix list.

Can't disagree with any of that, really.

begs the question of a new cyber weapon

Not even slightly.

I mean, I'm guessing you mean of the "Hey, you're an American company...we want you to nuke all your customers' machines in <country we don't like>" type.

Never gonna happen, that company would go bust in three seconds flat.

And besides, why do that with a security software company, when you've already got the operating system manufacturer in your back yard? Who...also wouldn't do ever do that.

This is why security services spend so much of their R&D budget writing viruses - no company would voluntarily say to the world "Hey, your machines are secure with us, as long as our government still likes you".

Hmm. No.

Remember Stuxnet ?

plausible deniability, or an off the books operation etc

Do you know how and why this issue happened with cloudstrike?

Only what the news says and not what the security services files say

Yes exactly.

When I worked in IT I enjoyed the global operating system bugs. Once I couldn't fire up my server long enough to get the patch updated. Three days of being unable to work.

Localised outages like someone doing a Unix patch and forgetting to reset a software switch, not so much. You'd at best have some manager breathing down your neck asking when you would fix it every five minutes for hours, or worse, you was all hauled into a conference room for a whole department interrogation. People forget that to de bug you need to be... debugging, not having meetings about it.

Part terror when when it went TU and part thinking 'this will take as long as it takes so let's just plod through the process'

People forget that to de bug you need to be... debugging, not having meetings about it.

Ain't that the truth!

Other than rampant unchecked paranoia, why should we believe this event to be a deliberate criminal act rather than just a fuck-up? Especially when we consider that even the CEO of the company that perpetrated the fuck-up has come out and said "Oops, we fucked up. Sorry about our fuck-up".

If its a fuck up if a patch goes out pdq from Microsoft or we are told how to fix / patch it. If your in IT you don't suffer from being paranoid. It's a fuck up or it's a hack no one lies to the programmers. We need to know the truth plus the knowledge base is extremely high, we would sniff out the truth. We had entire databases wiped by hackers just because it was doable. I was never lied to.

Imagine asking for the database and the front end to be fixed and prevent it happening again by lieing to them? You just can't. The reality is you, the superuser sees the problem then you find out what happened and you also fix it. Because its all visible right in front of you. If it's operating system level someone else can see all of that. That was a fun week..

I think it DOES kind of show how something like anti-virus software can be used as a cyber weapon. I've often felt a little worried that my anti virus software actually has full access to my PC, and all we can do is "hope" that the anti virus company are reputable and competent.

Yes, if such a company did plant a virus themselves, then it would definitely be game over for them in terms of reputational damage. BUT, if, say, the Russian government paid them many tens of million dollars, it would probably be worth more to them than future profits from their anti virus software firm and the person/persons bribed to do it could have free passage arranged to some country/island without an extradition treaty, or given a new identity, so they could walk away and disappear.

Think it through, please.

For it to be worth it for a hostile government to approach a company to do this, the company would have to have hundreds of thousands of customers, or at the very least many high-profile large customers. You don't do that by being a small company in the AV world, so - by definition - that company is going to have hundreds (if not thousands) of employees across the world. Crowdstrike, for example, has nearly 8000 employees.

For every single one of those employees, you'd have to bring them into the conspiracy, or ensure that they didn't notice what you were doing with the company's primary product.

All it would take is one developer out of hundreds/thousands - all extremely well-versed in security practices and threat detection - to spot something awry, and everybody's going to jail.

It's just not realistic.