This isn’t a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn’t log in.
Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)
Seems insane to me that one company’s messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.
While I don’t totally disagree with you, this has mostly nothing to do with Windows and everything to do with a piece of corporate spyware garbage that some IT Manager decided to install. If tools like that existed for Linux, doing what they do to to the OS, trust me, we would be seeing kernel panics as well.
Hate to break it to you, but CrowdStrike falcon is used on Linux too…
And if it was a kernel-level driver that failed, Linux machines would fail to boot too. The amount of people seeing this and saying “MS Bad,” (which is true, but has nothing to do with this) instead of “how does an 83 billion dollar IT security firm push an update this fucked” is hilarious
Falcon uses eBPF on Linux nowadays. It’s still an irritating piece of software, but it no make your boxen fail to boot.
edit: well, this is a bad take. I should avoid commenting on shit when I’m sleep deprived and filled with meeting dread.
Even if it doesn’t kernel panic, a broken eBPF program can break all networking and I/O and effectively cripple a “running” system.
eBPF is better in a lot of aspects, but it won’t prevent software intended to block syscalls from breaking your machines if the code breaks.
The solution posted everywhere, simply delete the broken driver files, isn’t difficult or time consuming, except for situations where tens of thousands of devices stop responding at once, or where every machine is asking you for the encryption key because you’ve altered your boot parameters. Linux’ saving grace here may be that Bitlocker-style encryption is a pain to set up so Linux servers typically don’t do the encryption at all, but the recovery process for enterprise customers would still be very manual and time consuming.
It was panicking RHEL 9.4 boxes a month ago.
Were you using the kernel module? We’re using Flatcar which doesn’t support their .ko, and we haven’t been getting panics on any of our machines (of which there are many).
Nah it was specifically related to their usage of BPF with the Red Hat kernel, since fixed by Red Hat. Symptom was, you update your system and then it panics. Still usable if you selected a previous kernel at boot though.
You’re asking the wrong question: why does a security nightmare need a 90 billion dollar company to unfuck it?
What’s your solution to cyberattacks?
Linux in the hands of professionals. There’s a reason IIS isn’t used anymore.
That doesn’t solve anything. Linux is also subject to cyberattacks.
And Macs, we have it on all three OSs. But only Windows was affected by this.
Hate to break it to you, but most IT Managers don’t care about crowdstrike: they’re forced to choose some kind of EDR to complete audits. But yes things like crowdstrike, huntress, sentinelone, even Microsoft Defender all run on Linux too.
Yeah, you’re right.
I wouldn’t call Crowdstrike a corporate spyware garbage. I work as a Red Teamer in cybersecurity, and EDRs are bane of my existence - they are useful, and pretty good at what they do. In the last few years, I’m struggling more and more to with engagements we do, because EDRs just get in the way and catch a lot of what would pass undetected a month ago. Staying on top of them with our tooling is getting more and more difficult, and I would call that a good thing.
I’ve recently tested a company without EDR, and boy was it a treat. Not defending Crowdstrike, to call that a major fuckup is great understatement, but calling it “corporate spyware garbage” feels a little bit unfair - EDRs do make a difference, and this wasn’t an issue with their product in itself, but with irresponsibility of their patch management.
Fair enough.
Still this fiasco proved once again that the biggest thread to IT sometimes is on the inside. At the end of the day a bunch of people decided to buy Crowdstrike and got screwed over. Some of them actually had good reason to use a product like that, others it was just paranoia and FOMO.
How is it not a window problem?
Why should it be? A faulty software update from a 3rd party crashes the operating system. The exact same thing could happen to Linux hosts as well with how much access those IPSec programms usually get.
But that patch is for windows, not Linux. Not a hypothetical, this is happening.
Your fixated on the wrong part of the story. Synchronized supply chain update takes out global infrastructure isn’t a windows problem, this happens on linux too!
Just because a drunk driver crashes their BMW into a school doesn’t mean drunk driving is only a BMW vehicle problem.
I love how quickly everyone has forgotten about that xz attack.
I use and love Linux and have for over two decades now, but I’m not going to sit here and claim that something similar to the current Windows issue can’t happen to Linux.
That has nothing to do with this. That was a security vulnerability, solved in record time, blame where it was due, and patched in hours.
You’re missing the point. That compromised xz made it into some production distributions. The point here is that shit can happen to Linux, too.
If BMW makes a car that has square wheels and needs to have everyone install round wheels so the fucking thing works you can’t blame a company for making wheels.
It’s a Microsoft problem through and through.
Your counter to the BMW Drunk driver example didn’t address drunk driving in volvos, toyotas, fords… you just introduced a variable that your upset with. BMW’s having weird wheels has nothing to do with Drunk Driving incidents.
Again your focused on the wrong thing, this story is a warning about supply chain issues.
Your just memeing on the hate for windows.
Have you never seen a DNS outage, a ansible outage, a terraform outage, a RADIUS outage, a database schema change outage, a router firmware update outage?
Again, you’re talking about something I am not. I am talking about THIS problem, right here, that is categorically a windows problem, in that it’s not on the linux kernel stack, or mac. How is this NOT a windows problem??
If an update to the proprietary Nvidia driver causes Linux to crash, that’s an Nvidia problem, not a Linux problem.
its a problem that happened ON windows, it isn’t fundamentally a windows problem
The fault seems to be 90/10 CS, MS.
MS allegedly pushed a bad update. Ok, it happens. Crowdstrike’s initial statement seems to be blaming that.
CS software csagent.sys took exception to this and royally shit the bed, disabling the entire computer. I don’t think it should EVER do that, so the weight of blame must lie with them.
The really problematic part is, of course, the need to manually remediate these machines. I’ve just spent the morning of my day off doing just that. Thanks, Crowdstrike.
EDIT: Turns out it was 100% Crowdstrike, and the update was theirs. The initial press release from CS seemed to be blaming Microsoft for an update, but that now looks to be misleading.
It is on the sense that Windows admins are the ones that like to buy this kind of shit and use it. It’s not on the sense that Windows was broken somehow.