The CrowdStrike file that broke everything was full of null characters?(twitter.com)
twitter.com
The CrowdStrike file that broke everything was full of null characters?
https://twitter.com/jeremyphoward/status/1814364640127922499
351 コメント
On a related note, I don't think that it's a coincidence that 2 of the largest tech meltdowns in history (this one and the SolarWinds hack from a few years ago) were both the result of "security software" run amok. (Also sad that both of these companies are based in Austin, which certainly gives Austin's tech scene a black eye).
IMO, I think a root cause issue is that the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture. For example, I can't speak so much for CrowdStrike, but it came out the SolarWinds had an egregiously bad security culture at their company. When the root cause comes out about this issue dollars-to-donuts it was just a fast and loose deployment process.
IMO, I think a root cause issue is that the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture. For example, I can't speak so much for CrowdStrike, but it came out the SolarWinds had an egregiously bad security culture at their company. When the root cause comes out about this issue dollars-to-donuts it was just a fast and loose deployment process.
On the plus side of this disaster, I am holding out some pico-sized hope that maybe organizations will rethink kernel level access. No, random gaming company, you are not good enough to write kernel level anti cheat software.
this seems like the second or third test file any qa person would have tried, after an empty file and maybe a minimal valid file. the level of pervasive incompetence implied here is staggering
in a market where companies compete by impressing nontechnical upper management with presentations, it should be no surprise that technically competent companies have no advantage over incompetent ones
i recently read through the craig wright decision https://www.judiciary.uk/judgments/copa-v-wright/ (the guy who fraudulently claimed to be satoshi nakamoto) and he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist (decompiling malware to c); he didn't know what 'unsigned' meant when questioned on the witness stand. he'd been doing infosec work for big companies going back to the 90s. he'd apparently been tricking people with technobabble and rigged demos and forged documents for his entire career
george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd... https://en.wikipedia.org/wiki/George_Kurtz
crowdstrike itself caused the same problem on debian stable three months ago: https://old.reddit.com/r/debian/comments/1c8db7l/linuximage6...
it's horrifying that pci compliance regulations have injected crowdstrike (and antivirus) into virtually every aspect of today's it infrastructure
in a market where companies compete by impressing nontechnical upper management with presentations, it should be no surprise that technically competent companies have no advantage over incompetent ones
i recently read through the craig wright decision https://www.judiciary.uk/judgments/copa-v-wright/ (the guy who fraudulently claimed to be satoshi nakamoto) and he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist (decompiling malware to c); he didn't know what 'unsigned' meant when questioned on the witness stand. he'd been doing infosec work for big companies going back to the 90s. he'd apparently been tricking people with technobabble and rigged demos and forged documents for his entire career
george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd... https://en.wikipedia.org/wiki/George_Kurtz
crowdstrike itself caused the same problem on debian stable three months ago: https://old.reddit.com/r/debian/comments/1c8db7l/linuximage6...
it's horrifying that pci compliance regulations have injected crowdstrike (and antivirus) into virtually every aspect of today's it infrastructure
I have flagged this because it is wrong, and there is no other way to kick it off the front page.
Finding a file full of zeroes on a broken computer does not mean it was shipped as all zeroes!
https://x.com/craiu/status/1814339965347610863
https://x.com/cyb3rops/status/1814329155833516492
Finding a file full of zeroes on a broken computer does not mean it was shipped as all zeroes!
https://x.com/craiu/status/1814339965347610863
https://x.com/cyb3rops/status/1814329155833516492
Yes, this fits in with what I heard on the grapevine about this bug from a friend who knows someone working for Crowdstrike. The bug had been sitting there in the kernel driver for years before being triggered by this flawed data, which actually was added in a post-processing step of the configuration update - after it had been tested but before being copied to their update servers for clients to obtain.
Apparently, Crowdstrike's test setup was fine for this configuration data itself, but they didn't catch it before it was sent out in production, as they were testing the wrong thing. Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure, in whatever post-mortem writeup they may release.
Apparently, Crowdstrike's test setup was fine for this configuration data itself, but they didn't catch it before it was sent out in production, as they were testing the wrong thing. Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure, in whatever post-mortem writeup they may release.
There's a claim over on Mastodon from Kevin Beaumont that the file is different on every customer he´s received the file from.
https://cyberplace.social/@GossiTheDog/112812454405913406
(scroll down a little)
https://cyberplace.social/@GossiTheDog/112812454405913406
(scroll down a little)
It's possible that these aren't the original file contents, but rather the result of a manual attempt to stop the bleeding.
Someone may have hoped that overwriting the bad file with an all-0 file of the correct size would make the update benign.
Or following the "QA was bypassed because there was a critical vulnerability" hypothesis, stopping distribution of the real patch may be an attempt to reduce access to the real data and slow reverse-engineering of the vulnerability.
Someone may have hoped that overwriting the bad file with an all-0 file of the correct size would make the update benign.
Or following the "QA was bypassed because there was a critical vulnerability" hypothesis, stopping distribution of the real patch may be an attempt to reduce access to the real data and slow reverse-engineering of the vulnerability.
Found this post on 4chan's technology board:
What actually happened was two-folded:
Step 1: CSAgent.sys is a certified durgasoft kernel driver that parses clownstrike virus definition files
Step 2: Clownstrike never fixed cases for malformed virus definition files that could trigger an invalid memory access in CSAgent.sys
Step 3: Clownstrike ships the buggy CSAgent.sys and it works for months without problems on customer computers with good virus definition files
Step 4: For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes
Step 5: All clownstrike updaters now download malformed LE NEXT GEN APT PROTECTION CLOUD AI LIVE UPDATES into C:\Windows\System32\drivers\clownstrike
Step 6: CSAgent.sys reloads virus definition files
Step 7: CSAgent.sys parses them and crashes with PAGE_FAULT_IN_NONPAGED_AREA (in kernel that means memory at an oopsie address was accessed)
Step 8: Computer BSOD and reboots
Step 9: CSAgent.sys loads virus definition files
Step 10: Goto Step 7
The kernel driver was a ticking timebomb just waiting for a bug in the CDN. I think it was some funny caching bug with le cloud http loadbalancer. Users reported that their C-00000291-00000000-00000032.sys contained random shit and actual real files that were a completely different part of the software, like US localization files.
You can see it in the diff between CSAgent.sys_18511.sys and CSAgent.sys_18513.sys, they changed size checks and increased sizes of buffers so that future malformed virus definition files wouldnt crash.
What actually happened was two-folded:
Step 1: CSAgent.sys is a certified durgasoft kernel driver that parses clownstrike virus definition files
Step 2: Clownstrike never fixed cases for malformed virus definition files that could trigger an invalid memory access in CSAgent.sys
Step 3: Clownstrike ships the buggy CSAgent.sys and it works for months without problems on customer computers with good virus definition files
Step 4: For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes
Step 5: All clownstrike updaters now download malformed LE NEXT GEN APT PROTECTION CLOUD AI LIVE UPDATES into C:\Windows\System32\drivers\clownstrike
Step 6: CSAgent.sys reloads virus definition files
Step 7: CSAgent.sys parses them and crashes with PAGE_FAULT_IN_NONPAGED_AREA (in kernel that means memory at an oopsie address was accessed)
Step 8: Computer BSOD and reboots
Step 9: CSAgent.sys loads virus definition files
Step 10: Goto Step 7
The kernel driver was a ticking timebomb just waiting for a bug in the CDN. I think it was some funny caching bug with le cloud http loadbalancer. Users reported that their C-00000291-00000000-00000032.sys contained random shit and actual real files that were a completely different part of the software, like US localization files.
You can see it in the diff between CSAgent.sys_18511.sys and CSAgent.sys_18513.sys, they changed size checks and increased sizes of buffers so that future malformed virus definition files wouldnt crash.
We've had security software in the past break software compilation in this method by replacing entire files with zeros. I'm not saying this is the case, but it wouldn't surprise me if it were.
Basically the linker couldn't open the file on windows (because it was locked by another process scanning it), and didn't error. Just replaced the object code to be linked with zeros.
People couldn't figure out what was wrong until they opened a debugger and saw large chunks of object code replaced with zeros.
Basically the linker couldn't open the file on windows (because it was locked by another process scanning it), and didn't error. Just replaced the object code to be linked with zeros.
People couldn't figure out what was wrong until they opened a debugger and saw large chunks of object code replaced with zeros.
I am going to take a guess on how the CrowdStrike event might have happened. This is based on two (2) observations:
(1) CS said they have always followed the same process in the past without issues
(2) (personal observation) When certain NAS drives get full and you modify a file, it ends up with all NULL bytes.
Here's my guess on how their process was:
Typically, when their content file is ready, they push it to their farm of QA test machines for testing and also to this NAS drive that's ready for deployment. And when all of their test deployments are successful, they press this BIG RED button that picks up this content file from the NAS drive for global deployment.
During their prior deployments, this NAS drive had plenty of free space and deployments went well. Now, when the NAS was full, upon copy, it produces a file with all NULL bytes.
In summary, the content file they tested with was not the file that got globally deployed.
At least it compressed well, which must have saved network resources during the update. :)
That probably explains how it got past internal testing. Something went wrong after that, during deployment.
I don't get it. Shouldn't the file have a standard format, with a required header and checksum (among other things), that the driver checks before executing?
Something like
zero_output_file(fh, len(file))
flush()
fill_output_file(fh, data)
We can argue all we want about CI infrastructure, manual testing, test nets/deployment, staged deployment.
All of that is secondary: they wrote and shipped code that blindly loaded and tried to parse content from the network, and crashed when that failed. In kernel mode.
Honestly it’s probably good that this happened, because presumably someone malicious could use this level of broken logic to compromise kernel space.
Certainly the trust they put in the safety of parsing content downloaded from the internet makes me wonder about the correctness of their code for reading data from userspace.
All of that is secondary: they wrote and shipped code that blindly loaded and tried to parse content from the network, and crashed when that failed. In kernel mode.
Honestly it’s probably good that this happened, because presumably someone malicious could use this level of broken logic to compromise kernel space.
Certainly the trust they put in the safety of parsing content downloaded from the internet makes me wonder about the correctness of their code for reading data from userspace.
Note to self: on Monday, add a null character check to pre-commit hooks, and add the same check to pipelines.
While I won't discount it entirely, I think the people acting like this (alone) implies malice are being very silly.
I assume the Kernel driver itself has been signed, tested and verified by MS. The channel update files would be the responsibility of the CS and contain most of the information, including urgent (untested) information. This CS kernel driver is there to inspect, collect everything, receive CS settings and commands, and act in privileged mode. It's not a so complex system, the intelligence is in CS, that's why MS partners with them. But the code in the driver could at least do a perfect input check.
Shits happens. Blaming MS: I want to believe they do extensive sandbox testing, but in this case they could do better canary deployments, staged updates and not affect worldwide.
This looks like a test file that got deployed. Perhaps a QA test was newly added which ran and overwrote the build. This is all I can think of.
So ... the checksum of all the Seinfeld episodes?
Lots of very strong opinions here what they should have done in terms of code/pipelines/etx from people who have no idea how CrownStrikes stack or build systems works and people who have no clue how to write that type of software themselves (including Windows kerneln modules).
I'm not versed enough into windows loading dll / driver, but isn't the caller able to handle that situation? Or windows itself? Does loading an empty file driver can be handled in a way that it does not make the OS crash?
Anytime critical infrastructure goes down I always have a fleeting thought back to “Spy Game” movie where the CIA cut power to part of a Chinese city to help with a prison escape.
Poor testing. But we also need to stop CISOs, etc doing "checkbox compliance" and installing every 3rd party thing on employee laptops. My prior employer, there were literally 13 things installed for "laptop security" - 1/2 of them overlapped. Developers had the same policy as an AE and as a Sales Engineer as well as an HR person. Crowdstrike was one of the worst. Updating third party packages in go was 30-40% faster in an emulated arm64 VM (qemu) - virtualized disk / disk just a large file - on an Intel MBP compared to doing the same operation on the native system in OSX.
I wonder if this is linked to the Azure outage that happened just prior to the Crowdstrike incident.
I'm starting to think that the timing (Friday) and the scale as well as other things (like this finding) might -- just might point to a bad actor.
We will probably have to wait for CS' own report.
We will probably have to wait for CS' own report.
Segue: What the heck is about Windows files and null characters? I've been almost exclusively dealing with POSIX file systems for the last 30 years, but I'm currently shipping a cross-platform app and a lot of my Windows users are encountering corrupted files which exhibit a bunch of NULs in random places in the file. I've added ugly hacks to deal with them but it'd be nice to get down to root causes. Is there a handy list of things that are safe on POSIX but not on Windows so I can figure out what I'm doing wrong?
I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?
I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?
[deleted]
Maybe Crowdstrike has adopted
the modern ethos
Move fast and break things
With continuous integration
we ship a thousand times a day.
fuck QA.
I’m not even particularly stringent when it comes to automated test across-the-board but for this level of criticality of system, you need exceptionally good state management
To the point where you should not roll to production without an integration test on every environment that you claim to support
Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support.
Who is running stuff over there - total incompetence