Initial details about why CrowdStrike's CSAgent.sys crashed

They probably don't get to claim agile story points until the ticket is in finished state. And they probably have a culture where vanity Metrics like "velocity" are prioritized

Surely, CrowdStrike's safety posture for update rollouts is in serious need of improvement. No argument there.

But is there any responsibility for the clients consuming the data to have verified these updates prior to taking them in production? I haven't worn the sysadmin hat in a while now, but back when I was responsible for the upkeep of many thousands of machines, we'd never have blindly consumed updates without at least a basic smoke test in a production-adjacent UAT type environment. Core OS updates, firmware updates, third party software, whatever -- all of it would get at least some cursory smoke testing before allowing it to hit production.

On the other hand, given EDR's real-world purpose and the speed at which novel attacks propagate, there's probably a compelling argument for always taking the latest definition/signature updates as soon as they're available, even in your production environments.

I'm certainly not saying that CrowdStrike did nothing wrong here, that's clearly not the case. But if conventional wisdom says that you should kick the tires on the latest batch of OS updates from Microsoft in a test environment, maybe that same rationale should apply to EDR agents?

For me, number 1 is the worst of the bunch. You should always expect that there will be bugs in processes, input files, etc… the fact that their code wasn’t robust enough to recognize a corrupted file and not crash is inexcusable. Especially in kernel code that is so widely deployed.

If any one of the five points above hadn’t happened, this event would have been avoided. However, if number 1 had been addressed - any of the others could have happened (or all at the same time) and it would have been fine.

I understand that we should assume that bugs will be present anywhere, which is why staggered deployments are also important. If there had been staggered deployments, the. The damage would have happened, but it would have been localized. I think security people would argue against a staged deployment though, as if it were discovered what the new definitions protected against, an exploit could be developed quickly to put those servers that aren’t in the “canary” group at risk. (At least in theory — I can’t see how staggering deployment over a 6-12 hour window would have been that risky).

And here I thought shipping a new version on the app store was scary.

Is there anything we can take from other professions/tradecraft/unions/legislation to ensure shops can’t skip the basic best practices we are aware of in the industry like staged rollouts? How do we set un incentives to prevent this? Seriously the App Store was raking in $$ from us for years with no support for staged rollouts and no other options.

Malware signature updates are supposed to be deployed ASAP, because every minute may count when a new attack is spreading. The mistake may have been to apply that policy indiscriminately.

A lot of snarky replies to this comment, but the reality is that if you were selling an anti-virus, identified a malicious virus, and then chose not to update millions of your machines with that virus’s signature, you’d also be in the wrong.

The triggering file was all zeros.

It is not possible that only this pattern caused the crash, and fuzzing omitted to try this unfuzzy pattern?

AV software is a great target for malware, badly written, probably runs too much stuff in the kernel, tries to parse everything

You are seriously overestimating the engineering practises at these companies. I have worked in "enterprise security" previously though not at this scale. In a previous life I worked with of the engineering leaders currently at Crowdstrike.

I'll bet you this company has some arbitrary unit test coverage requirements for PRs which developers game be mocking the heck out of dependencies. I am sure they have some vanity sonarqube integration to ensure great "code quality". This likely also went through manual QA.

However I am sure the topic of fuzz testing would not have come up once. These companies sell checkbox compliance, and they themselves develop their software the same way. Checking all the "quality engineering" boxes with very little regards for long term engineering initiatives that would provide real value.

And I am not trying to kick Crowdstrike when they are down. It's the state of any software company run by suits with myopic vision. Their engineering blogs and their codebases are poles apart.

Since the issue manifested at 04:09 UTC, which is 11pm where Crowdstrikes HQ is, I would guess someone was working late at night and skipped the proper process so they could get the update done and go to bed.

They probably considered it low risk, had done similar things of times hundreds of times before, etc.

It’s a tough problem, because you also don’t want the system to start without the CrowdStrike protection. Or more generally, a kernel driver is supposedly installed for a reason, and presumably you don’t want to keep the system running if it doesn’t work. So the alternative would be to shut down the system upon detection of the faulty driver without rebooting, which wouldn’t be much of an improvement in the present case.

There is sandboxing API in Windows. It's called running programs in userspace.

I had read somewhere that the definition file was corrupted after testing, during the final CI/CD pipeline.

How is that different from point 2?

The file extension is `sys` by convention, it's nothing magical to it and it's not handled in any special way by the OS. In the case of CrowdStrike, there seems to be some confusion as to why they use this file extension since it's only supposed to be a config/data file to be used by the real kernel driver.

He probably means work was sent offshore to offices with cheaper labor that's less skilled or less vested into delivering quality work. Though there's no proof of that yet, people just like to throw the blame on offshoring whoever $BIG_CORP fucks up, as if all programmers in the US are John Carmack and they can never cause catastrophic fuckups with their code or processes.

It's probably just the common US-centric bias that external development teams, particularly those overseas, may deliver subpar software quality. This notion is often veiled under seemingly intellectual critiques to avoid overt xenophobic rhetoric like "They're taking our jobs!".

Alternatively, there might be a general assumption that lower development costs equate to inferior quality, which is a flawed yet prevalent human bias.

Heh, touché.

Not really a surprise, to be honest. "Deserialisation" encapsulates most forms of injection attacks.

OWASP top-10 was dominated by those for a very long time. They have only recently been overtaken by authorization failures.

Yup. Almost all of them are various flavor of fucking up a parser or misusing it (in particular, all the injection cases are typically caused by writing stupid code that glues strings together instead of proper parsing).

Maybe there isn't a switch that says "content version",but from end user perspective it is a new version. Whether it was a content change, or just a fix for typo in documentation (say) the change being pushed is different than what currently exists.And for the end user the configuration implies that they have a chance to decide whether to accept any new change being pushed or not.

Yeah, even if you are only parsing "safe" inputs such as ones you created yourself. Other bugs and sometimes even truly random events can corrupt data.

"And aren’t they kinda right to not trust their employees if they employ 50,000 people with different skills and intentions?"

Yes, in a 50k employee company, the CEO won't know every single employee and be able to vouch for their skills and intentions.

But in a non-dysfunctional company, you have a hierarchy of trust, where each management level knows and trusts the people above and below them. You also have siloed data, where people have access to the specific things they need to do their jobs. And you have disaster mitigation mechanisms for when things go wrong.

Having worked in companies of different sizes and with different trust cultures, I do think that problems start to arise when you add things like individual monitoring and control. You're basically telling people that you don't trust them, which makes them see their employer in an adversarial role, which actually makes them start to behave less trustworthy, which further diminishes trust across the company, harms collaboration, and eventually harms productivity and security.

Security is a process not a product. Anyone selling you security as a product is scamming you.

These endpoint security companies latch onto people making decisions, those people want security and these software vendors promise to make the process as easy as possible. No need to change the way a company operates, just buy our stuff and you're good. That's the scam.

I disagree with the concept of surveillance altogether. Computer users should be educated about security, given control of their devices, and trusted that they will do the right thing. If a company can't do that, that's a sign that they don't have good security practices to begin with, and don't do a good job at hiring and training.

The only reason this kind of software is used is so that companies can tick a certification checkbox that gives the appearance of running a tight ship.

I realize it's the easy way out, and possibly the only practical solution for a large corporation, but then this type of issues is unavoidable. Whether the product is free or proprietary makes no difference.

Limit their abilities using OS features? Have the vendor fix security issues rather than a third party incompetently slapping on band-aid?

It's like you let one company build your office building and then bring in another contractor to randomly add walls and have others removed while having never looked at the blueprints and then one day "whoopsie, that was a supporting wall I guess".

Why is it not just completely normal but even expected that an OS vendor can't build an OS properly, or that the admins can't properly configure it, but instead you need to install a bunch of crap that fucks around with OS internals in batshit crazy ways? I guess because it has a nice dashboard somewhere that says "you're protected". Checkbox software.

I don't have first-hand experience with Kolide, as I refused to install it when it was pushed upon everyone in a company I worked for.

Complaints voiced by others included false positives (flagging something as a threat when it wasn't, or alerting that a system wasn't in place when it was), being too intrusive and affecting their workflow, and privacy concerns (reading and reporting all files, web browsing history, etc.). There were others I'm not remembering, as I mostly tried to stay away from the discussion, but it was generally disliked by the (mostly technical) workforce. Everyone just accepted it as the company deemed it necessary to secure some enterprise customers.

Also, Kolide's whole spiel about "honest security"[1] reeks of PR mumbo jumbo whose only purpose is to distance themselves from other "bad" solutions in the same space, when in reality they're not much different. It's built by Facebook alumni, after all, and relies on FB software (osquery).

[1]: https://honest.security/

If that’s all it takes an attacker, you’re doing AWS wrong.

It’s not your machine, it’s your employer’s machine.

Agreed. I'm fine with this, as long as the employer also accepts that I will never use a personal device for work, that I will never use a minute of personal time for work, and that my productivity is significantly affected by working on devices and systems provided and configured by the employer. This knife cuts both ways.

What does “our computer” mean when it is not owned by you, but issued to you to perform a task with by your employer?

Well, presuming that:

1. the employee is issued a computer, that they have possession of even if not ownership (i.e. they bring the computer home with them, etc.)

2. and the employee is required to perform creative/intellectual labor activities on this computer — implying that they do things like connecting their online accounts to this computer; installing software on this computer (whether themselves or by asking IT to do it); doing general web-browsing on this computer; etc.

3. and where the extent of their job duties, blurs the line between "work" and "not work" (most salaried intellectual-labor jobs are like this) such that the employee basically "lives in" this computer, even when not at work...

4. ...to the point that the employee could reasonably conclude that it'd be silly for them to maintain a separate "personal" computer — and so would potentially sell any such devices (if they owned any), leaving them dependent on this employer-issued computer for all their computing needs...

...then I would argue that, by the same chain of reasoning as in the GP post, employers should not be legally permitted to “issue” employees such devices.

Instead, the employer should either purchase such equipment for the employee, giving it to them permanently as a taxable benefit; or they should require that the employee purchase it themselves, and recompense them for doing so.

Cyberpunk analogy: imagine you are a brain in a vat. Should your employer be able to purchase an arbitrary android body for you; make you use it while at work; and stuff it full of monitoring and DRM? No, that'd be awful.

Same analogy, but with the veil stripped off: imagine you are paraplegic. Should your employer be allowed to issue you an arbitrary specific wheelchair, and require you to use it at work, and then monitor everything you do with it / limit what you can do with it because it’s “theirs”? No, that’d be ridiculous. And humanity already knows that — employers already can't do that, in any country with even a shred of awareness about accessibility devices. The employer — or very much more likely, the employer's insurance provider — just buys the person the chair. And then it's the employee's chair.

And yes, by exactly the same logic, this also means that issuing an employee a company car should be illegal — at least in cases where the employee lives in a non-walkable area, and doesn't already have another car (that they could afford to keep + maintain + insure); and/or where their commute is long enough that they'd do most non-employment-related car-requiring things around work and thus using their company car. Just buy them a car. (Or, if you're worried they might run away with it, then lease-to-own them a car — i.e. where their "equity in the car" is in the form of options that vest over time, right along-side any equity they have in the company itself.)

Does that also apply to the operator at a switchboard…

Actually, no! Because an operator of a switchboard is not a “user” of the computer that powers the switchboard, in the same sense that a regular person sitting at a workstation is a "user" of the workstation.

The system in this case is a “kiosk computer”, and the operator is performing a prescribed domain-specific function through a limited UX they’re locked into by said system. The operator of a nuclear power plant is akin to a customer ordering food from a fast-food kiosk — just providing slightly more mission-critical inputs. (Or, for a maybe better analogy: they're akin to a transit security officer using one of those scanner kiosk-handhelds to check people's tickets.)

If the "computer" the nuclear-plant operator was operating, exposed a purely electromechanical UX rather than a digital one — switches and knobs and LEDs rather than screens and keyboards[1] — then nothing about the operator's workflow would change. Which means that the operator isn't truly computing with the computer; they're just interacting with an interface that happens to be a computer.

[1] ...which, in fact, "modern" nuclear plants are. The UX for a nuclear power plant control-center has not changed much since the 1960s; the sort of "just make it a touchscreen"-ification that has infected e.g. automotive has thankfully not made its way into these more mission-critical systems yet. (I believe it's all computers under the hood now, but those computers are GPIO-relayed up to panels with lots and lots of analogue controls. Or maybe those panels are USB HID devices these days; I dunno, I'm not a nuclear control-systems engineer.)

Anyway, in the general case, you can recognize these "the operator is just interacting with an interface, not computing on a computer" cases because:

• The machine has separate system administrators who log onto it frequently — less like a workstation, more like a server.

• The machine is never allowed to run anything other than the kiosk app (which might be some kind of custom launcher providing several kiosk apps, but where these are all business-domain specific apps, with none of them being general-purpose "use this device as a computer" apps.)

• The machine is set up to use domain login rather than local login, and keeps no local per-user state; or, more often, the machine is configured to auto-login to an "app user" account (in modern Windows, this would be a Mandatory User Profile) — and then the actual user authentication mechanism is built into the kiosk app itself.

• Hopefully, the machine is using an embedded version of the OS, which has had all general-purpose software stripped out of it to remove vulnerability surface.

Does the switchboard in a nuclear missile launch facility run Crowdstrike? I picture it as a high quality analog circuit board that does 1 thing and 1 thing only. No way to run anything else.

Globally networked personal computers were kind of cultural revolution against the setting you describe. Everyone had their own private compute and compute time and everyone could share their own opinion. Computers became our personal extensions. This is what IBM, Atari, Commodore, Be, Microsoft and Apple (and later desktop Linux) sold. Now given this ideology, can a company own my limbs? If not, they can't own my computers.

I could see an open source solution with "private" or vendor specific definition files. But I think I'd disagree with the statement that open sourcing everything wouldn't cause any problem. Engineering isn't necessarily about peer reviewed studies, it's about empirical observations and applying the engineering method (which can be complemented by a more scientific one but shouldn't be confused for it). It's clear that this type of stuff is a game of cat and mouse. Attackers search for any possible vulnerability, bypass etc. It does make sense that exposing one side's machinery will make it easier for the other side to see how it works. A good example of that is how active hackers are at finding different ways to bypass Windows Defender by using certain types of Office file formats, or certain combinations of file conversions to execute code. Exposing the code would just make all of those immediately visible to everyone.

Eventually that's something that gets exposed anyways, but I think the crucial part is timing and being a few steps ahead in the cat and mouse game. Otherwise I'm not sure what kind of proof would even be meaningful here.

I have a different take on this.

I feel having the solution open sourced isn't bad from a code security standpoint, but rathee that it is simply not economically viable. To my knowledge most of the major open source technologies are currently funded by FAANG and purely because it's needed by them to conduct business and the moment it becomes inconvenient for them to support it they fork it or develop their own, see Terraform/Redis...

I also cannot get behind a government funding model purely because it will simply become a design by committee nightmare because this isn't flashy tech. Just see how many private companies have beaten NASA to market in a pretty well funded and very flashy industry. The very government you want to fund these solutions are currently running on private companies infrastructure for all their IT needs.

Yes opensouring is definitely amazing and if executed well will be better, just like communism.

I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world.

You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.

Oh absolutely. There’s many levels of failure here. A few that I see as being likely:

- Lack of testing of a deployment - Lack of required procedures to validate a deployment - Engineering management prioritizing release pace over stability/testing - Management prioritizing tech debt/pentests/etc far too low - Sales/etc promising fast turnarounds that can’t be feasibly met while following proper standards - Lack of top-down company culture of security and stability first, which should be a must for any security company

This outage wasn’t caused only by “the intern pushing release.” It was caused by a poor company culture (read: incorrect direction from the top) resulting in a lack of testing of the program code, lack of testing environment for deployments, lack of formal deployment process, and someone messing up a definition file that was caught by 0 other employees or automated systems.

Detecting system crashes would be hard. You could try logging and comparing timestamps on agent startups and see if the difference is 5 minutes or less. Buggy kernel drivers crash Windows hard and fast.

It’s kind of hard to pitch “zero-day prevention” if you suggest people roll out definitions slowly, over the course of days/weeks. Thus making it a lot harder to charge to the moon for your service.

Now, if these sorts of things were battle tested before release, and had a (ideally decade+-long) history of stability with well-documented processes to ensure that stability, you can more easily make the argument that it’s worth it. None of those things are close to true though (and more than likely will never be for any AV/endpoint solution), so it is very hard to justify this sort of configuration.

I don’t work with CS products atm, but my experience with a big CS deployment was exactly like this. They were openly quite hostile to any suggestion of testing their products, we were frequently rebuked for running our prod censors on version n-1. I talked about it a bit in this comment.

https://news.ycombinator.com/item?id=%2041002864

Very much not surprised to see this now.

Ah that was me. We don’t accept “content updates” and they are staged.

We got this update pushed right through.

What recourse any customer can have in in such a case ?

Sue them and use something else.

If it is a standard production rocket, I agree. If it is a first of kind or even third of kind launch, celebrating the lessons learned from a failure is a healthy attitude. This production software is not the same thing at all.

spaceX celebrating when their rocket blows up after a certain milestone it's like us devs celebrating when our branch with that new big feature only fails a few tests. Did it pass no? Are you satisfied as first try? Probably

security industry's favorite language is nothingspeak

raises hand

Some people don't jump on every fad out there. Most of the people who miss out on fads quickly realize that they aren't losing out on much simply because fads are so ephemeral. As far as I can tell, this is normal (though different people will come to that realization at different stages of their life).

I don't have any social media of any kind, unless you count HN.

My wife only uses Facebook, and even then pretty sparingly.

I never had insta. Why would anyone use that.

Sadly enough the "average" instagram user doesn't use threads. It's just a weird subset of them that use it, and imo it's not the subset that makes Instagram great lol. (It's a lot of pre 2021 twitter refugees, and that's an incredibly obnoxious and self centered crowd in my experience)

Some of us stay far, far away from Facebook.

Maybe the experience varies depending on where the user is located. Users near Mastodon servers (possibly on the US East or West Coast) may not feel the slowness as much as users in other parts of the world. I notice noticeably slower response times when I use Mastodon in my location (Korea).

Same. I have no issues at all on Mastodon. I’m quite happy with it.

Mastodon - mixed feelings.

In my experience, Mastodon is nice until you want to partake in discussions. To do so, you need an account.

With an account you can engage in civilized discussions. Some people don't agree with you, and you don't agree with some people. That's fine, maybe you'll learn something new. It's a discussion.

And then, suddenly, a secret court convenes and kills your account just like that; no reason will be given, no recourse will be available, admins won't reply, and you can do two things: go away for good, or try again on a different server.

I'm happy with a read-only Mastodon via a web interface.

But read-write? Never again, I probably don't have the correct ideology for it.

No one wants to pay for anything, and that's the true root of every issue around this. People complain YouTube has ads, but wont buy premium.

Perhaps if buying into a service guaranteed that they would not be sold out then there would be more engagement. When someone signs up it is pretty much a rock-hard guarantee that their personal information will be marketed and sold to any entity with the money and interest to buy it - paying customers, free-loaders, etc.

When someone chooses to buy your app or SaaS then they should be excluded from the list of users that you sell or trade between "business partners".

When paying for a service guarantees that you're selling all details of your engagement with that service to unrelated business entities you have a disincentive to pay.

People are wising up to all this PII harvesting and those clowns who sold everyone out need to find a different model or quit bitching when real people choose to avoid their "services" since most of these things are not necessary for people to enjoy life anyway. They are distractions.

EDIT: This is not intended as a personal attack on you but is instead a general observation from the perspective of someone who does not use or pay for any apps or SaaS services and who actively avoids handing out accurate personal information when the opportunity arises.

It is not so bad with Mastodon but much fedi software gets slower the longer it's been running. "Akkoma Rot" is the one that's typically most talked about but the universe of misskey forks experiences the same problems, and Mastodon can sometimes absolutely crunch to a halt on 4GB of ram even for a single user instance.

Relatedly, it's crazy to me how many people still read the news!

I agree the problem is hard from a technical level.

The problem is also getting significantly worse because it's trivial to generate entire pages of inorganic content with LLMs.

The backstories of inorganic accounts are also much more convincing now that they can be generated by LLMs. Before LLMs, backstories all focused on a small handful of topics (e.g. sports, games) because humans had to generate them from playbooks of best pracitces. Now they can be into almost anything.

I don't necessarily think that's true anymore. YouTube comments are important to the algorithm so creators are more and more active in the comment section, and the comments in general have been a lot more alive and often add a lot of context or info for some type of videos. YouTube has also started giving the comments a lot more visibility in the layout (more than say, the video description). But you're probably right w.r.t platforms like TVs.

Before this wave of insane bot spam, the comments had started to be so much better than what they used to be (low effort, boomer spam). In fact I think they were much better than the absolute cringy mess that comments on dedicated forums like Reddit turned into

Viruses don't move that fast

Yes they do? And it’s more akin to a shared immune system than a single organism.

In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.

https://en.wikipedia.org/wiki/SQL_Slammer

There is no real "speed limit" on malware spread.

The scanner is a Ring 0[0] program. Windows only has 2 options 0 and 3. 3 won't work for any kind of security scanners, so they're forced to use 0.

The proper place would be Ring 1, which doesn't exist on Windows.

And being a kernel-level operation, it has the capability to crash the whole system before the actual OS has any chance to intervene.

[0] https://en.wikipedia.org/wiki/Protection_ring

Because kernel needs to parse the data in some way and that parser apparently was broken enough. Whether it could be done in a more resilient manner, I don't know, you need to remember that antivirus works in hostile environment and can't necessarily trust userspace, so probably they need to verify signatures and parse payload in the kernel space.

Even if it is a staged rollout why would one do it in 24 hour phases ? It can be a hourly (say) staggered rollout too.

There should have been a test catching the error before rollout, however this doesn’t require a staged rollout as suggested by the GP comment, testing the update at some customers (which would still be hosed in that case), it only requires executing the test before the rollout.

The update files are almost certainly signed.

Even if the HTTPS channel is compromised with a man-in-the-middle attack, the attacker shouldn't be able to craft a valid update, unless they also compromised CrowdStrke's keys.

However, the fact that this update apparently managed to bypass any internal testing or staging release channels makes me question how good CrowdStrike's procedures are about securing those update keys.

My speculation is the bit of code/data that was broken, is added after the build and testing precisely to avoid the $5 wrench attack.

That is, the data is signed and they don't want to use the real signing key during testing / in the continuous build because then it is too exposed.

So it's added after as something that "could not break". But it of course did.

that's assuming they don't do cert pinning. Moreover despite all the evil things you can supposedly do with a $5 wrench, I'm not aware of any documented cases of this sort of attack happening. The closest we've seen are misissuances seemingly caused by buggy code.

Microsoft signs its updates. There's no restriction on where you can get them from.

As a former member of the Windows Update software engineering team, I can say this is absolutely false. The updates are signed.

One of the most dangerous versions of this IMO is someone who compromises a NPM/Pypi package that's widely used as a dependency. If you can make it so that the original developer doesn't know you've compromised their accounts (spear-phished SIM swap + email compromise while the target is traveling, for instance, or simply compromising the developer themselves), you don't need every downstream user to manually update - you just need enough projects that aren't properly configured with lockfiles, and you've got code execution on a huge number of servers.

I'm hopeful that the fallout from Crowdstrike will be a larger emphasis on software BOM risk - when your systems regularly phone home for updates, you're at the mercy of the weakest link in that chain, and that applies to CI/CD and end user devices alike.

I mean, isn't that roughly the solarwinds story? There is no real shortage of supply chain incidents in the last few years. The reality is we are all mostly okay with that tradeoff.

Around -4 years? [1]

[1]: https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

You mean like the SolarWinds hack that happened a lil while ago?

https://www.techtarget.com/whatis/feature/SolarWinds-hack-ex...

I mean kernal level access does provide feature not accessible in userspace. Is it alsooverused when other solutions exist, you bet.

Most people don't need this stuff. Just keeping shit up to date, no not on the nightly build branch, but like installing windows update atleast a day or two after they come out. Or maby regular antivirus scans.

But let's be honest, your kernal drivers are useless if your employees fall for phishing or social engineering. See then its not malware, its an authorized user on the system....just copying data onto a USB drive or a rouge employee taking your customer list to your competition. That fancy pants kernal driver might be really good at stopping sophisticated threats and I'm sure the marketing majors at any company cram products full of buzz words. But remember, you can't fix incompetent or malicious employees unless your taking steps to prevent it.

What's more likely: some foreign government hacking khols? Or a script kiddie social engineers some poor worker pretending to be the support desk?

Not here to shit on this product, it has its place and it obviously does a good job....(heard its expensive but most xrd/edr is)

Seems like we are learning how vulnerable certain things are once again. As a fellow security fellow, I must say that Jia Tan must be so envious that he couldn't have this level of market impact.

Start a story for them: "and then, the hackers managed to install a rootkit which runs in kernel mode. The rootkit has sophisticated C2 mechanism with configuration files pretending to be drivers suffixed with .sys extensions. And then, they used that to prevent hospitals and 911 systems around the world from working, resulting in delayed emergency responses, injuries, possibly deaths".

After they cuss the hackers under their breath exclaiming something like: "they should be locked up in jail for the rest of their lives!...", tell them that's exactly what happened, but CS were the hackers, and maybe they should reconsider mandating installing that crap everywhere.

Well, millions of PCs bluescreening at the same time does help stop a rapidly spreading malware.

Only this time, crowdstrike itself has become indistinguishable from malware.

I feel like they need to at least first rollout to themselves

They say that, but all I hear is immune system triggering a cytokine storm and killing you because it was worried you may catch a cold.

But is there a need for quick global releases?

Is it realistic that there's a threat actor that will be attacking every computer on the whole planet at once?

I can understand that it's most practical to update everyone when pushing an update to protect a few actively under attack but I can also imagine policies where that isn't how it's done, while still getting urgent updates to those under attack.

A customer should be able to test an update, whether a signature file or literally any kind of update, before rolling it out to production systems. Anything else is madness. Being "vulnerable" for an extra few hours carries less risk than auto-updates (of any kind) on production systems. As we've seen here. If you can point to hard evidence to the contrary, where many companies were saved just in time because of a signature update and would have been exploited if they'd waited a few hours, I'd love to read about it. It would have to have happened on a rather large scale for all of the instances combined to have had a larger positive impact than this single instance.

You do remember Solarwinds right? This is an obvious high value target, so it is reasonable to entertain malicious causes.

Given the number of systems infected, if you could push code that rebooted every client into a compromised state you’d still have run of some % of the lot until it was halted. That time window could be invaluable.

Now, imagine if you screw up the code and just boot loop everything.

I’d say business wise it’s better for crowd strike to let people think it’s an own-goal.

The truth may be mundane but a hack is as reasonable a theory as “oops we pushed boot loop code to world+dog”.

To be fair, the xd backdoor wasn’t immediately obvious https://www.wired.com/story/xz-backdoor-everything-you-need-...

This is going to code as radical, but I always assumed it was derivable from bog-standard first principles that would fit in any economics class I sat in for my 40 credits:

the natural cost of these bits we sell is zero, so in the long run, if the bar is "just write a good & tested kernel driver", there will always be one more subsequent market entrant who will go too cheap on engineering. Then, they touch the hot wire and burn down the establishment.

That doesn't mean capitalism bad, but it does mean I expect only Microsoft is capable of writing and maintaining this type of software in the long run.

Ex. The dentist and dental hygienist were asking me who was attacking Microsoft on Friday, and they were not going to get through to the the subtleties of 3rd kernel driver release gating strategy.

MS has a very strong incentive to fix this. I don't know how they will. But I love when incentives align and assume they always will, in the long run.

"Attempt to read from address 0x9c" doesn't strike me as "null pointer". It's an invalid address and it doesn't really matter if it was null or not.

Probably not.

R8 is 0x9c in that example, which is somewhat typical for null+offset, but in the twitter thread it's 0xffff9c8e0000008a.

So the actual bug is further back. It's not a null pointer dereference, but it somehow results in the mov r8, [rax+r11*8] instruction reading random data (could be anything) into r8, which then gets used as a pointer.

Maybe this is a use-after-free?

NULL isn't always the integer 0 in C. It's implementation-defined.

What? If you have a null pointer to a class, and try to reference the member that starts 156 bytes from the start of the class, you’ll deference 0x9c (0 + 156)

If you have a page mapped at address 0, accessing address 0 is valid.

but they are able to insert themselves into this many enterprise machines! So regardless of your security credentials, they made good business decisions.

On the other hand, this may open the veil for a lot of companies to dump them.

They are probably pivoting to charging ransoms aka "consulting fees" to fix crashing systems and those are priced in.

Thousands of litigants

i would imagine a class action suit instead of individual cases if this were to happen.

So CrowdStrike has direct access to a lot of critical infrastructure? LOL.

What is French mutation testing? A casual kagi seems to imply its a type of genetic testing, or perhaps just tests that have been done in France?

I'm not overly familiar with crowdstrike processes, but assume they are long running. If it's all loaded to memory, eg a config, I can't see how you'd get any performance gain at all. It just seems lazy.

Maybe, maybe, but if it's not in a hot loop, why would the performance gain be worth it?

It's for incompetence reasons.

wild speculation aside, i'd say a little less performance is preferable to this outcome.

Shouldn't IOMMUs block that these days?