73 replies
1d7h
It's good to remind people that general liability waivers you often find with license agreements have no meaning outside of US jurisdiction if you're doing business in another jurisdiction.
56 replies
1d7h
Can someone explain to me why the protections that Falcon provides, are not provided by the OS itself? I am not completely naive, I've secured quite a few critical Linux servers, but with Windows it seems that there do not exist the same clear roles of security. Contrast with Red Hat or even Canonical, where is feels like I'm (correctly) fighting the security of the systems to get them into a state where my users can use my applications.
15 replies
1d7h
How do you actively detect a malware agent running in user space using stealth or a kernel. Authors of such are fully aware of Linux hardening like SELinux / AppArmor and work around it.
8 replies
1d7h
How do you actively detect a malware agent running in user space using stealth or a kernel.
You start with correct design.
The system has a root of trust (ideally you skip the insane level of complexity that is Secure Boot + TPM and use something simple, testable, and verifiable — this isn’t actually that hard). Only authorized images will boot, and, more importantly, nothing else on the network trusts the machine until it proves it’s running the right image.
Then you make the image immutable. Want to edit a system file? You can’t. Maybe in developer mode you can edit an overlay.
All configuration is stored in a designated place, and that configuration is minimized. A stock image from the distro vendor has zero configuration, so there is no incomprehensible soup in /etc to audit. Configuration is also attested.
Persistent data is separate from configuration. All persistent data is considered suspect. Any bug that allows malicious persistent data to compromise anything is a blocker, including corrupt filesystem metadata.
A root-of-trust attestation has limited lifetime. The system forcibly re-verifies periodically. This either means rebooting or doing a runtime “dynamic root of trust” attenuation. The latter is complex.
Complicated messes like kernel “lockdown” and the stock Secure Boot signatures have no place. Usermode root and the kernel are approximately equally trusted. SELinux is barely necessary, if at all, unless the actual user code wants it to control access to persistent data. But there are simpler, better schemes that are easier to reason about.
Sadly the industry doesn’t think this way. I’m regularly surprised that Apple hasn’t gone in this direction more aggressively than they are with their MacOS products.
4 replies
1d7h
What you've answered is a great (if not the best) way to defend against attackers, but not what was asked,They asked how to detect.
I'll strongly disagree on selinux, I have seen it work in practice to defeat attackers many times, that provide features that seccomp and cgroups etc do not.
3 replies
1d5h
re: SELinux, I think it depends on your use case.
If your system is a flight information display, then you may well have two userspace processes that do anything of significance: the display manager and the actual app. There is no persistent state. At this point, SELinux is purely overhead and extra attack surface — what would it even protect.
If your payload is a container (database server, microservice, whatever), and you’re doing some form of best-practice volume management, then only the database’s own data is mounted for it. SELinux is a real PITA to get working in a context like this, and it’s not really clear what it would add. (Okay, maybe you get fancy and use it to restrict what can talk to the microservice. Or maybe you use network namespaces.)
If you’re running a desktop or a more conventional server setup, then, sure, MAC policy has its place.
2 replies
1d2h
Absolutely depends on the use case. I'm attempting to talk in the generic case. If you limit policy to the minimum attack surface from outside the process including permissions and capabilities which are significantly more fine grained in selinux compared to normal Unix permissions, you reduce the the capability of the attacker once they gain access to the system.
Imagine if they got access to local code execution... Binding to sctp protocol would instantiate the whole protocol in kernel. Effectively opening up whole new attack vectors. I can't see any other techniques (other than selinux like AC) that enables this kind of attack space reduction as easily.
I am aware that you can blacklist modules,etc but this is just one of many examples.
0 replies
1d1h
You can use SEccomp for some of it as well. But for SEccomp something in the hierarchy needs to do this actively
While SELinux can be set up somewhat orthogonal to the running system. OTOH systemd should make it easy to confirm every service process
0 replies
15h38m
For this sort of kernel attack surface reduction, I would use a combination of seccomp and runtime module loading restriction.
In the specific example of sctp, one can turn off loading of modules at runtime entirely.
1 replies
1d1h
Only authorized images will boo
How do you do this on modern commodity hardware without secure boot?
Or do you assume something in the category of embedded systems that allow to blow some efuses to get similar trusted boot?
0 replies
15h40m
How do you do this on modern commodity hardware without secure boot?
It’s not necessarily easy without Secure Boot, sadly. The actual straightforward solution is boot ROM. It would be nifty if someone made SD cards, eMMC devices and such meant for this use case for independent use. Most Android vendors manage to use boot ROM.
0 replies
1d5h
You haven't answered anything interesting. Any software system that anyone cares about operates on state - user documents, a database, other bespoke systems etc. If the operator of that system accidentally deploys malware to it, how to you ensure that this malware doesn't destroy, replace, or exfiltrate this state the the system normally operates on?
Malicious code doesn't need to run as root in order to completely destroy a business.
Not to mention, all of the things you describe are very nice if the kernel is perfectly secure. But it's not, so it's always possible and even likely that compromising any user on the system is equivalent to compromising root. And if you compromise one system, you can then exploit bugs in other systems' kernels that might allow RCE through well-crafted packets or other exploits that gain access without running through any user-space code that might validate those attestations.
And finally, when a vulnerability is found allowing such exploits, you now need to update all of these readonly systems - and this happens at least once a month. Do you go with a USB stick to each of 10k systems on five continents to update them?
This kind of smug "I know better than the rest of the industry, security is easy if you do things my way" is rarely productive or applicable.
3 replies
1d7h
How do you actively detect a malware agent running in user space using stealth
Depending how advanced the attacker is, check the executing binary maps back to the actual expected name and location on disk. Make sure the executable and libraries used at runtime are the correct ones matching hashes of known good qualities.
Ensure the process tree structure has an expected structure, ie "bash" isnt starting a process called apache.
Make sure the selinux policy is correct for the process that is running. (I have no idea about apparmor)
Check to see if its linking to the expected binaries, that its not using 'hidden' files (starting with a dot or directory with a dot), or deleted files.
Confirm that the process is opening sockets and files that. you expect it to (ie, apache shouldn't open files that are outside its configuration directive).
The process should not be making outgoing socket connections unless it is a client.
It should not be running with capabilities(7) that it does not require. It should not be executing from a setuid binary.
Check the process name, quite often attackers rename the running executable, so you'll see /proc/pid/cmdline renamed with a bunch of null bytes at the end.
Some malware has 'anti debugging' tactics, ie, they have traced themselves to prevent you tracing them, you can find this as one of the lines in /proc/pid/status iirc.
There are more, but thats the few off the top of my head.
or a kernel.
This is a MUCH harder problem, because attackers can always disable any security mechanism assuming they kernel code execution. However, assuming they are not too focused..
If the system is booting in secureboot mode, it should be enabled, and no extra / unused / out of date kernel modules loaded.
I know that code injection at the memory level means that attackers can inject unsigned code, so in this case you would want to periodically sample the code and ensure that execution context would only have the processers EIP in known areas where the kernel would map executable code. You could do an additional check to see if the areas are mapped by userspace processes (it might be too late) so you can find offending attackers.
If the host is virtualized, this becomes easier to do and mapping and comparing memory from the guest kernel for the executable code sections means that its harder for an attacker to work around by being able to disable a mechanism.
Usually attacker kernel exploits do not persist long temr in kernel space, (they abuse kernel space to allow for userspace privilege escalation ie make a binary setuid or modify permissions on a /dev/) because the longer they are there the more likely they are to panic the system.
Some of the more advanced attacks I have seen are from people uploading system kernel panic images, where I have a 'snapshot' of the running system and can work around attackers mitigation techniques.
1 replies
1d5h
Who collects and maintains all these lists of known good/expected configurations? Should the kernel know that apache shouldn't be launched from root? How about autocad, is that ok to be launched from bash? What directories should autocad be reading/writing?
0 replies
1d3h
Seeing how on users' machines the most interesting data to read is in the user's home folder, I'd argue it's actually pretty easy to partition these. Autocad should read and write in ~/autocad. Maybe in ~/Downloads? But definitely not in ~/.ssh or ~/.aws.
Stock Windows actually implements something along these lines, called "protected folders" or similar. It's inactive by default (meaning every program can access every folder). It's quite easy to define a list of "protected" folders. But the implementation is quite stupid: if a program asks for access to one of the folders on the list, you can either refuse, or allow it to access it... as well as everything else on that list!
0 replies
1d6h
[...]
There are more, but thats the few off the top of my head.
And that is probably like 80% what EDR product will be doing, checking that the code that is executing is trustworthy and not doing some weird unexpected things.
0 replies
1d7h
If you have good examples, I'd love to see it, A writeup even more so on the techniques they used. My findings so far in the wild (and on my honeypot) is really amateur level garbage.
I spent a weekend and abused a c&c infrastructure server to fix the clients and remove the flaw and malware. I see very little sophistication there.
0 replies
1d5h
This is a much harder problem than prevention, which is what the OS should be doing.
11 replies
1d5h
There are 2 possible questions.
(1) - Why is a crutch like "anti-virus" software needed? Essentially trying to reactively cat-and-mouse hostile software that the OS has let execute on the computer.
(2) Why doesn't Windows provide AV?
Question (1) is more interesting - and (2) is addressed by other comments.
I think both MS and their customers have very seldom prioritized security over even small compromises in functionality. We loudly blame MS but they are the vendor MS customers deserve. While it's not a democracy, there are parallels to the popular sport of blaming politicians for eg not doing hard choices against climate change while holding the voters innocent.
10 replies
1d5h
The cat-and-mouse game is between OS security features and hackers. AV software is not a crutch, it's an extra level of defense. All OS kernels are vulnerable to malware - this is a 100% given at this moment in history. The question is how to mitigate this problem, and AV is one component of that, as are firewalls, network-level intrusion prevention systems, and a whole host of other security software.
Maybe some day someone will write an OS that is "fully secure" and then they'll be able to confidently run a system whose users can confidently click a link in an email, download an .exe from there, and run it, without fear of losing or leaking a single bit of data. That day is definitely not here, and until then, we all do the best we can through education and security appliances.
3 replies
1d4h
I think the mental model that security is attained by adding more security features just leads to sprawling complexity and awful things like AV.
Secure operating system designs tend to simplify and take away stuff rather than add more bells and whistles.
2 replies
1d4h
Is there an example of a real OS for desktops and servers that is secure from this point of view?
I think SeL4 might qualify, but that can only realistically be used for embedded applications, it doesn't have, at this time, many of the features you'd need to build, say, an HTTP API server for it.
1 replies
1d4h
I think the absence of real world usable secure alternatives is not really strong evidence, operating systems are like web browsers, there's such huge inertia and network effects in the apps that competition doesn't tend to spring up, "build it and they will come" doesn't work.
On the research side there's lots of stuff. Singularity, the various capability based systems, Qubes (granted more towards the adding-features dimension), etc.
0 replies
1d3h
I agree to some extent, but still: if you were starting your own company, would you wait until someone wrote a secure OS? Or would you provide your developers and sales people etc. with an existing OS, and run your servers on an existing OS, and deploy other security tools to mitigate the bugs in those existing OSs?
3 replies
1d4h
I don't want an OS that lets me run executables from email - I've never actually has to do that. I do want an OS that I can tell to run "Firefox, Anki, Thunderbird", once, and nothing else will run.
2 replies
1d3h
Ok, how about an image in an email? Or a PDF receipt? How about clicking a link online? All of these have a serious potential to infect your system with malware.
1 replies
22h10m
I really don't want executables in PDF files or email. Really, I don't.
0 replies
12h15m
PDF parsers, and really all complex format parsers, are very often exploitable. Maliciously crafted documents trigger a buffer overflow, and now they can take control of the process and execute arbitrary code, code that almost certainly has access to your other documents as well.
Also, how about malicious scripts that I convince you to explicitly give execute permissions to and run? How about Git repos that I convince someone to clone, compile, and run, that have malicious code?
Signature-based heuristics can help protect from all of these things that the OS is powerless to help against with only traditional security measures.
1 replies
1d1h
AV software is not a crutch, it's an extra level of defense.
The issue is that it's the only "level of defense" which introduces arbitrary non-deterministic behavior. An executable which correctly follows all the APIs as documented and implemented, and which does nothing malicious, might arbitrarily be denied or even erased, and this behavior changes daily or even hourly due to factors outside the control of the computer's user. Even ASLR, which uses non-determinism in its implementation, doesn't cause non-deterministic behavior when an executable correctly follows the API.
And it's also a "level of defense" which famously causes frequent performance issues, to the point that "tell the AV to ignore that folder" is a common recommendation. I wonder how many gigawatts of electricity are wasted daily due to AV software slowing things down.
Finally, it's been reported several times that this "level of defense" is often poorly implemented, to the point that it can act as a backdoor to bypass other levels of defense. If you can compromise a parser running as SYSTEM, or even within the kernel, you don't have to worry about all the normal rules which prevents you from running code as SYSTEM or within the kernel.
People's dislike of AV software does not come only from some abstract purity ideal; it also come from plenty of negative experiences with it.
0 replies
12h9m
All of these are legitimate issues with AV software. However, they don't mean that AV is a crutch, or that it could easily be supplanted by other security features. There is simply no good alternative to AV for systems where it's likely the user will interact with untrusted input, such as receiving documents, receiving email, browsing the internet, downloading code from GitHub etc
Of course, when you have a locked down system such as a server or an embedded device, the need for AV protection drops down significantly. But on a wide open system, there's really no alternative.
10 replies
1d7h
Windows does have Defender, which does some amount of tracking signatures and heuristics of various types of malware.
It has not, however, proved enough to fend off different real world problems like ransomware.
Hence, the market for 3rd party solutions that are more aggressive. And to keep up with real world threats, they have to update often. And have to run at high privilege levels. So now you have the situation where those third-party solutions have the ability to create a bsod and/or a boot loop. Which should mean that they have a very well thought out way to roll out updates.
2 replies
1d7h
But randsomware is mostly targeted to servers, many of the devices affected were clients
1 replies
1d7h
Is it? I think ransomware affects clients more than servers, doesn't it?
0 replies
1d5h
Yes, or rather, it creeps into systems through workstation clients.
2 replies
1d5h
I read that a lot, but nobody ever provide supporting evidence. To me, this sounds a bit like 3rd party security marketing being really effective.
1 replies
1d3h
There are actual differences, and eval frameworks to get the details you're asking for.
A screenshot of one comparison from Mitre:
You can do more of them here: https://attackevals.mitre-engenuity.org/
It's not a huge difference, but there's a difference.
Also, I have no relationships or investments, etc. Not shilling.
Edit: Also, that url slug from imgur. Heh.
0 replies
23h56m
I wouldn't have noticed the slug if you hadn't mentioned it. Made my day, very appropriate ))
2 replies
1d6h
Microsoft has a high share in this area but enterprise security is generally a very competivite market. Microsoft may even move into #1 position as a fallout from this debacle becasue the market share between them and the #1 CS is very small (that does not mean people actually buy more Ms btw... if that needs to be said ;)
This is not neccesarily a good thing for MSFT as it will 100% trigger regulator rage in the EU.
https://www.statista.com/statistics/917405/worldwide-enterpr...
1 replies
1d5h
Maybe a better market share graphic without a paywall. A little dated, but close enough.
https://www.microsoft.com/en-us/security/blog/wp-content/upl...
0 replies
1d3h
CrowdStrike moved ahead in 2023 for some reason to be #1
0 replies
1d7h
Very much every 3rd party anti-virus software I tried (and paid for) caused data loss or other problems (a few catastrophic) in the long run. One product didn't even stop a virus getting in.
Since then I just use Defender and never had any trouble or a virus or ransomware. Only issue is that sometimes the antimalware service takes a lot of CPU.
4 replies
1d6h
Actually, arguably Windows has some impressive security features unseen on any other mainstream OS, they're just not used by default and - realistically - would be hard to enable on general purpose / non-corporate computers.
For example, by comparison, Linux is in the stone age here.
Do you even need AV if untrusted code can't run in the first place?
* Application whitelisting - with just bare old AppLocker, Windows can be configured to only allow execution of trusted executables, DLLs and scripts by path, hash or software vendor (digital signature). Now, technically AppLocker is not a security feature, i.e. a hard security boundary.
The next level functionality, Windows Defender Application Control (WDAC) [1], however, is. I believe Microsoft was offering up to a $1M bug bounty for WDAC bypasses?
With WDAC kernel mode code integrity enabled, only trusted digitally signed kernel modules can be loaded into the OS kernel [2]. WDAC user mode code integrity provides the aforementioned protection AppLocker provides.
With AppLocker / WDAC enabled, the OS built-in script interpreters (Windows Script Host, PowerShell) either refuse to execute unsigned scripts completely or operate in restricted mode with reduced functionality.
- By comparison, Linux only has fapolicyd which is only supported on Red Hat and can only rely on path-based rules because binaries are not directly signed on Linux. None? of the common interpreted languages (Python, Perl, Ruby, Bash) on Linux support digitally signed scripts and locking down interpretation.
* Authentication material protection - Windows has Credential Guard [3] for protection of authentication material - Kerberos tickets and other material are placed in a separate container protected by hardware virtualization [2] and accessed via RPC so you can't dump process memory to compromise them. Even kernel level compromise is not enough.
- By comparison, Kerberos tickets on Linux reside as files on disk, SSH user & host keys reside as files on disk and loaded into sshd/gpg-agent memory, x.509 keypairs reside as files on disk & process memory etc etc. Wouldn't it be nice to have them protected somehow? To my knowledge, nothing exists for this on Linux.
[1] WDAC - https://learn.microsoft.com/en-us/windows/security/applicati...
[2] VBS - https://learn.microsoft.com/en-us/windows-hardware/design/de...
[3] Credential Guard - https://learn.microsoft.com/en-us/windows/security/identity-...
3 replies
1d5h
- By comparison, Kerberos tickets on Linux reside as files on disk, SSH user & host keys reside as files on disk and loaded into sshd/gpg-agent memory, x.509 keypairs reside as files on disk & process memory etc etc. Wouldn't it be nice to have them protected somehow? To my knowledge, nothing exists for this on Linux.
I have always wondered about that; there has to be a more secure control method for those secrets.
2 replies
1d3h
There is, the TPM. SSH keys can easily be stored and used from there.
1 replies
1d
I can do that as a user? With what utility?
0 replies
22h56m
4 replies
1d7h
I read an article that stated that Microsoft lost an anti-trust court case against the EU in which the EU mandated that they allow third party competitors to provide this service. Microsoft has its own solution called Windows Defender.
https://www.theregister.com/2024/07/22/windows_crowdstrike_k...
3 replies
1d6h
It's more nuanced than that. They have to provide the same APIs to third party security vendors that they use themselves.
They can come up with something more shielded as Apple has done, they just have to eat their own dog food and can't make an exception for defender. That's all.
Blaming the EU here is pure spin.
1 replies
23h50m
And Microsoft doesn't even offer the option of userspace anti-malware hooks, which they could easily do in conjunction with the kernel stuff. I think all they have is AMSI, which is only for scanning PowerShell scripts and such.
If you want to hook process execution or file access, you're writing a kernel driver.
0 replies
23h16m
Yes indeed. But the point they keep making is that the agreement with the EU somehow stopped them from doing this. Which is BS.
They could easily have added a userspace API if they wanted to. It could have existed side by side with the kernel option, as long as they keep using that for Defender too. Only once they stop using kernel access in their own security products can they force the other vendors to use a new API, which makes sense. Otherwise they'd use it as a sales bullet point ("Our product has full system access, others don't"). Which would destroy the antimalware market. The US benefits from this too.
0 replies
21h51m
yes (it's a spin) also e.g. on Linux Falcon could have conceptual created the same kind of driver as for windows but opted to use eBPF
for a lot of things on Windows there isn't anything like eBPF (yet, it's wip, but likely will still take quite a while until it's usable)
the EU spin would only work if CrowdStrict is fully incompetent like a lot of people want you to believe. I.e. they don't do any testing, don't do any config validation and doesn't know what they are doing at all
but that simply isn't true at all
This doesn't mean that they didn't act negligent, as far as we can tell they relied on some data format validation instead by their server + signing (or something similar) instead of _also_ having robust parsing and that is enough against best practices to be called negligent. And there were other points which bubbled up in the last week which point to other negligent behavior unrelated to the bug. But company ending up with some negligent behavior and them being fully incompetent are very far away, let's be honest most IT companies today have ended up with some negligent behavior they have lite direct/short term/fast feedback motivation to fix (hence it doesn't happen)
1 replies
1d7h
Linux can't be secured out of the box to do anything that Falcon does. If you use AuditD, eBPF and things like GRSecurity patches you might get into a good state, but it's still not the same thing at all. it might be secure depending on your linuxfoo, but it's not the same thing as running EDR which will help correlate system behavior across different systems etc. and look with much more depth into process behaviors and system interactions.
Also, you don't want operating systems to provide this actual EDR program. They need to provide the facilities for EDR vendors / creators to tap into and do their work properly. You don't want a butcher to rate their own meat... you want a third-party to do this. As Example: MS Defender is totally rubbish (general sentiment for a lot of people in security, hence they run falcon or cortex XDR etc.) at defending Windows.... and it's by Microsoft. They should focus on building an auditable OS and let auditors do the auditing...
The best thing imho is a tool like CSF but integrated with network appliances (which CS doesn't do i think), which is where the strength of such tooling really comes together, correlating network data / behaviors to endpoint behaviours and having a full 'causality chain' of processes / systems and network traffic invovled in an attack.
And you are right on the balance of security being dramatic. using crypto is still hard as ever, and allowing external parties to interact with your users is just impossible to do right (let alone have users in the right awareness mode). This last is a problem of security industry imho, making tools so difficult.
Someday maybe rather than EDR tools and firewalls, cybersecurity companies will deliver 'secure business services' which are easy to use, userfriendly services that are secure by default. - maybe in like the year 3042.
0 replies
18h6m
I disagree that windows defender is trash. Its’ initial introduction mitigated a lot of malware problems of the early 2000’s.
Sure, it may not be the best, but most vendor solutions aren’t either. Case study: crowdstrike.
0 replies
1d5h
Falcon provides many levels of protection (in principle - in practice, given the extreme incompetence demonstrated in this case, I doubt they do much more than sell snake oil), some of which have OS-native alternatives, some of which do not, and most of which Linux definitely doesn't have built-in. For example, the Linux kernel team doesn't have a DB of known malware signatures that the kernel or init system runs or shell runs any new software component against - Falcon does this. Another example - neither Linux nor any common Linux userspace natively integrates with with a fleet management system to check if the current user is allowed to run a particular piece of software. And there are many other similar questions.
Finally, even when the OS does natively provide services like these (Enterprise versions of Windows do provide all the features I mentioned above), it's perfectly reasonable to prefer a different vendor for those solutions. Maybe people trust CrowdStrike's malware signature lists more than they do Microsoft's, for example: a good reason to buy CrowdStrike instead of using Windows Defender.
I'm not trying to defend CrowdStrike or Windows here. But I think it's obvious that there are many features that fall under the umbrella of security that you wouldn't want to build into the OS itself, and even when a version of them exists built-in, that a company may wish to source from a different vendor.
0 replies
1d6h
Can someone explain to me why the protections that Falcon provides, are not provided by the OS itself?
They are. It doesn't, y'know, do anything. It ticks the box for your auditors and occasionally makes your computers stop running, which is par for the course in regulated environments.
0 replies
1d6h
While Clowdstrike Falcon EDR is in some sense an AV on steroids and Crowdstrike not only does EDR. While they are obviously deployed on lots of systems, less than 1% of Windows systems means it still operates in an absolute niche. Most people didn't know CS even fewer know any of the competitors.
I think one massive difference between CS and AV is also, you don't expect a human to be in the loop because it would be too expensive. Nor would it be feasible for consumer software because of privacy.
Also even within this small niche, the solutions are very heterogeneous and make little sense for single boxes - in fact may even be designed to run on a network level.
0 replies
1d7h
You can do dangerous actions in user space without any need for escalated permissions.
E.g. downloading a file and running the contents as code, or uploading/encrypting all files you have access to.
Crowdstrike and Defender handle those possible but suspicious actions.
0 replies
1d7h
not to defend its "you must accept updates" insane /inane fail, but, the suite of crowdstrike inc falcon stuff we have enables the response side of EDR pretty well, and for a mixed windows, linux, mac shop, where we would like the same agent on all systems, it does a better job than most. Not as good as Jamf on Mac mind you, but better than than most "windows ecosystem". And if you run jamf for policy and detection, but not response, you sort of get it all. So, that's why not "just defender" - at 10k+ systems the anti-malware is just the beginning. What do you do when that fails and ...yeh.. anyway.. there is more to it.
As to why windows is not more locked down- that's on the shoulders of the admins. But out of the box, you are right, it is to permissive. But apparently users and management like it that way.
19 replies
1d6h
I'm not a lawyer, and I'm definitely not a French lawyer, but I don't think the OVH comparison is valid.
In the OVH case, their backup system (as a whole) failed. Many customers were left with 0 data, and per the article "the court ruled the OVH backup service was not operated to a reasonable standard and failed at its purpose".
Meanwhile CrowdStrike "just" crashed their customer's kernels, for a duration of about 1 hour (during which they were 100% safe from cyber attacks!). Any remaining delays getting systems back online were (in my view) due to customers not having good enough disaster recovery plans. There's certainly grounds to argue that CrowdStrike's software was "not to a reasonable standard", but the first-order impacts (a software crash) are of a very different magnitude to permanently losing all data in a literal ball of fire (as in the OVH case).
Software crashes all the time. For better or for worse, we treat software bugs as an inevitability in most industries (there are exceptions, of course). While software bugs are the "fault" of the software vendor, the job of mitigating the impacts thereof lies with the people deploying it. The only thing that makes the CrowdStrike case newsworthy, compared to all the other software crashes that happen on a daily basis, is that CrowdStrike's many customers had inserted their software into many critical pathways.
CrowdStrike sells a playing card, and customers collectively built a house with them.
(P.S. Don't treat this as a defense of CrowdStrike. I think their software sucks and was developed sloppily. I think they should face consequences for their sloppiness, I just don't think they will, under current legal frameworks. At best, maybe people will vote with their wallets, going forwards.)
10 replies
1d6h
It didn't just crash, it crashed 100% of computers running it at that time and in a way that required physical intervention to fix. So I think you can considers this quite different from regular crashes because recovery is much more difficult and because it affected a lot of computers simultaneously.
On top of that there are companies that had failures of their own in their recovery procedures. But even with good procedures this can be a significant outage because it is not trivially reverted and would typically affect many configurations that are redundant for many other failures.
9 replies
1d6h
If the uptime of 100% of your computers depends on a single vendor not writing software with bugs in it, you have a problem.
7 replies
1d6h
That would mean that you always need a fully redundant copy of everything based on entirely different OSes and software with no common component. That is obviously not realistic.
5 replies
1d6h
No. You just need to not update them all at the same time.
4 replies
1d6h
Unfortunately, CrowdStrike decides when it's time to upgrade CrowdStrike software, not the admins.
3 replies
1d5h
1. Your IT department shouldn't buy a product that let a third party change files on your system remotely. This one is the basis of computer security.
2. Your IT department shouldn't buy a product that doesn't give you control on when updates are applied.
These are 2 huge security failures from your IT department.
2 replies
1d4h
Automatic security updates are widely touted as the gold standard in IT security, at least for anything that is not a life-support system.
1 replies
1d4h
I am talking about control, I am not talking about disabling automatisation of updates.
You can have automatic security updates with delay between non prod and prod environments so that you can detect failures or possibly intrusions.
0 replies
1d2h
My understanding is that customers believed they had control as Crowdstrike gave them configuration options to delay updates / stagger them. Apparently many of them were surprised that Crowdstrike had the ability to bypass all these configuration options and force the update. I think that is where Crowdstrike's liability skyrockets through the roof.
0 replies
1d6h
Building high-assurance systems is expensive. Anyone not doing so must accept the associated risks (which is fine, not everything needs to be high-assurance).
0 replies
1d5h
What if the uptime of 50% of your computers does, but you need 100% percent of your computers to run at 100% capacity? If a shop has two lathes and one crashes, and now the shop has 50% capacity, is not losing money because of CrowdStrike's incompetence?
6 replies
1d6h
for a duration of about 1 hour
Not even remotely correct.
Most computers that were affected by the fault needed physical remediation via safe mode boot to fix the issue because they were not able to download a fix because of being stuck in a reboot loop. The understanding is that for most cases, the fix needed to be applied by an IT technician dispatched to physically access the computer.
A week or 168 hours later, there are still many, many computers out there that remain bricked by this fault because it is so heinously difficult to fix.
4 replies
1d6h
See the sentence I wrote just after that one.
3 replies
1d5h
How is it someone other than CrowdStrike's fault that the systems failed again at every reboot until someone with physical access and know-how deleted the crashing driver manually from recovery mode? What should a company operating, say, an MRI machine protected by CrowdStrike have done to recover access in a reasonable amount of time?
2 replies
1d5h
CrowdStrike's software should not be installed on an MRI machine, per CrowdStrike's own guidance:
"Neither the offerings nor crowdstrike tools are for use in the operation of [...] direct or indirect life-support systems [...] or any application or installation where failure could result in death, severe physical injury, or property damage."
0 replies
1d4h
Ok, replace MRI with ATM then.
0 replies
1d5h
If the PC controlling an MRI crashes nothing will happen to the instrument itself. The data might be lost and you can't continue using the MRI until this is fixed, but not more. This would not violate these guidelines.
0 replies
1d5h
For what it’s worth - I got the BSOD, once I got the email from IT with the instructions, it took me about 20min to apply the fix. Almost all of the company employees who were affected were able to easily apply a self help fix.
I could imagine this was not the case if you had to physically access remote servers, or didn’t have access to bit locker recovery keys
0 replies
1d5h
It should not be your vendor that triggers your disaster recovery plans. It should be you know, a disaster, that does.
19 replies
1d7h
Holy shit (hits the fan). For sure CrowdStrike will be held accountable in several countries, but I believe that some conclusions need to be drawn also from a customer/user perspective.
- Is it reasonable to grant such privilege access to a piece of software that ultimately is a black box ?
- Is it reasonable to put a Microsoft / Commercial / Closed source OS in critical infrastructure ? If not considered as critical, then “important” infrastructure ?
- Is it reasonable to have more than 70% of the computers/servers that run important infrastructure on the same OS / software ? How about the mitigation of the risks etc…
I sincerely hope that all of this CrowdStrike mayhem will push stakeholders to draw some conclusions and actions.
4 replies
1d7h
- Is it reasonable to grant such privilege access to a piece of software that ultimately is a black box ?
As I said in the previous thread: explaining to execs that giving root to someone on your machines means they have root is a very difficult concept for them to understand.
2 replies
1d7h
Then the exec should be held responsible?
0 replies
1d7h
The exec just follows the instructions provided by their CISO, who adheres to the information security standards used in audits.
These standards are influenced not only by actual threats but also by lobbying from Endpoint Detection and Response (EDR) systems like SentinelOne and Crowdstrike. For instance, in 2021, the White House issued Executive Order 14028, which mandates the Federal Government to implement a robust EDR solution. Consequently, standards such as those from NIST and ISO27001 have increasingly emphasized malware detection and response.
When onboarding any large enterprise, you will encounter these requirements before the enterprise can proceed with procuring your service. This compels B2B organizations to implement this software to be successful.
^1 https://www.opensecrets.org/federal-lobbying/clients/summary...
^2 https://www.opensecrets.org/federal-lobbying/clients/summary...
0 replies
1d7h
That responsibility (and associated risk) is often the justification for C compensation. Whether that is a good argument I have no opinion.
0 replies
1d7h
How about the good old analogy of giving your house/car/safe keys to a total stranger while going on vacation?
3 replies
1d7h
EDRs are the devil's spyware. Especially since corporate "security" people are now pushing for EDRs to run on Linux. Argument is that the cloud nature of the thing makes it necessary that it runs everywhere. Fact is, since my company forced me to install this black box, my system is definitely less secure. Before that, I didnt have a single incoming port enabled. Now, my system talks to all sorts of external things which I have no knowledge about and no control over.
2 replies
1d7h
If your system was processing any valuable information owned by the company (code, PII, etc) than the company is likely much safer today than it was when you had exclusive control over that system, even if they introduced several vulnerabilities. Previously, if you decided/were coerced to do something against the company's interests, you could do whatever you wanted from that system and they never would have even known. Now, they have some chance to prevent you from doing that, or at least find out in a reasonable amount of time.
Security is a complicated topic, and employees are also potential attack vectors. A system that is in the complete control of a malicious employee is a security problem for the company just as much as a system that was corrupted by an external cracker.
1 replies
1d6h
Well, now we're getting somewhere. If my company distrusts me so much that it needs to put a black box in place to prevent me from fucking it over, it shouldn't hire me as an admin for tons and tons of infrastructure. Distrust goes both ways. Increase the pressure, and maybe, maybe, your employee will just leave for another company that doesn't behave that way (yet). The timing is great, because some employees still remember how they were treated during 2020/21.
0 replies
1d2h
Any company that fully trusts all of its employees to handle my secrets is a company I don't want to do business with. I would bet you don't want, say, every hospital janitor to have access to your personal medical records either. So, you probably also want the hospital not to trust its employees and to keep certain data under lock and key. Same with a bank and your money.
It's no different with software.
2 replies
1d7h
Only your third point makes any sense. For the other two, obviously the answer is yes, that's entirely reasonable. Businesses and government organizations use plenty of commercial tools that they have no way of designing or understanding on their own. Software is no different from hardware from this point of view.
A hospital doesn't have, and couldn't use even if it did, the blueprints for an MRI machine or an old-fashioned iron lung. And those machines are built by commercial companies and contain plenty of trade secrets.
If anything, using open-source software that you maintain yourself in critical infrastructure is the more bizarre practice from a historical or industry-level perspective. Even in software, things like Solaris, IBM OSs etc. are much more common than OSS. And even when using FOSS, a commercial distribution like RHEL is far more common than using your own Linux.
1 replies
1d5h
But do we really need "trade secrets" as a society?
0 replies
1d2h
Even if companies were forced to publish every detail of their devices (which is the only way to not have trade secrets), any decently complex products products would still be black boxes to every company who is not specialized in creating them.
Even something like a fountain pen is used as a black box, I'm not even talking of anything truly complex. Even the buildings we work in are black boxes that we get from third parties, not to mention all the systems powering and heating or cooling them.
1 replies
1d7h
- Is it reasonable to have more than 70% of the computers/servers that run important infrastructure on the same OS / software ? How about the mitigation of the risks etc…
This is the problem as far as I'm concerned. Industry "best practice" is "use the same thing everywhere"
A diverse ecosystem is the best defence.
You could run 100% FreeBSD and be hit by say a hidden kernel bug which occurs on Jan 15th 2027 when unix time goes from 1.7b to 1.8b (I've seen that code before where time is assumed to be below X)
If you run 50% FreeBSD and 50% Windows you will only lose half your service.
0 replies
1d6h
You would have a hard time denying 20% of users their first choice.
1 replies
1d7h
None of them is reasonable. Open source and regulation on software safety is required. The society at large has been too lenient with poor quality software.
0 replies
1d2h
And the solution to that may be worse. Do you want to saddle all open-source with strict regulatory compliance on safety?
0 replies
1d7h
Alas, you didn't need a global incident of this scope to draw those (perfectly valid) conclusions.
The hallmark of intelligence is to observe a situation and the structure of a system, reason about it, draw analogies with past experience and pre-emptively take corrective measures.
The stark truth is that we don't live in a "reasonable" world.
Poor governance, short termism, lack of transparency, incompetence, captured regulation, obsolete ideology etc. are not exceptions but rather the essence of how things "work".
The existential question is whether our demonstrable ability to achieve some learning will be sufficient to deliver solution on the face of increasing risks.
0 replies
1d7h
Is it reasonable to grant such privilege access to a piece of software that ultimately is a black box ?
According to Microsoft its not but they were forced to. Interesting how the EU executive is now getting mixed up in this saga: https://www.euronews.com/next/2024/07/23/european-commission...
0 replies
1d7h
Is it reasonable to grant such privilege access to a [ company ] that ultimately is a black box ?
This is common enough in the corporate world and precedence in similar circumstances will come into play in various lawsuits.
Examples:
XYZ Security Guards: a third party physical security provider that hires people to watch and patrol buildings, assets, with access to keys, timetables, security logs, etc.
ABC Armoured Transport: third party physical transport provider for cash, sensitive documents, etc.
When AcmeCorp Inc. hire XYZ & ABC it's on the basis of reputation, contracts, and things generally not to do with peeking inside how the cake is baked (hiring records, etc).
15 replies
1d6h
French here, and working for another french CSP. We lived the OVH incident live and saw the whole aftermath.
OVH was held liable because of the data loss, not for the service interruption. Data loss is something irremediable, permanent, definitive. Some businesses were basically ruined from this incident because they had no more data to operate. To add insult to injury, they sold offsite backups in the datacenter literally meters away. A service interruption, well, shit happens, and this is handled by SLA contracts that both parties agree to. You don't ruin a business (read: close a company) for a few days of outage.
I doubt CrowdStrike will be held liable for much; from corporations at least. They cannot repay the damage done, or they close the door. The healthcare sector is another beast, but I think it will come to more regulations for critical entities.
12 replies
1d6h
IMHO it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors...
like if the largest outage in history was caused by you due to a config parser failing and it looks as far as I can tell that they didn't follow industry best practices when it comes to config/parsing handling and probably also didn't follow some best practices when it comes to kernel module programming then honestly it would be really strange if you didn't had to declare bankruptcy due to damage payments (which doesn't mean the software is now gone/unmaintained, there are a lot of ways to make sure that doesn't happen, e.g. MS anyway had interest in buying Falcon).
5 replies
1d4h
I understand that CS doesn't draw much sympathies even before this happened - from myself included - and it is easy to pin point systemic issues to a single failure point and make it liable and financially responsible for all downstream failures.
But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses.
I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?
2 replies
14h58m
In the end it's a matter of cost. Having two independent supply chains for anything as complex as healthcare infrastructure is essentially unaffordable.
0 replies
10h12m
essentially unaffordable
[citation needed]
0 replies
5h14m
Can't help but think that from this perspective buying a solution from a vendor is less about solving a problem and more about buying a blame-insurance in case something comes crashing down.
1 replies
1d1h
I read that healthcare was mollified by a facility whereby they could choose when and which updates to allow, which CS intentionally bypassed with this latest update. This was during the chaotic day-of though, so possibly false.
0 replies
10h9m
There can always be lingering bugs that trigger at a certain time on all systems or for certain input data so the only way to have redundancy is to have actually heterogenous systems.
2 replies
1d4h
They didn't follow testing or deployment best practices either.
1 replies
1d4h
IIRC their QA team was impacted by the most recent round of layoffs. Dumping those responsibilities onto devs isn't a great solution to begin with, and especially not when the product is a complete trash fire.
0 replies
21h40m
yeah you really shouldn't layoff a QA team if you do something like that that's just pure negligence
(through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)
1 replies
1d5h
it would send really wrong signals if this doesn't end up with CrowdStrike closing their doors
I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.
0 replies
1d4h
That's only a low-ball for fortune 500, it's a small part of the damage they done.
0 replies
1d3h
I don't understand why there is so much attention on the deployment and testing side of the coin. Yes, better testing and rollout strategy should have prevented this specific occurrence of a failure. But these strategies aren't bulletproof and things go wrong. You need defense in depth, and some responsibility has to lay on the consumer side for that to happen - particularly for fundamental humane industries like transportation and healthcare. These industries should not be allowed to run any software like this - privileged and without controlled rollouts. I'm all for shaming CrowdStrike's lack of focus on reliability, which they deserve, however there's a bigger issue here of trying to avoid or mitigate risky dependencies in the first place that I hope we also get to explore.
0 replies
8h3m
As I understand it, anyone with bitlocker irretrievably lost all their data.
Logic dictates that the more critical the data - the more likely it is gone. An informal anecdotal survey says that alot of users use bitlocker - which means a lot of data loss.
EDIT: I see that in many cases one can recover Bitlocker encrypted drives. I wonder how much real data loss there is.
0 replies
1d6h
If a business closed down because of the OVH incident, how were the damages calculated? 1x annual revenue? Profit? 5x?
15 replies
1d6h
What is hilarious to me is how the US government or courts doesn't seem to give a shit about this.
Corporativism in US is a thing. Companies can brick hospital systems killing patients, drive self-driving cars and run over people but don't get sued, and if they do, they settle for very little.
Just look at the recent Boeing incident where people were killed, the company clearly misled the US authorities and settled only a $0.5B fine.
Those companies in those scenarios should pay the fine that they should ($20B+), and if it means the company would go bankrupt, do it and form a new company diluting the previous shareholders.
Without doing this, shareholders and CEOs will have the incentive to carry on with their unfair practices that leads to dead people and deadlocked systems.
7 replies
1d6h
Just look at the recent Boeing incident where people were killed, the company clearly misled the US authorities and settled only a $0.5B fine.
The problem is when you fine a company, they will just turn around and offload that cost to their customers. Which in this case is the US government in a very large way. Boeing will make their part in the SLS a few billion more expensive again to offset it and even gain some profit. The US government are just fining themselves.
Fines just aren't an effective deterrent for companies. They should go back to imposing personal sentences on their leadership. But this is really unpopular because these guys are so well connected. So basically nothing is done and everyone goes free.
3 replies
1d6h
That only works when the company has full power to set prices unilaterally, i.e. when it has monopoly power. Which is a separate problem that should be prevented separately. If Cisco gets fined a billion dollars, it can't just hike up the price of a router, as it will lose plenty of business to Juniper/Arista/F5/etc.
2 replies
1d5h
They can in the short term because they have tons of companies way too invested in their ecosystem to change.
See what VMWare did after the broadcom takeover. They did exactly that.
1 replies
1d4h
Then what's stopping them from charging more today, in the absence of any fine?
0 replies
23h15m
Probably that they care about their customers, unlike Broadcom who just wants to milk the fat whales and isn't interested in the small fry. In other words: long-term vision over short-term gains.
1 replies
1d6h
You could force the company to pay the fine in the form of a % ownership stake in the company. Then if the company raises prices to hose the government, the extra profit flows back to the government in dividends.
0 replies
1d6h
That actually sounds pretty compelling...
0 replies
1d5h
The problem here is that "the company" is liable for the negligence of the person in charge. The person in charge is not liable for anything. And like you said, if the company has to pay, then the money obviously comes from its customers, not from the person in charge.
3 replies
1d6h
what if the fine was giving up some shares to the government? With such a rule, after enough fines, the company would basically automatically become a public company.
1 replies
1d6h
Sounds too much like socialism to go anyhere.
0 replies
1d6h
It depends on who you want to punish, the shareholders, who have to give up their shares ... or the company, which doesn't really give a crap, as a whole.
0 replies
1d6h
what if the fine was giving up some shares to the government?
What is the advantage over just fining? We keep trying to reïnvent the fine, which is great for those who would otherwise be fined.
1 replies
1d6h
I think you meant US companies. The fine for Volkswagen was $20B+.
0 replies
1d2h
That's why the EU needs to kick out big tech and be much more unfriendly to US companies, have its own atomic bombs and security and not be dependant on a country that every election it's foreign policy completely changes (Democrats vs. republicans)
It feels like US can hit as much as they can EU companies, but EU needs to create a whole new regulation to slap a $1B fine in a 2 trillion company from the US.
0 replies
1d6h
If you want this to change, ban the CEO from holding a similar job or sitting on a board. If that doesn't work ban them from Aspen.
10 replies
1d7h
The 10$ gift cards were just hilarious. How could they possibly expect anyone to take them seriously?
3 replies
1d7h
Not a lawyer, but the cynic in me assumes that it is legal bait: if someone at the company cashed the $10 gift card one could argue that compensation for damages has been accepted and no further liability applies. At least legally speaking, obviously this is completely morally bankrupt.
0 replies
1d6h
Maybe this would work if the damage was $100.
But you would not get away from $mil of damages with a $10 voucher. The courts are not dumb and don't work like that.
0 replies
1d7h
Any normal legal system should have an option for avoiding this, like "reasonable" compensation (reasonable would then be argued in court but I'm pretty sure you can find a lawyer who can argue a uber gift card is not).
0 replies
1d7h
I hope that happens to every company that simply updates terms of service by sending email...
1 replies
1d7h
wait, that was not a joke?
0 replies
1d7h
Sure wasn’t. $10 Uber Eats card
1 replies
1d7h
Was this possibly some way to influence liability limitation? If you accept a $10 gift card, could that be argued as an acceptance of compensation?
0 replies
1d6h
Hopefully judges aren't that stupid / our legal systems so easily gamed
0 replies
1d7h
I read somewhere that the 10$ gift card was for crowdstrike partners who are working to fix the issue, and not it's customers.
0 replies
1d7h
I'm having a really hard time lately distinguishing between memes and actual news. The news sound a lot more meme like than the actual memes.
9 replies
1d7h
Surely, there must be a gigantic number of claimants already taking to their lawyers about how to get compensation? Not just in France but across the planet?
I wonder how this kind of thing is organised, since there's all these jurisdictions.
8 replies
1d7h
"I wonder how this kind of thing is organised, since there's all these jurisdictions."
In theory simple. Crowdstrike is doing buisness in state X, so compensation claims will be settled in court in state X. So lots of courts and lawers all around the world, will be quite busy for some time with the case.
4 replies
1d7h
It's a B2B tool, which means it's quite likely the contract/license states that all disputes are to be settled in a court appointed by them. This is not valid for consumer disputes, but businesses are free to do what they want. Perhaps this will let them off the hook?
OVH is different in that it's actually a French company.
0 replies
1d6h
It's a B2B tool, which means it's quite likely the contract/license states that all disputes are to be settled in a court appointed by them.
Many businesses will simply refuse to buy your product if the contract says the dispute has to be settled under foreign law or by a foreign court. The customer's lawyers will flag such a term as an unacceptable legal risk. And if your competitor isn't demanding that term, you are giving them a big reason to choose the competitor instead.
Random example: Oracle's standard contracts with their Australian customers says disputes will be settled under Australian law (New South Wales state law) in an Australian court (in Sydney). [0] And Oracle's standard agreements for France nominate French law and the courts of Paris. [1]
If Oracle can't get away with forcing foreign law/courts on their customers, I'd be surprised if CrowdStrike can.
Might be a different story for smaller countries, but most businesses in major economies are used to vendors offering contracts under their own national law.
[0] for example https://www.oracle.com/us/corporate/contracts/cloud-csa-v012... – see clause 14 on page 7
[1] for example https://www.oracle.com/assets/cloud-csa-v012418-fr-eng-44198... – see clause 14 on page 6
0 replies
1d6h
Many companies are actually not us companies.
In the end probably only the us companies will be left empty-handed.
0 replies
1d2h
but businesses are free to do what they want.
this is not true
just because you write into your contract that something will be settled in a specific jurisdiction doesn't mean it's legally actually the case
0 replies
1d6h
In several European countries, those parts of a license that put the customer in a worse position than what the law stipulates are simply invalid.
1 replies
1d7h
And if judgements are found, local to the jurisdiction assets (including and money being forwarded by banks, eg via credit cards, or wire transfers) can be seized.
If that doesn't work, judgements can be registered in other courts for collection purposes.
0 replies
1d6h
Many countries also have treaties to accommodate this process across borders.
0 replies
1d7h
I cannot see how they will get over this... It's CIO snakeoil to begin with, but this was not a simple mistake; it shows the entire lack of process and responsibility.
9 replies
1d7h
Awesome. Falcon has been widely known (for years) as an utter piece of shit (code wise).
Maybe now ClownStrike will start testing it properly, hopefully thereby fixing the stability and other issues.
3 replies
1d7h
Likely not by the people who make the decisions to purchase this. They're usually hearing more from the marketing/sales people than from those actually having to deal with this.
Personally, I don't expect this to make much of a difference, if any.
2 replies
1d7h
Personally, I don't expect this to make much of a difference, if any.
While you're probably right, I'm hoping ClownStrike's court results so absolutely dwarf their insurance coverage that it's nearly company ending.
ie something to actually get them to improve things, not just generate empty PR platitudes: https://www.youtube.com/watch?v=SiL2AjOtjZI
1 replies
1d7h
Meh. We (techies) always knew the risks of running random crap in kernel space, especially when it runs junk it downloads from the internet.
So, I expect this to be spun somehow along the lines of "sure all our boxes were down, but look, you've brought them all back up, didn't you? Now think about all the bad guys this protects us against! Of course the risk was worth it!". Also, "everybody does this! we couldn't have known!"
"security people" are scared shitless of the whole "the world is ending! there are threats everywhere!" discourse that vendors peddle. The less technical, the more scared they are.
Many of these people don't really understand what they're talking about and what compromises their decisions actually imply. Losing a day of work is simply dwarfed by "all your data is gone!".
0 replies
1d6h
I like to poke the security bears at my company by suggesting that the security team is actually a double agent team whose real employer is our competitor. They never find it funny for some reason.
1 replies
1d4h
Awesome. Falcon has been widely known (for years) as an utter piece of shit (code wise).
Right, but other commenters call it the best EDR out there; so it is really hard for those of us outside the loop to understand what the hell is going on. Is CS, or any other EDR, actually preventing attacks that would pass through if absent? To what extent? Where are the numbers? Who audits CS code? I have seen no real data, only assertions.
0 replies
1d4h
Yeah, I'm as surprised as you that people have been saying ClownStrike Falcon was good.
I guess the people saying that are security folk who look at things from a high level place of some sort (?). Because they don't seem aware of (or don't care about) the many problems it causes on the servers it gets deployed to.
As we've now had amply demonstrated. Globally. ;)
1 replies
1d7h
The problem is that you think that those managers are nice and reasonable people like yourself. They are not. What will happen is that some manager will yell at some other manager that will yell at some other manager that will yell at some tester that works on the cheapest virtual machine possible, on which it takes 5 minutes to log in and it disconnects after 2 minutes of idle. All the while, not changing anything.
I'm pretty sure that all their resources are allocated to lawyers right now and their managers try really hard to gaslight their customers by telling them it was not that bad, they came up with the fix really fast (ignoring the fact that the fix was not possible to be applied) and so on.
0 replies
1d7h
hopefully the lawyers are demanding payment in advance
in many countries there are very strict limits on excluding liability for negligence
0 replies
1d6h
It's CIO magazine snakeoil. You don't buy it for any other reason than tick off some risk thing without knowing/thinking about it. Everyone does it, so he. Terrible.
4 replies
1d7h
I was aware of this being the case when dealing with consumers, but had assumed that because B2B contracts are assumed to be between 2 sophisticated parties that there is little legislative protection that could override the terms of the contract.
My understanding of law is generally UK based, but I'm not aware of legislation what would supersede a contract term limiting liability when the event that created the liability was one of general diligence/competence in carrying out the contract rather than relating to health and safety or some other area that is heavily legislated.
For that reason I'm unconvinced on the article's statement that this isn't just a "French Legal System" thing and that the same kind of judgement might be made in other jurisdictions.
1 replies
1d7h
As the article already states, in most jurisdictions you cannot void gross negligence liability in contracts. It will probably come down to that in those jurisdictions.
If they willfully did not implement staged rollouts that look like negligence to me but ianal. You kill canaries for a reason.
0 replies
1d6h
Kind of looks like they had no canaries. https://securityboulevard.com/2024/07/crowdstrike-pir-canary...
0 replies
1d6h
I think the general idea is that gross negligence is a breach of contract. Every contract implicitly assumes that both parties are making a good faith effort to honor the terms of the contract. If you are not doing that, you may be in breach of contract, and the liability limitations may no longer apply.
0 replies
1d7h
Well for starters it did impact health and safety domains; hospitals and emergency services were severely degraded. There absolutely will be preventable deaths directly traceable to Crowdstrike.
3 replies
1d6h
This headline is kind of misleading. It's actually someone's personal (educated) opinion on a blog, not a statement of fact. Should be something more like "I think CrowdStrike will be liable" or "CrowdStrike should be liable"
2 replies
1d6h
the full headline (at this time at least) is more nuanced than seen here in hn: CrowdStrike will be liable for damages in France, based on the OVH precedent.
0 replies
1d4h
Doesn’t really make it any less misleading. It is still just an opinion.
0 replies
1d6h
It's also in the URL. Submitter, please don't remove important parts of headlines.
2 replies
1d7h
"It is not an isolated incident. The same thing happened few weeks earlier with the CrowdStrike agent on Linux, nuking the system and there may be other occurrences before."
Is there a link with this incident?
0 replies
1d7h
"CrowdStrike broke Debian and Rocky Linux months ago", https://news.ycombinator.com/item?id=41018029
"CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes", https://news.ycombinator.com/item?id=41030352
2 replies
1d7h
time to issue 50€ gift cards!
0 replies
1d7h
...that "ne fonctionne pas"
0 replies
1d6h
Time to settle in a jurisdiction where gift cards are a legal currency and only do business within there
1 replies
13h41m
What, the $10 gift certificate for customers isn't enough?
0 replies
12h17m
Virtual gift cards offered by Cybersecurity firm CrowdStrike to those who aided customers through the global IT outage have been blocked and flagged for potential fraud by Uber.
https://www.abc.net.au/news/2024-07-26/crowdstrike-gift-card...
I almost feel sorry for CrowdStrike.
1 replies
1d6h
This article feels like it was written or augmented with an LLM.
0 replies
1d5h
I didn't get that impression. Were there any particular "tells" you spotted?
1 replies
1d7h
I wonder what happens if the damages exceed whatever assets they have in France.
0 replies
1d7h
They have at least a B.V. with assets in the Netherlands and usually that one contains money for "tax reasons" (e.g. avoiding taxes), and they can lay claim on that.
1 replies
1d7h
Does CrowdStrike do any testing whatsoever? Obviously they didn’t or the incident wouldn’t have happened.
Eh, parts of this article aren't very reasonable. Even if they did a buttload of testing, it only takes one failure in one part of the chain (near the end).
They didn't test something they should have, sure, but obviously they didn't do "no testing whatsoever"
0 replies
1d6h
Deploying untested changes isn't "near the end of the chain", and it voids any buttload of testing of something else.
1 replies
1d7h
This is the way
0 replies
1d7h
Episode 5: The Crowd Strikes Back
1 replies
1d6h
*might be liable
And if France comes down hard on them, they may simply not do business in France.
0 replies
1d6h
If they are liable they maybe go out of business globally.
1 replies
1d7h
And yet there is no mention on the end-customers Change Management and Patch Management practices. Who pushes an update on 1000-5000-10000 machines without testing it?
To whoever does this I have only one quote from Jaws:
You go in the cage, cage goes in the water, you go in the water, shark's in the water, our shark. Farewell and adieu to you, fair Spanish ladies. Farewell and adieu, you ladies of Spain.
0 replies
1d6h
Who pushes an update on 1000-5000-10000 machines without testing it?
No-one is seriously claiming CrowdStrike did that.
0 replies
1d7h
Sorry, but I feel the author is reaching for a conclusion.
From OP, in the OVH-case liability seems to override the contract / waivers when OVH was both the storage And backup provider and did not actively underline that this solution is suboptimal, in a situation where multiple data centers are physically very close. That's a chain of evidence.
For CrowdStrike, it is clear that the offering is to more mature counter parties (thus raising the B2B standard of evidence) and that CrowdStrike very essentially did not do / support staging, whatever. This is indeed bad industry practice, but one that can thought to be explicit from the start of the agreement. At least in my locale you either make explicit agreements OR industry standards are leading. We do not do industry standard X is pretty clear. Read the list in OP, replace CrowdStrike with Microsoft and then think of the international liability cases you've heard from where Microsoft was found liable for downtime, hacks and other issues.
Look, liabilities will always arise in such situations. But I expect only minor liabilities will arise. Mostly (AFAIK IANAL) the terms & conditions are applied in B2B-cases. This case is pretty obvious: you got what you signed up for. CrowdStrike with full scale access to your machines and no guarantees. On the other hand, Crowdstrike lost 125 billion in market cap. That's an indication of {liabilities + loss of future profits}. Pretty massive event for not being willing to do staging. But I expect it's mostly that CrowdStrike is tainted from now on. A friend of mine had a very bad stint as an employee of CrowdStrike recently and from what I learned from that case, I'm happy that the nature of the firm is somewhat more in the open now.
0 replies
1d6h
Why does article say "On 19th July 2019, CrowdStrike pushed an update" ? Is it another incident in the past, same as OVH, or a typo? I'm kind of lost in context
0 replies
1d6h
Sounds like a positive one for insurance industry.
0 replies
1d7h
Great! This kind of stuff will finally make companies start taking quality seriously.
0 replies
1d6h
I'm actually surprised the damage value I'm hearing about is not even $10B , I guess most of the downtime was on the weekend, but such a large scale 1-3 business day outage I'd think would a lot more. or perhaps it is because most small and medium businesses don't have crowdstrike because it is too expensive and they were not affected. Or another reason might be, indirect losses like the impact of delayed flights on individuals is not being considered.
I think if the total liability for Crowdstrike is less than a few years worth of revenue, they'll come out unscathed because as I understand, they are still not profitable, their valuation is purely on speculation on future revenue. Their biggest paying customers still care a lot about getting compromised, it isn't just a box checking exercise like many have suggested.
0 replies
1d5h
Another point against CrowdStrike: they did not have any "try once and if it fails, stop trying" logic. It cannot be the first time any CrowdStrike engineer saw the crash loop phenomenon. And so, a professional would have filed a high priority bug saying, "we need a way to stop crash loops definitively and automatically".
That would have been literally the headline I'd choose for the bug.
This is incompetence that in a just world would result in the corporate death penalty.
0 replies
1d6h
CrowdStrike will be liable for damages in France
...based on the OVH precedent
0 replies
1d6h
not just in France
most(all?) EU have laws which limit how much you can opt out of liability _no matter what you write into a contract_
while I'm not sure about the exact boundaries per country but I'm pretty sure that at least all hospitals, emergency call services etc. can sue for a non-negligible part of the damages that outage caused directly
private people which where harmed by not getting operations done in time most likely can also sue them for the full damages caused to them (through it's hard to assess the damages and it might need to be indirectly by suing the hospital and the hospital sues for more damages)
what you likely will not be able to sue for is the lost opportunity cost, the man power needed to fix it etc.
also my guess is that for a lot of cases which are not as sever as human damages or as indirect as lost opportunity cost a huge factor will depend on the degree of negligence judges believe happened. And here "negligence" isn't limited to the specific change which caused the bug but also if they kept they due diligence in choices of tooling, approaches, business processes etc. to reasonable minimize the risk. (like e.g. was their way of parsing configs inadequate/did it follow industry best practices (IMHO it doesn't seem so), or was it adequate to mark the driver as required to allow boot (else windows would have auto disabled it and then restarted) etc.)
0 replies
1d7h
Good. Without consequences that hurt the perpetrators nothing will ever change.
0 replies
1d7h
On 19th July 2019, CrowdStrike pushed an update to their software.
I assume the year was meant to be 2024.
0 replies
1d6h
Complete title: CrowdStrike will be liable for damages in France, based on the OVH precedent
0 replies
1d4h
How deep does liability of a electricity provider go when they have major power outage? even if due to gross neglicence? would they be liable for all downstream failures including loss of life?
The number of US tech businesses that are surprised they need, or think they can ignore the need, to obey employment and data protection laws when working in other jurisdictions is simply bonkers.
Well, it'd be a lot easier if most US entities understood that M/d/yy(yy) format is rare, or that default to Frankenstein degrees is pretty much the same/awkward (even Microsoft reset their weather widget to F on regular basis).
The root of issue, not understanding local laws/culture, is very similar - surrounded by a vast market/culture (US +Canada) dulls your senses for the rest of the globe.
> surrounded by a vast market/culture (US +Canada)
US companies don't think about Canada as anything but an afterthought, and struggle with the same issues here that you just mentioned.
Canada is metric, uses different spellings (closer to UK English.. colour, not color [Chrome just marked my spelling as wrong despite me having Canadian English as my setting]) and is officially bilingual with localization laws requiring companies to provide French versions for certain kinds of services. US companies either don't bother, or usually get this entirely wrong.
e.g. It's been how many years? And navigation on Android / Google Maps can't pronounce French names for streets/places while driving around in bilingual places in Canada. Just completely butchers them, their system can't "understand" the concept that you could have your language set to English but still need to hear French place names, or vice versa.
Honestly, I think this is the right approach, and I'm speaking as a bilingual French/English speaker.
Google Maps doesn't know that you are bilingual. So it has two choices: pronounce words the "right" (i.e., native) way, or pronounce them the "English" way. If someone who is unused to hearing the native pronunciation, their understanding is going to be impeded. Google Maps tells you for example to turn onto Passage du Grand Cerf in a French accent, a non-Francophone is going to have a harder time finding that.
(Now, okay, maybe you should have a way to say that you're bilingual and want to hear the French pronunciation, but I understand why they don't do it.)
But it does know, e.g. "Accept-Language" header. Of course, google resent that part and travelling across Europe results in having a different language every day.
Google outright ignores Accept-Language: en-US and randomly resets your cookies to use the local language based on your IP. They've been hiding the link to change back to English more and more too and have started to show local language search results above english ones (e.g. for Wikipedia) even with the interface set to English. I have nothing but the worst wishes for those responsible.
adding a param "hl=en" (your language) in the URI query is what it takes, it has been this way for around two decades (iirc)
I'm not bilingual but I'm British and the most irritating thing driving through France is hearing Google/Siri butcher French words. It makes it basically impossible to navigate!
I imagine most Brits would be able to hear a French word (spoken slowly enough) and recognise it written down.
I agree. I would guess it might not be that difficult for Canadian people as they are all to some degree exposed and as such familiar with both languages' sounds. But I remember the first couple times I was in China a good decade ago, it was much easier for me take the older subway lines than the newer ones in Beijing.
On the old lines, they had what sounded like a native English speaker do the announcements of the upcoming station, including the station name. On the newer ones it sounds like they just recorded the "we're arriving at" bit once and then spliced in the read-by-a-Chinese station name, which is much harder to understand as a foreigner. So on the new lines I was constantly staring at the displays.
Huh? No, when driving in Quebec or Ottawa etc, there might be an "English" way to pronounce the names (like the kind of thing my anglo self would use, bad pronounciation but making an attempt), but it doesn't do that... it actually just reads out this weird literal phonetic output as if it was an English word and it's unrecognizable to anybody from the area or anybody who grew up in Canada and has a basic idea of how these words are pronounced.
Even, like, names ... Duplessis or Cartier or whatever... just completely unrecognizable.
Or take the way it reads out the exits off the 401 in Toronto, reading the full bilingual sign (where there is one) with the English first and then the French part as if it was a continuation ... in English ("Sud" read out like you'd say that if it was an English word, hard D and everything.) Clearly, they should just drop the French part, but their system can't comprehend that a sign could be bilingual. Because America.
Look, I don't speak French, I'm not bilingual, and it drives me nuts when I'm in QC or even just Ottawa etc. I literally can't tell what street it's talking about, it doesn't correspond with the sign. It's incoherent.
What's mind boggling is that Google navigation is completely fine reading out Spanish words when I drive around California. C'mon.
There are plenty of English place names with surprising pronounciation too. I don't think out software should encourage us to stay ignorant of the proper pronounciation - it's not like the map software is the only place wehre you are ever going to encounter spoken place names.
Circular reasoning. If common software used the right pronounciation more people would be used to it.
The spelling of +Canada is very much on purpose (no space). I am very much aware Canada is metric and bilingual.
Due to the proximity of US (tooling/documentation), lots of the trades still operate in non-metric (or some unholy combo). Most folks would be a lot more familiar of PSI when it comes to tyre pressure (compared to bars), etc. I don't know if L/100km is the standard unit to measure fuel consumption (efficiency).
L/100km is standard, especially because our roads use km measurements. I use both KPa and PSI. Measure my personal weight in pounds, but doctor and etc work in kg. Trades people work in both, by necessity. (My father was a machinist / tool&die maker who trained in Germany... drove him nuts).
It's just the reality of "sleeping with the elephant" as the expression goes here. We just have it worse than Europe.
Google Nest prononciation of EN songs names in middle of sentence in other language was fun to ear. It’s been a wile it does pretty well (way better than at the beginning at least) so they probably have the tech to achieve street names prononciation.
As I said elsewhere, it handles Spanish place names etc perfectly fine while driving around California. So, comes down to motivation / $$ it seems.
I acquainted with a guy at a conference in US and he was genuinely surprised I had no idea, how long US mile is. I explained him, we use metric system and his response was “but don’t you learn *the standard* system in ache school?” I did not know, how to respond.
It’s just a difference in travel and seniority. If you aren’t talking across continents there is no need to speak two languages.
Continents have nothing to do with this. If you live in the UK and need to talk to people in the USA and Australia, you can be monolingual and still speak with people in three continents. If you live in Switzerland, you may need to speak 3 languages just to be able to talk with all your neighbors.
Languages as in knowing two systems. Sort of like how most people don’t need to know international date or thousands/decimal separator conventions, but those functioning internationally—whether due to being well travelled or senior enough to conduct international trade and/or relations—do.
My going to a conference in India and arguing over the lakh/crore system isn’t useful to anyone [1].
[1] https://en.m.wikipedia.org/wiki/Indian_numbering_system
Even then, continents have little to do with it. The Indian numbering system is indeed used in much of Asia - but it's not used in Russia for example. If you live in Vladivostok, you might need to learn these two systems even if you never do business with anyone farther than 300km from you.
And in Europe there are numerous differences between countries of this kind - Germans and a few others use different number separators (1,000 is 1000 in France or the UK or Spain, but 1 in Germany or Romania). Several places drive on opposite sides of the road. The UK uses many imperial units. I'm sure there are others I haven't even come across yet.
No, France and Spain follow the same standard as Germany: dots as thousands separators, a comma as the decimal separator. Actually most of Europe does the same, the only exceptions being the UK and Ireland:
https://en.wikipedia.org/wiki/Decimal_separator#/media/File:...
Got it, you’re parsing continents literally. I was speaking colloquially. Read it as “cultures” in the first comment.
There is also an abundance of people who don't speak English, or prefer not to, right here in North America. The Canadian province of Quebec, for instance, legally mandates bilingual signage and generally prefers French. And Mexico is right there too.
There are also a great many families in the US whose first-generation members have limited English.
There are also many of us who have family in the US since the 1860's and still speak (Alemannic) German at home and in the surrounding community.
This also goes notwithstanding the indigenous peoples, whose Diné bizaad and Tsalagi, for example, are also spoken here.
AFAIK USA people learn both systems in school. So it understandable if they are not aware that the rest of the world don't know about their system, miles, inches, feet, gallons, pounds, etc, unless they are into American culture (books, movies).
One of the things that USians are taught in school when they learn these two systems is that everyone else uses metric. We're typically taught it in our science classes, because even in the US, scientists still use predominantly (exclusively?) metric. It follows that there's no parallel reason for most foreigners to learn US imperial units, especially in an institutional setting like public school.
I am not sure it's fair to include Canada in the same basket. We don't use freedom degrees, we know that numbers should start with the most significant digits and I believe liability waivers have no value here as well.
Am curious: where does that not happen in the US?
(And in parts of Canada they say 4-20-10 to mean 90 :-)
I know I am in Quebec and speak French. I would prefer the Belgian way of octante/nonante. L.
It happens with dates and makes them unsortable.
I did respond to another comment. The punctuation was very much on purpose, "+Canada", w/o the leading space to denote that there is a separation.
I always prefer, default to, and advocate for yyyy-MM-dd format for dates to avoid confusion. Once you're used to them, no other format will do. Also, files with such names (backups, etc) will also be in a natural order.
RFC 3339 / ISO 8601 FTW!
The worse offender on this I suspect it's Apple. Where half of their localization stuff doesn't work or works in a weird way
It even affects open source projects like KDE: https://bugs.kde.org/show_bug.cgi?id=340982
I thought Frankenstein degrees was a clever complaint about us companies wanting degrees for tech jobs
That's Fronkensteen.
It's not unusual in the US to assume the US are the only planet in the universe.
Special mention of the expression “the west” which Americans like to use to mean the USA and some amorphous blob I don’t really want to think about but I’m going to pretend is exactly the same as the USA.
Somehow related, expressions like "next summer", "starting this spring" and such on public global announcements make absolutely nonsense if you are in the southern hemisphere (like a big percentage of the global population)
What are they supposed to use instead? "Starting in Q3/H2"?
yes, many non US firms do exactly that for international announcements:
- use "second half of ", "begin of", 3 quartal of, etc.
- or a specific month if they want to be more precise
also for western focused announcements they also use "holliday session" as their tends to be a holliday session in most countries in both summer and winter (through their start differs _a lot_, but it tends to just work out if you release early enough)
The meteorological dates for "summer" correspond to June 1 to August 31. That straddles 2 quarters and both halves of the year. What are you going to do if a product launch is in July (+- 1 month)? You can't really use Q3 or H2 because neither of them fully captures that 3 month period.
say roughly around July but the we have not yet committed to an exact release month
"roughly around July but the we have not yet committed to an exact release month" sounds way more clunky than "this summer".
That is winter in the southern hemisphere.
I'm amazed at the need to have to explain this to a grown adult: https://spaceplace.nasa.gov/seasons/en/
Yes, I neglected to qualify "summer" with "northern hemisphere", but the substance of my comment doesn't meaningfully change with or without it, nor have I denied in other of my comments that "summer" is ambiguous depending on which hemisphere you're in. You're clearly just looking for stuff to nitpick.
Then use two seasons in the announcement. It's not hard. ;)
And they sometimes use their internal fiscal year, which doesn't align with the calendar year. So sometimes, when they speak of the "fourth quarter" of an year they are talking about the beginning of the next year, or in the opposite direction, they might speak of the "first quarter" of an year but they're talking about the end of the preceding year.
Classic example!
In the US, people will normally assume you are talking about Christmas holidays. In Blighty, people will assume summer holidays.
Maybe you are not aware but while it is summer in the northern hemisphere in the southern hemisphere you have winter (and so on). So, speaking of seasons means exactly the opposite depending on which hemisphere you are.
What's wrong with using a calendar date like may the 1st? I know that there are other calendars too. But is more manageable IMO.
Usually it's because they want to keep it vague because the exact date (or even month) hasn't been set yet.
"a big percentage" being only 13%.
I think this one is understandable.
That's easy, it's the US and any Canadian city with an NHL team.
I'd argue the UK is part of "the west"?
The UK is the old west (but not the Old West).
London is in the global west, except for the whole thing with the Greenwich Meridian going through… Greenwich, east London. Not to be confused with East London, which is in South Africa, which fortunately is also in south Africa.
The UK is definitely in the west of Europe though.
Its the World Series of Baseball.
You mean self declared (rightfully or wrongfully) leaders of the free world? :)
There's a documentary about exactly that: https://www.imdb.com/title/tt0372588/ :)
Is it bad that I knew exactly what this was before I clicked on it?
Nope, it just shows you're aware. :)
This is true for every large culture.
Not really.
This is the sad truth. I have a translation/localization company in the US. When pitching investors, first/second gen immigrant investors were always very interested, while multi-gen Americans would always ask "but what's wrong with google translate?"
Americans have a very strange and tunnel-vision world view.
I blame most of this on the fact that when you turn on any news channel, in an average 1 hour new program, less than 10% of the time is spent talking about anything outside of the US (and when they do talk about international news, it's always tied back to how it impacts the US - no one is listening to international news just for the sake of knowing what's happening in the world)
When working for a large US company they insisted we do not accept returns, they always were astonished that in Germany there is a law for 14 day returns, no questions asked. They could not understand that this is a law in Germany.
Isn’t it 60 days in the whole EU for physical objects bought via internet?
14 days - 1 year of repairs if you didn't break it
https://europa.eu/youreurope/citizens/consumers/shopping/gua...
2 years in most places --
But after 1 year you have to show that it's the vendors fault instead of the vendor having to show it was your fault so often "de-facto" 1 year.
Various exceptions include for stuff like food, underware etc.
Thanks for the link!
Yes, IMHO the German law was earlier.
What were they astonished at? The existence of such a law, or that they were subject to it?
It appears to be an increasingly clear risk to LLM model vendors that EU cares about where personal data came from, how it is stored, where it is stored, how accurate it is and whether there is a mechanism for it to be removed
Yes however it goes both ways. TPB was excellent in telling the US lawyers to f-off: https://web.archive.org/web/20110623123349/http://thepirateb...
I wonder if there is a site that covers common software license and has liability maps by country as to how much liability is waived based on the laws there.