CrowdStrike's impact on aviation

Tech Radar quotes Tom's Hardware; Tom's Hardware quotes a tweet.

Not a tweet from Southwest, mind you. Not even a tweet from someone who says that they used to work for Southwest. Just... a tweet.

keep hearing the Windows 3.1 story repeated

It’s wrong [1] and serves as a litmus test for whether an outlet independently verifies its claims.

(“The systems [Southwest] developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.” That got mangled into they run 3.1.)

[1] https://www.osnews.com/story/140301/no-southwest-airlines-is...

Can anyone working at Southwest confirm that their main scheduling system is running on Windows 3.1?

I can't confirm that, but I can certainly confirm lots of hospital equipment is still running Windows XP and lots of hospital personnel browse the internet with Internet Explorer.

I worked for SITA ( https://en.wikipedia.org/wiki/SITA_(business_services_compan... )back in the late 2000's. They had a massive X25 serial network connecting airlines across the globe. Some of its customers were still running Windows 3.11 in the data center on old AT system. We would buy old computers on craigslist and ebay to keep hardware around for when it failed. I wouldn't be surprised if those systems are still in use today.

The guy that started it all said it was just a "troll tweet":

https://x.com/ArtemR/status/1815408553131426179

This story is another example how hallucinations from LLM can successfully replace many "news" portals.

The "Southwest uses Windows 3.1" claim is false, and is a great example of how bullshit can spread on the Internet once some semi "reputable" organizations repeat the false rumor:

https://kotaku.com/southwest-airlines-windows-3-1-blue-scree...

The San Francisco subway runs off of 5-inch floppy disks.

https://sfstandard.com/2023/02/02/sfs-market-street-subway-r...

That article links to an (only slightly older) article about British Airways loading navigation updates every month off of the fancy new 3.5-inch floppy disks.

Your “modern and robust cloud” is my “why on Earth doesn’t this thing work offline”.

The world is absolutely full of things that have worked for decades to centuries without the Internet, are eventually more or less consistent (remember carbon paper credit card machines?), and did an amazing job of keeping the world running despite, wars, network partitions (the “network” would basically always be partitioned), mistakes, entire branches offline, etc.

Sure, a lot of things are easier when centralized, and “the cloud” is incredibly powerful. But it’s not necessarily more robust. Also, depending on any sort of cloud means you’re also depending on the network, and networks are far from infallable. There’s a reason that a lot of stored-value transit systems still track balances on the card and will let people in even if a fare gate cannot connect to a cloud service.

And CrowdStrike took out plenty of cloud instances, and recovering them can be worse than recovering physical hardware, as the “robust cloud” has an absolutely terrible ability to do anything outside the happy path of booting an instance normally.

This could have been fixed by having a minimal baseline of machines not running the same software

Resilience comes from diversity, in computing and in biology. Whether that's having critical workloads on multiple cloud providers or having one user interface on windows on network A (Arista) with crowdstrike and one on a mac on network B (cisco) with Sentinal one

Sometimes perhaps you can't eliminate a single point of failure, but you can sure reduce them to a minimum.

Or you can choose to increase next years bottom line and thus your bonus by not having a robust DR plan or system. You can also skip on boring things like raid and backups.

The trick for a CxO is to ensure that when failure happens, it's massive and widespread. Then it's not your fault. The CxOs in a given industry won't be fired because their DR plans didn't work because they believed Gartner and all their CxO chums in competitors did the same thing.

Nobody got fired for choosing IBM/Microsoft/Cisco/Crowdstrike/Azure, even if it's worse than the alternatives. People do get fired for bucking the trend even when it's measurably more reliable.

I'm not sure I follow, I doubt the web vs native implementation of an application makes much difference when the terminal used to access it is unavailable. A cloud based web-app is not much help if no one has a working computer and browser.

I'm not sure we're quite at the stage where a check-in agent using their personal un-managed devices to handle passenger data via a web-app is a great idea.

> training people to use an iPad or a web browser seems too complicated

iPads aren't designed to be turned into kiosks or airport departure displays and web browsers aren't operating systems (except maybe ChromeOS). So this advice boils down to don't run Windows, but CrowdStrike has caused outages of Linux as well.

Try making a graph in excel online and then come back to tell us everything needs to move to the cloud asap.

let's not throw the baby with the bath water.

native desktop apps are absolutely necessary for most professional / serious work and native desktop apps need offline support too.

with cloud - your risk factor goes up massively.

the risk here is that most of these companies are reliant on windows and of course snake-oil salesman of antivirus tools.

if you have a proper native desktop app, that runs in a sandboxed environment then you simply wouldn't need crowdstrike and the likes.

unikernels / bsd jails are things that have been well known and will easily mitigate "security" issues.

even windows these days has sandbox mode.

but incentives rule the world.

Counterpoints:

- Latency

- Security

- Legal obligations

- Offline work

- Managing the different sources of locking.

- Avoiding a single point of failure (I get the irony).

Because they used Windows 3.1

Re Delta

It's not so much a severity as "hard"; but with the hub and spoke model that Delta uses, scheduling being down (at all on Friday), combined with FAA hour limits. It becomes exponentially difficult to reschedule flights.

Put more plainly, on Friday, your scheduling software is down for 4 hours in the morning, so you "borrow" any replacements you need for employees that are late or sick. This ruins the availability for the next flights, at which time you hope the system is up again; but if it's not, you borrow from the evening flights. Combine this with each flight that was late/cancelled as you were hoping to fill now affects the hours available for the employees that were available. Finally, as you've cascaded this, you head into a weekend trying to catalog how many hours each crew member did or did not log, and you're not sure how to get them back in time.

One other compounding problem is that Delta's headquarters and main traffic patterns are on the east coast. Crowdstrike affected all the airlines at roughly the same time. This gave them roughly one to two fewer hours to respond before they hit their morning peak flights.

As someone else pointed out, they probably weren't ready by the time they needed their systems for the morning rush so they went to their business continuity strategy (manual). This has a throughput and recovery time penalty and obviously it compounds the longer they are in that mode.

I think what we're finding with the Southwest meltdown and now the Delta meltdown is that the big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity. I do think this should be investigated. Hopefully financial penalties incentivize action but time will tell.

>> Why were other airlines able to get back to normal so much faster than Delta?

I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that.

I heard on the radio (maybe NPR, not sure) it wasn't about the computers, it was about Delta's response.

According to the report, the other airlines delayed flights, while Delta cancelled them outright. That left Delta with more people and planes in the wrong places, making it harder to recover.

You don't want to know what OS Sabre (backend for 30% of world's airlines) is using on their mainframes.

Thirty years of progress is still progress for managing the additional complexity that modern software needs.

A lot of software doesn't need that additional complexity. Having thirty year old software, if properly sequestered (since there are security holes large enough to fly a 737 through), means that this is software that has been working for three decades. It has issues (as the mess they had previously showed), but Southwest appears to be able to be able to manage that to some degree without needing to incur the additional complexity of managing a modern software stack for application software that doesn't need it.

The ability to play minesweeper on critical computing equipment without impacting it isn't necessarily a desirable feature. Having the computer boot in five seconds and run the desired application is.

And there are a number of ways to handle that ... running old operating systems is one of the ways. Space Force S02E07 is not a desirable situation https://youtu.be/xDLvUqhwHZc . You could also have a kuberentes cluster with multiple replicas and load balancing and all of that additional complexity that takes more people to be able to manage without any real gains in what the application itself is doing.

is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems?

On a whim I tried playing Solitaire on windows the other day. You know, that game that’s shipped with windows since forever. Well, it’s horrible now. When I tried firing it up, it first spent several minutes downloading software updates. Then it loaded in some horrible “casual games bundle” app which felt laggy like a web app - complete with Xbox cloud sync for my progress, and daily achievements and other junk.

The game used to run flawlessly on my old 486. My computer now is orders of magnitude more powerful - but solitaire feels laggy. I bet the entirety of windows XP is smaller than the “update” it performed to install solitaire.

I have a personal theory that there’s always something that gets the attention of the best engineers. Decades ago it was human interface guidelines and UI toolkits. Today it’s LLMs and AAA game engines. Most of the rest of the software in the world is worked on by the B team. And they don’t blink an eye at the idea of rewriting solitaire for windows on top of electron. If JavaScript is all their team knows, so be it. Heaven forbid we have to learn how to properly build software for windows.

In the long run the hardware that can still run Window 3.1 will become harder and harder to find and they'll be forced to upgrade, but currently enjoying the benefits of "if it ain't broke don't fix it". Plus, there were literally millions of systems made that can run Windows 3.1 so it will be many many years before the hardware is too hard to find.

We're talking about a problem on the scale of 4,000 flights per day. Assuming you avoid O^2 complexity computations that's the sort of thing even 90s computers could handle easily.

productivity wise 70's and 80's peaked with those thin terminals with every action carried out by keyboard... workers tapped at light speed due to muscle memory, didn't look fancy, but it got the job done. GUI is sexy but like short videos ultimately did nothing for the user

Well, what actual new features does Windows 11 give you compared to Windows 3.1?

It will support a huge number of new chips, new peripherals, more memory, and so on.

If I'm running Southwest's crew scheduling software, how much of that do I care about? Do I care that it will now support the latest Bluetooth? Do I care that it now has the same UI as tablets? Do I care that it has better ads to display on the start menu? No, no, and no.

The only thing might be more memory. (I mean, the UI might not look like it belonged in the Stone Age, so that's something, I guess...)

There hasn't been a real fundamental improvement in the functionality of OSes since Windows 3.1. It's all been device support (including new classes of devices), new CPU support, and new UI styles. (The security improvements in Windows were a legitimately big deal, but those were fixing what was broken, not adding new functionality.)

And I'm sure that, having said this, someone is going to point out something really important that I forgot...

It was a "troll tweet" says the guy who started the rumor:

https://x.com/ArtemR/status/1815408553131426179

It is BS. Continuous updates for security notion, especially so. That said, the barrier to entry for programmers did come down significantly.

Obviously Southwest is not using Windows 3.1, and you should probably be thinking hard about the trustworthiness of any outlet or article that repeats that claim: https://www.osnews.com/story/140301/no-southwest-airlines-is...

It’s like a calculator, if you need to do basic math, you can use an old calc.

Well all you'd really need to do to avoid this outage is not run auto-updating proprietary kernel modules in the early anti-malware environment. Bare Windows 11 would have been fine - the problem was Crowdstrike.

Almost every industrial embedded system I've ever used runs Windows XP at the absolute newest, and it is not uncommon in the slightest to see stuff as old as 95/3.1. These are computers that operate machinery that costs 6 or 7 figures. If it ain't broke, don't fix it.

Don't get me wrong, my Macbook is an absolute beast for all of my work tasks, and my gaming PC is an utter joy to use for my recreation time, but at the end of the day, for a ton of applications, a computer doesn't need to do shit beyond sending a lot of signals out of a parallel/RS232 port to control systems to operate... I mean Christ, anything. CNC mills, building lighting/security systems, packing machines, or to do things like issue tickets to people parking in a ramp. Like... a lot of this stuff just does not benefit at all from a modern software stack. Stick a crappy PC inside instead, load it up with the same image it had before which includes firewall rules that shut down every last port and connection apart from whatever needs to manage it, and you're done.

Don't fix what ain't broke.

From my memory, this wildly circulating Windows 3.1 quote is inaccurate. The software they were running was compared to running something like Windows 3.1, but it wasn't actually running on Windows 3.1, as far as I understand.

Edit: https://kotaku.com/southwest-airlines-windows-3-1-blue-scree...

is computing actually a solved problem and we're really just mostly reinventing the wheel and enshittifying perfectly already working systems?

80% of the work is json bureaucracy

The other 80% is adapting to new requirements

And if you’re lucky maybe 0.1% of the time you get to build something new.

Fear not, a lot of this stuff was perfectly solved with pen and paper long before us computer nerds came to play in the big boy sandbox

Not sure it's fair, but I am certainly waiting for it to become a verb or a noun.

    crowdstrike. n.
     1. A set of major disruptions caused by an update that was not tested enough, pushed to many devices across the globe.
     2. The name of such an update.
     3. (by extension) a joke so bad it causes major disruptions.

     For instance:
       - Congrats for your crowdstrike! Now my weekend is ruined as I'll be the one who'll be asked to fix this mess.

    crowdstrike. v. (simple past crowdstruck or crowdstriked¹, past participle crowdstricken, or crowdstruck, or (obsolete, regionalism) crowdstroke²)
     1. Action of pushing an update to many devices that causes a global outage or major disruptions in various sectors.

     For instance:
       - We've been crowdstruck. Again.

    crowdstrike. adj.
     1. Qualifies an update that, when pushed to many devices across the world, causes major disruptions across the globe.
     2. Qualifies such a (set of) event(s).

    For instance:
       - We are sorry for the crowdstrike event we caused. We gently remind our kind customers and their end users that per our ToS, we will issue no refund, and that no liability can be held against us. Customers who don't try to contact us in the following month will get a discount for their next contract renewal. You will hear us speak before the Congress, who nicely invited us for some comedy in the hope it will appease you all. Make sure you like the related videos on the various online platforms. We wish you a nice end of the week and nice, relaxing summer holidays.

² some people have tried to push crowdstricken, which first caught on in some areas or particular contexts. The idea that this form likens the qualified subject to the bearer of some sickness has eventually seduced a critical mass of people after some initial push back. Please also see the usage notes for strike for other, rarer, alternative forms [*].

[*] https://en.wiktionary.org/wiki/strike#Usage%20notes

(Thanks to the contributors in this thread)

Rebranding project coming up at CrowdStrike?

Does anyone know (or have any guesses as to) why the founder(s) named it "CrowdStrike"? What was (or might have been) the idea behind the name? I'm guessing it's not patterned after "crowdfunding" "crowdsourcing" "crowdlending", etc.

The cute name was Blue Friday, but it doesn't seem to have caught on.

I don’t know if this is standard for software

This is pretty standard. There is almost identical language in the Windows and macOS EULAs, for example.

My experience has been better legal counsel has the relevant terms struck before the deal is signed. In this case it would have been the terms around Aircraft and aviation

There often are limits to how much your can disclaim in your T&C. If under the same terms you cause damages deliberately you'll be held liable, and obvious gross negligence can be a factor as well.

There are often 3 opinions between any 2 lawyers so we have a chance to learn the outcome many months and millions of dollars later.

Usually the largest of companies will have their own customized T&Cs governed in their Master Services Agreement (MSA) which are often very modified versions of these publicly available ones

I suspect it is incredibly challenging to keep a non trivial number of computers available, but somewhat airgapped. Too many ways to unintentionally bridge the networks without extreme diligence. Which is slightly incompatible with something like airlines which employ enormous numbers of people.

Hmm. I think this is a pretty shallow take.

My experience from the airline industry is that the vast majority of systems classified as flight safety critical are not connected to the internet, or large networks at all. Which is good.

But unless we want to drastically change how airlines operate, the rest need to be online.

Today, you can purchase a ticket (or rebook an existing one) on your phone really close to the departure time. When that happens, a gazillion interconnected systems, across legal entity borders, need to cooperate to take you (and your luggage) to the destination.

To put all of this in a large non-internet network seems pretty pointless.

If we wanna go down that route, the only real "security improvement" I can think of is to dismantle the digital systems and go back to paper. Like Ryanair did during this incident. Handwritten boarding passess, verified against print-outs of passenger manifests.

The related question is "How do you run your business out of downtown Fort Worth (American Airlines) and get your updates to 350 airports in 60 countries?"

Saying "run your own network" isn't exactly practical. Even imagining the very small airlines (that partner with big ones for the last leg) that only service a handful of rural airports, this doesn't seem practical.

The days of point to point updates over modems are overish with the amount of data that needs to be consistent transmitted and available.

I can imagine a modem at each airport and a phone bank of about 700 modems that are each getting or sending updates. The long distance calls to distant countries for that data could get expensive. Woe to the power outage in Texas that takes down the phone bank for a day or two or three or four.

Alternatively, there's a system that was developed to do this and it works pretty well most of the time. Combine this with having redundant systems there that are geographically separated. It isn't turnkey, but its probably better than other options that would involve home grown solutions.

It's an interesting thought experiment to consider everything that would have to go into running operations for a business like one of the largest airlines in the world using a non-Internet connected network. Among other things, you probably lose the ability for your employees who aren't physically in the office (which is kind of a lot of them if you're running an airline) from interacting with your operations network. If you're an airline trying to schedule employees and share information with them while they're in hotel rooms, that's probably a deal breaker.

That's really only a secondary problem though, because disconnecting a network from the Internet isn't a replacement for security software or software updates, so you wouldn't even avoid the root cause of the issue here. I'm not saying CrowdStrike is essential software for Internet connected computers either, but if your business thinks it is, you should probably be running it on your "airgapped" computers too. And you should definitely be installing updates, so you can still fall victim to a bad updates regardless of which software you run. At best you perhaps increase the likelihood of hearing about a problem with an update before you deploy it on disconnected computers, but you can get a similar effect by delayed deployment of updates even on Internet connected networks.

Exactly, most of these systems probably wouldn't require EDR software if networking was done correctly in the first place.

Air-gapped networks have gotten more scarce in the day and age of the cloud computing. Expect for certain DoD and cleared spaces, I've even seen PLC networks internet connected....

So Web 2.0 was a mistake I take?

Problem once again is humans. Humans need to interact with systems, either receive information from them or give it to them or use the systems to process it. And for efficiency in general it nowadays happens online. It could be offline, but that would be slow. Or it could be segregated networks but that would get really expensive. Imagine having own fiber line for your instant messaging and email? With different terminal...

In the end most of these affected computers are on Internet for very good reasons. And this model really is working vast majority of time generating lot of efficiency.

My guesses: -no one really cancels their security vendors since security budgets don't shrink -they have a big moat so their customers won't be able to leave them

i replied upthread i think the stock price has bottomed from this event. It's way to hard to switch vendors like this at an enterprise scale. What's going to happen is the cloudstrike account reps are going to get yelled at and abused, some discounts are going to be offered for annual renew, then two years from now all will be forgiven/forgotten. In a year or so the stock price will recover and trend to more or less where it was before this event. I've already bought as much stock as I could.

Good news then! You can short it and make a ton of money if you're confident this share price increase is a mistake.

CrowdStrike makes it easy to pass your security audit. That's where the value is.

1. Compliance. No protection at all isn’t a contractual option in many cases.

2. Companies react slowly. When has a vendor paid a high price for failure? Boeing can kill people and fail time after time still sell planes.

Catastrophe always changes less than anticipated.

Southwest does not use Windows 3.1. Why does not one read the article?

Southwest wasn't affected because they don't use Crowdstrike. That's it.

I find this more surprising, even if the Southwest & Win3.1 claims were true, I would expect most Ericsson systems to be Erlang based and thus happily chugging along on a (perhaps ancient) Linux box.

For everyone flabbergasted by Southwest running Windows 3.1

Southwest isn't running Windows 3.1, though. That's some rather lame, but predictable, truth-through-repeated-assertion thing on social media.

Not everyone uses CrowdStrike, and in this case SW was the lucky one that didn't.

Good thing a lot of our banking still runs on mainframes, will never be taken out by crowdstrike

has hired a law firm and will seek compensation from Microsoft and CrowdStrike

Going after Microsoft seems like a misguided move here. What does Microsoft have to do with a third party driver installed by your own IT department?

Meta is one of the most valuable companies in the world with the most resources to buy the best of everything. At 1,280 Billion dollars of market cap it is 30x bigger than American, Delta and United put together. It made $39 Billion last year compared to $7.8 Billion for all US airlines together. Of course it has better systems.

Windows has always been terrible for reliability. Adding a "security" system which is invasive and always-updated makes the reliability worse.

I wouldn't overestimate FAANG's immunity to crash-the-world config updates. Facebook had everything including engineers' access to the datacenter down for hours in 2021:

https://news.ycombinator.com/item?id=28750894

> infrastructure as compared to an OS

By the way, I don't think quality of Microsoft's infrastructure is relevant here.

Because one is in a data center that can be controlled, and the other is deployed to user owned hardware that cannot.

Because IT is your social media's business. They know IT inside and out. They understand what can go wrong and how to mitigate it. The business for airlines (for example) is to fly planes. They are pretty damn good at it. IT, however, is just a tool to them that they buy elsewhere. They don't understand it in the same way as social media. They rely on outside contractors to do it right: outside contractors who get the job based on being the cheapest or convincing the buyers their service is "industry best practice."

Compare the salaries, working conditions and prestige offered by tech jobs at a social media companies vs some large legacy company like a bank or airline.

In the former, you are paid well and have some sort of prestige and political capital. In the latter, you are underpaid and your prestige/political capital is often equivalent to the janitor's.

Because they don't do mass rollouts on the servers. Then again those companies could fail if they had single point of failure with automatic mass deployments...

This could happen for anything that supports this type of automatic mass deployment. Just in this case that thing was popular enough and happened on one of the most popular platforms.

This isn't true, and that should have been obvious to technical people. It's so sad that we have a tech media that doesn't give a damn about making things up.

Southwest experienced this kind of scheduling issue in 2021 [1], and again in 2022 [2]. Honestly, if they're running win 3.1 or win 95 as suggested, I think that puts them in a better place tech wise than keeping up with the Joneses on the upgrade treadmill --- although they should consider updating to windows 3.11, because they have a workgroup :P and the microsoft hearts network is pretty cool; but they have historically done poorly on scheduling after a significant disruption. An article from last year [3] says they updated their crew assignment software as well as increased staffing in colder airports and in general and got more deicing equipment. We won't really be able to tell if it works, until they experience another disruption.

[1] https://www.cnbc.com/2021/10/12/southwest-airlines-reduces-c...

[2] https://www.npr.org/2022/12/26/1145536902/southwest-flight-c...

[3] https://www.npr.org/2023/11/09/1211064462/southwest-airlines...

You don't even have to wait it's been happening

Southwest had two of these recently. It was widely reported:

https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...

Hard to say, it could actually increased emissions. As when the timings of things don't align correctly it's common to cause an increase of resource usage. Eg. people travelling less optimal route, extra commutes back and to the airport. People having to physically travel to datacentres in order to fix things, just rebooting the machine without need will use more CPU.

In my country, companies are required by law to keep track of their CO2 emissions.

In the case of CrowdStrike, they would have been able to deduct this event from their emissions for decades.

Always look on the bright side!

It's more accurate to say "any airline not using Crowd Strike is not on that list."

Blaming Windows for this outage is like blaming Linux for Apache bugs. The two systems are distinct.

It just so happens that Crowd Strike was very successful at selling to large corporates. That includes some airlines.

99.4% of Windows machines were unaffected. Including those of airlines using Windows, but not Crowd Strike.

every airline in the world "uses linux", the core reservation and distribution systems were migrated from TPF to Linux over the past 20 years

Can you name one?

Maybe I'll do a Part 2: The World.

love to see the airlines using linux and what kind of problems, if any, they experienced that day

I’d love to have some solid numbers of “global cancellations due to” - I heard a bunch of varying figures so far.

In the original thread there were some reports of people having their Linux systems taken down by Crowdstrike as well. At separate times, of course, and I supposed the greater heterogeneity of Linux distros prevents events of this magnitude. But that would be little consolation when it takes down your systems.

Same thing for a lot of industries actually. PROD runs on Linux and probably has some delay to prevent this. Corp gets hosed.

Our systems worked fine, we expect things to fail - including software like sentinal one, crowdstrike, etc, and have DR systems which can keep us limping along. We have DR systems which will work should other things happen - say the Thames barrier fails (i.e. no docklands)

Unfortunately some of our outsourced suppliers didn't have such attitudes.

I've never seen it.

Obviously some selection bias there, but I'd love to hear some success stories.

They certainly have DR infrastructure primed and ready to go… with Crowdstrike pre-installed on every DR server.

I see just moments after I posted, someone posted this: https://news.ycombinator.com/item?id=41103486

So, yeah, lack of DR is why Delta was so screwed.

German IT is often hit hard by such things. Unless of course they're still running on paper.

At least they immediately mentioned it on their website, as a banner right at the top. The immigration office's appointment system has been down for over a month, and it took them 3 weeks to just acknowledge it.

I'm still shocked that Brandenburg is actually open!

It was widely reported to be the busiest travel day for quite a long time, which compounded problems.

Yeah, shouldn't have been too hard to add a couple more weeks so you at least get an idea about variance.

It’s also not true

That’s why I think the suit against crowdstrike and Ms is mostly a dud. First you have to get around the waiver (much harder for business than a consumer) and then you have to deal with comparative fault - ie delta’s disaster recovery system sucked.