According to Dragon Sector, Newag entered code into the control systems of Impuls trains to stop them from operating if a GPS tracker indicated that the train was parked for several days at an independent repair shop.
Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
Here is a better write-up of the story:
https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
> A condition has been written in the computer code to disable the ability to run a train if it spends at least 10 days in one of these workshops.
Wow, that's just... so wrong.
Uh, maybe not. These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
This makes little sense. It's pretty reasonable that any machine that is sent for repair may take longer than expected in the workshop. Parts availability for one. Also, manpower shortages, scheduling (we don't need it back until next month) and so on all make this "heuristic" more likely be a scam. This idea that the manufacturer is entitled to a lifetime stream of repair revenue has to stop.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
Yeah if you're going to try to pull a scam like that the US is not your best option:
https://nypost.com/2022/12/27/texas-mechanic-executed-over-5...
https://www.thetrucker.com/trucking-news/the-nation/pennsylv...
https://apnews.com/article/auto-shop-shooting-florida-c4d45f...
If you just search "truck owner murders mechanic" you get tons of unique results.
I feel like blaming guns might upset some people, but I don't feel like this type of thing happens as much in the rest of the western world, where gun laws are typically a lot stricter.
I assume most of it is access to guns. But I do wonder how much is due to people are not using giant ego-soothing trucks in the parts of the world where petrol doesn't get subsidies. So there are fewer truck owners to do the murders.
Total nonsense, car owners murder truck drivers too.
Don't you ever care about upsetting people who live in countries with unreasonable gun laws. Especially third world countries like the US.
That's nuts. Who says those were scams? To me it reads as though the customers had a couple of loose wires.
I'm not saying these were scams. I was just pointing out that customers in the US that feel like they were scammed can often get deadly violent.
The top comments on that nypost article are insane, blaming the victim for probably doing "unauthorized repairs", and saying that the "southern border" is at fault.
Wow.
These things are safety critical and ten days in a workshop is probably a reasonable heuristic for this thing has to be taken out of service and recommissioned from scratch.
I know HN is a great place to practice PR-speak, but could you please stop fucking with the gullible people?
The words were chosen to indicate uncertainty on my part. A train parked in a repair depot for a fortnight triggering a need for more checks before putting it in service? I can totally imagine that spec being signed off in meetings. Fortunately the voting system on here does an excellent job of hiding unpopular thoughts so I doubt I've confused anyone.
This is stretching my sarcasm meter to the breaking point, either you are very good at deadpanning or you are trying really hard to muddy waters that are crystal clear. If the former, well played.
You’re just pushing a pretty blatantly uninformed opinion supporting a frankly indefensible position. My car had minor collision damage and it took the manufacture’s 1st party service center 3 weeks to fix it. The assumption that manufacturers are uniquely positioned to repair their products is a poisonous tactic being used by insidious players to discredit right to repair.
Was that sarcasm?
No sarcasm, just in total disagreement with the (initial/apparent) HN consensus. I've put a top level comment which is hopefully clearer.
You are also disagreeing with the owner and operator of the train, who is responsible for its safe operation.
The manufacture's current position is someone hacked their systems and changed the code so that it would make them a lot of money, they totally didn't do that.
"The president of Newag contacted me," Cieszyński wrote. "He claims that Newag fell victim to cybercriminals and it was not an intentional action by the company. The analysis I saw indicated something else, but for the sake of clarity, I will write about everything.
Newag president Zbigniew Konieczek said that "no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software."
Which is obviously bullshit made up by the PR agency they hired, designed to muddy the waters. For example, researchers dumped the Train's firmware (in presence of third party witnesses) before and after sending it to the Newag's service, and found that the a backdoor code was recompiled and updated there.
Ok so no sarcasm, well, there are many reasons a train can spend a long time in the workshop:
Complete checkup takes time, or one important mechanic is sick, delaying things or whatever. It is not the responsibility of the manufactor anymore. (A different company has the official service contract.) If you have other informations pls share.
In at least two cases, the train was simply unused for 10 days for unrelated reasons.
After which the two specific trains this happened to "gained" GPS lockout.
That’s not a long outage. If someone decided to paint it, the clean, prep and paint could take that long. And now you have a dead train.
Of course, because we all know that only the manufacturer of such a complicated device can be trusted to maintain it. Oh wait, maintenance of trains is a competitive process? The manufacturer provides a 20,000 page maintenance specific manual? Maybe trains really are simpler than iPhones.
/s
Parent comment is really an indication that society has lost the plot when it comes to ownership. The trains do not belong to the manufacturer after sale. They can't introduce anti-competitive code and pretend that users want it, like phone companies can.
No, absolutely not.
(1) This was scheduled maintenance service, not a quick repair. Maintenance is complicated and is expected to take more than 10 days:
Maintenance a train is a complicated affair – it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested. The SPS carries out the maintenance procedures according to the relevant maintenance manual (some 20,000 pages) provided by the manufacturer, but the train does not start after being put together.
From [1].
(2) If this was a legitimate check then there should be a legible error code instead of the train randomly locking up with no explanation why. This was clearly designed to sabotage competing maintenance service companies.
[1] https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
It's like the code was designed to get my wife on my case after, "Honey, don't worry, I can fix it myself and save a bundle not having to go to the dealership."
"Told you you would make it worse" incoming.
That headline was so misleading. I'm interested in the topic and saw that link on HN, but scrolled away because dieselgate is "obviously" about emissions and I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
They should have mentioned something about vendor lock or right to repair in the headline.
I assumed that nothing new will come from the article about train emissions, we all know already they are big polluters but we kinda have to deal with it.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
Modern diesel engines yes, but the old ones? They got barely any emission controls, only for locomotives built after the 90s there is regulations [1]. On top of that comes brake dust [2] which is only irrelevant in powered-car electric passenger trains with regenerative braking - every other train releases insane amounts of it, no surprise given that the brakes have power outputs in the megawatt range.
[1] https://www.epa.gov/regulations-emissions-vehicles-and-engin...
[2] https://www.railwaygazette.com/vehicles/reducing-brake-dust-...
Right above you it explicitly states the trains are •electric• and not diesel. RTFA
As said: Electric trains are not emissions-free due to the brake dust.
What is it with this thread? You're the second person to bring up the most wild stuff.
When I walk I kick up dust. And I breathe. I guess I'm also not emissions free even when I cycle? Brake dust and all that?
By that measure nothing is emissions free, me walking down the road will generate a bit of rubber / asphalt dust.
Also regenerative braking... Trains are amazing
Many trains have rheostatic brakes, where the kinetic energy is converted to heat using a bank of resistors.
In absolute values, not relative per ton of cargo or compared to other modes of transport. Maybe I wasn't clear enough, for that I'm sorry. I do support trains everywhere and think they are the best land transport we have by far. I was merely thinking that a diesel engine is a diesel, even if a train vendor fudge the emissions somehow (hence the reference to the Volkswagen dieselgate, I thought it meant) it wouldn't matter much on the planetary scale. Thus not particularly interesting.
But instead this was all about electronics and malicious locks, which wasn't apparent from the headline.
IMO this is really what Dieselgate was about. Not specifically fudging the emissions, but hiding secret code to do so during specific test profiles, then lying about it afterwards. Analogously, these trains hid secret code to disable them after being taken to certain locations.
Especially since these trains in particular are electric.
we all know already they are big polluters but we kinda have to deal with it.
There is a lot of variation in train emissions, but most modern ones (electrified) have minimal emissions - and far lower than electric cars even, per passenger mile or per tonnage transported.
Even the oldest ones in common use (diesel electric) are more efficient per ton-mile and passenger mile than a typical car.
I've tried to explain myself in the comment below, sorry for the confusion.
And the relevant HN discussion: https://news.ycombinator.com/item?id=38567687 5 days ago, 289 comments
Which again is a successor to: https://news.ycombinator.com/item?id=38530885 8 days ago, 357 comments
The OP article is worth reading for the brazen blustering statements made by Newag management, I paraphrase "you can't prove that the software found on our trains that sabotages our competition was installed by us!"
This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
On the other hand, trains that refuse to start don't immediately kill everyone on board, and trains that have fall off the tracks do tend to kill people. So when a train fails and people are dead, and people go in search of who is responsible, it seems decently likely that ambiguity between the third party repair shop and the company responsible for the train is a problem.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There should be an opt out system - something where the train operator contacts the train supplier and says "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Or to bring it closer to home, say one of us breaks into our tesla to exercise our God given right to change the software stack, and then it kills the driver and various people around it, to what extent is that Tesla's fault? Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
You are putting forth some general rules that seem reasonable, but they don't really apply in this situation.
From what I've read, the train operator bid out the maintenance for these trains, and the manufacturer lost the bid to a third party. So your "opt-out" system should absolutely have been triggered here.
If there was some reason to believe that the low bidder would be incapable of safely performing the maintenance, the train manufacturer should have raised that issue publicly, not silently sabotaged the low bidder's ability to perform the maintenance.
Maybe? I'm speculating here as it's not my field. The obvious explanation to me is very different to the obvious explanation to every other comment I've seen so far. That difference seemed interesting enough to be worth posting.
The question of where liability falls after a repair isn't covered in the articles linked unfortunately. It seems plausible that it was initially whoever wrote the software and is now someone else.
I think it's extremely likely that this disable-train-on-various-conditions behaviour is exactly as specified and documented since train software is a bit obsessed with formal methods and documentation. It's then only "silent sabotage" to the extent that said docs were ignored.
It's credible that a way to easily disable these safety checks for the case where it's now someone else's liability wasn't considered worth paying for by whoever bought the train. It's tomorrow's problem after all. Also the customer may not be totally convinced of the necessity of the software dev paranoia, especially if they're not liable for failures.
So sure, maybe this is evil/negligent software people at the train company. I don't see that conclusion well supported by the article.
So all of this obviously biased and incorrect speculation on your part is based on.... not being an expert in the field... and not having read the actual article? Thanks man, very insightful.
Your argument is based on an incorrect understanding of liability, and is also nullified by the fact that these measures, allegedly justified on safety grounds, were not just undocumented (see other sources) but clearly deliberately hidden, to the point where the manufacturer pretended not to know why the trains would not start.
Update: here's a quote from the Ars Tech. article itself, showing a) no tampering with the trains' software or hardware was necessary to get them running, and b) there's no even slightly plausible case for believing the manufacturer was unaware of this. Furthermore, if it really was somehow unaware of what was going on, then these 'features' cannot be justified as a safety measure.
Dragon Sector got the trains running after discovering "an undocumented ‘unlock code’ which you could enter from the train driver’s panel which magically fixed the issue."
Update 2: It is barely plausible that the ability to track the trains' presence in other maintenance shops was initially added to gather evidence for liability and warranty purposes, and then someone had the dumb idea of using this to covertly disable trains when this happened.
Note that the use of a third-party maintenance operation was not a secret: the train operator solicited bids for the work, and the manufacturer tendered one. Clearly, third-party maintenance was not in violation of any contract (and if somehow it was, the manufacturer did not need to gather any covertly-acquired evidence for a breach-of-contract suit.)
It's then only "silent sabotage" to the extent that said docs were ignored.
From what I've read, the maintenance company did scour all documentation and found no explanation for why the train was disabled. It was an entirely undocumented feature. Maybe they were just lying, but this idea you have in your head that it was just a fly-by-night operation who couldn't be bothered to read the documentation is contradicted by the available public record.
Sorry to be blunt, but your longwinded "speculation" in fields you don't understand is not a useful contribution to the discussion.
I haven't read the Ars article yet, but imo the evil/negligence of the train manufacturer is reasonably well supported by this article: https://www.404media.co/polish-hackers-repaired-trains-the-m...
The difference is:
- there's a proper registry of who is responsible for maintenance of the train, including chain of liability. Crucially, the vendor wasn't in it for the affected trains - they'd only be liable if design issues were found.
- maintenance companies was appropriately certified and verified - we're talking professional MRO, not random workshop
- owners of the trains were also registered as responsible parties for maintenance and repair
- vendor was supposed to provide documentation that would allow such a workshop to perform maintenance and repairs up to P5 (largest scale) maintenance
- vendor never disclosed lockouts they implemented, thus committing fraud at the very least.
Sounds reasonable to me. Plus you've used domain specific terminology which suggests you know more about this than I do. Thanks for the context.
I definitely haven't gone looking for the documentation on the train system, partly because I assume it's unavailable to the public and mostly because I assume it's enormous in extent.
This is on the general topic of right to repair and freedom of fixing and so forth so naturally HN hates the train company.
Actually, the issue here is that the train company is on the wrong side of this issue.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There are tons of cases where the repairer, not the OEM is held liable. The manufacturer is at fault for the defective parts/product. The repairer is liable for a defective repair. Courts have been sorting this kind of thing out for centuries and it's not a difficult thing to deal with.
"we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Except that really isn't the case at all, and hasn't been the case. If the repairer does not do the repair correctly, the repairer is liable, not the manufacturer. Most mechanical shops even carry insurance and offer their own warranties on their work. Most cities (and hopefully consumers) will only do business with repair shops that are insured for this exact reason.
Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Yes. It is not their car. It is my car.
Ford is not liable if that variable geometry camshaft I installed in my Mustang causes the engine to blow, killing six people with shrapnel. I would be liable for that. Why should Tesla be liable for someone hacking the software? The issue here is the world doesn't work the way you are describing, where manufacturers are held liable for the work of third parties. Tesla is held liable for their own warranties and laws applying to product safety.
I once saw a Boeing 727 that had been retrofitted with winglets. That was allowed, but Boeing no longer guaranteed the wings.
So yes, it works just as you describe. (Planes, not trains, I know, but it's still a datapoint on how liability works out there.)
The same viewpoint applies to any motor vehicle too. How many road accidents are due to mechanical error introduced by a mechanic (that does not operate on behalf of the car manufacturer)? What makes an "official" mechanic employed by the manufacturer superior to a third party?
This is an interesting discussion, thank you for your nuanced opinion. I don't know where I stand on it but you are spot on about the demographics of HN. I certainly have a knee-jerk reaction in the direction of operator freedom.
Edit: Additionally, if something like this goes wrong (say the train derails) it will be a massive blow to right to repair. Even if it is later found that the third party repair is not responsible, the media will scoop up this story and it will forever warp the public's ideas about right to repair.
In theory, I agree with your viewpoint. Yes, obviously, safety critical systems need to remain safe, and a workshop that isn't capable of performing maintenance to the degree mandated by the train software shouldn't be doing it at all.
That is not the case here.
If that was indeed the case, one would expect the train's diagnostic systems to report that it has detected certain issues that need to be resolved before the train will start up again. But that didn't happen, instead the train simply refused to start, no error codes, nada.
Not only that, but further investigation revealed that the train's firmware contained code that would disable the train if it was present in a competitors shop for a considerable time.
There should be an opt out system
This mechanism for disabling the trains, was never mentioned in the discussions or contracts for purchasing the train (which is why a third-party could win the repair contract). There was never an option presented to "opt out", that is why people are saying that the trains were sabotaged.
to what extent is that Tesla's fault?
It isnt. You made unsafe modifications that can be clearly shown to have caused damage, it's your liability.
Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Depends, did their software present an actionable error or not? If the car gave an error code, and you look in the manual and it says that too much current is going to the motors (idk), and then you fix that and the car starts? Great! But if the car just refuses to start without any indication as to what is wrong, and further even if all repairs are reversed it still no longer starts? That sounds like deliberately sabotaging third-party repair.
I sure hope the dev who was made to implement it kept the paperwork that proves who made them do it.
Otherwise, they’ll find out how loyalty is rewarded nowadays.
Even if they did keep that paperwork, they should be punished alongside the manager who asked them to do it. They knew it was unethical and did it anyway. They should have refused. Risk of being fired is not as big of a problem for a programmer as it is for almost anyone else.
I don't think that's a good way of thinking about it. Just because programmers might have better job security doesn't mean other motivators for not wanting to get fired exist.
Whatever other motivation you have it should not be enough for you to do illegal/highly unethical things.
This is why we need very strong whistleblower protections.
Which also applies to everyone else. The programmer job security just makes that easier in most cases.
Which is not really my point as to why they should be held responsible. The reason is the real world consequences of their actions and the scale and ease of introducing negative consequences by tech creators.
I've got mixed feelings. I would be interested in hearing the whole story first. The "hey, I can program the trains to break if we're not the ones to run maintenance, should I do that?" or the "yeah, I guess I could make the trains break if we're not the ones to run maintenance" are both pretty clearly unethical.
However, the "yeah, sure I can add in a GPS locator module" and the "yeah, I can add analytics that reports when the train is in a maintenance hanger" and the "the catastrophic program halt code module used in cases of extreme failure is located here, but why do you want to know that?" all seem less than unethical.
Theoretically you only need one unethical line of code, so how it got there, I think, is pretty important to know before passing ultimate judgement.
EDIT:
Of course for something like train control software, you really should have a process or at the very least responsible engineers that would notice a middle manager with limited technical skills asking suspicious questions and then pushing up a PR that is self approved.
I would be more than willing to entertain an ethical debate along those lines. Although, like I said, I think it's important to understand the whole story because the specifics really do make a difference.
To be honest, with or without a paper trail, society need less people who do shit like this.
Whoever implemented this, if you read this, you are a very bad person and destructive for society.
I disagree with the framing "You are a bad person" (although I think I understand the sentiment), because it implies they can't change (or understand the error). It seems better to leave it at "You did something very harmful, destructive for society".
I agree. I understand sometimes a person is under enormous pressure to do something they know is not right.
TL;DR:
* Polish train maintenance company, SPS, was getting suspicious as trains made by a company, Newag, kept on "randomly" breaking and couldn't be fixed. They was getting fined millions by Polish government as they had a contract that fined them for being too slow with repairs.
* They secretly hired literal hackers (Dragon Sector) for 2 months to dig around Newag train code.
* Hackers found out some incredible things, generally that fit under the umbrella of "late-stage capitalism", or more specifically, corporate protectionism, sabotage, ransom, etc.
Some examples of the secret code that the hackers found:
* Breaks the trains if they go into geo polygons that are right around the warehouses of 5 Polish train maintenance companies, including SPS.
* Breaks the trains after 1 million kilometers.
* Breaks the trains if they don't move for 10 days.
* Secret button press combination (basically Tekken, Street Fighter, etc.) to disable the "malfunctions".
Do you know if the low-level technical report is available? I love reverse engineering firmware, and this sounds like a holy grail. I mean a freaking train? Ugh someone should drop binaries.
Not found one yet. The linked article contains a very small amount of detail, such as the lat-long coordinate values they found within the dissassembled code, etc., but not much else unfortunately.
I'm waiting for the stuxnet-like report on this as much as anyone.
Some info is here https://badcyber.com/dieselgate-but-for-trains-some-heavywei... , but the actual details will be presented at the Chaos Communication Congress at the end of December.
What noobs. Everyone knows you've got to put that chicanery behind a web service call so they can't find the evidence directly from the client.
Let's not give anybody ideas now shall we.
https://api.newag.pl/shouldirunornot?lat=&long=&trainid=
can't wait to see the swagger docs on this...
Just realized: “late stage enshittification of capitalism” is essentially “verelendung” as prophesized by Marx
What really bugs me about all this: not only were they designed to break down, they messed that up to the point that the trains could have broken down while in service. The fact that a manufacturer would risk the lives of the passengers of the trains should result in personal liability for all of the execs of that company.
https://news.ycombinator.com/item?id=38641289
I'd love to see that angle researched more because I think it changes the game from something commercial to a far more important level.
The rail industry has a death grip over congress, which also why hey got away with over working staff during covid and got the green light from congress and the president to do so when staff went on strike.
Read the article. Wrong country
The article describes Polish trains.
I was expecting some malicious code like "knock 3 times on this door to enter, otherwise the door locks for good". No. Instead it was "if you're geographically parked at our competitor's lat/long then we'll brick the train".
There's ways to add back doors and other fun easter eggs in code, but wow they picked a really cumbersome and obvious method of exploitation.
You almost guessed it though. There was a (since removed) secret combination of buttons on driver console that magically unlocked the train.
"hesoyam"
Reading their official statement it's hard to not get the impression that it wasn't even consulted with their attorney - they deny specific allegations in a way that just begs further investigation.
I mean, I'm Polish and worked for such people in the past. The statement reeks of an approach to business that I wish went extinct already, but apparently you still see pockets of it here and there.
Is there any conceivable point where people will be charged and jailed for fraud after selling sabotaged goods?
Why would the powerful punish themselves?
Because more powerful entities might be enraged.
Theoretically, the recently-revealed prosecution started with notice of two crimes that include 0.5 to 8 years jail time.
[dupe]
Does no one read the site anymore?
original news story discussion just over a week ago: https://news.ycombinator.com/item?id=38530885
And the followup from the company
Polish train maker denies claims its software bricked competitor rolling stock https://news.ycombinator.com/item?id=38570654
Not to mention the 404media dupe with a ton of votes just yesterday also!
>Does no one read the site anymore?
I don't know, is it some kind of requirement to read every piece of news on the main page every single day to keep visiting? Or can I log in every few days, read a few stories, upvote the ones I like, then visit again few days later? Or is that not allowed?
More on the submitter in this case.
Upvote what you see/like, sure, all good.
But if you're not around, you miss some stories, that's it. (especially when it's come up so many times over weeks) There is a current events aspect to HN.
Does no one read the site anymore?
I don't read the vast majority of sites linked here. I come here for the comments. I assume anything that's important enough to know will be directly quoted in a comment here.
I have seen developers lock/encrypt their code they write and then when they get fired for it, they want the company to pay a fee to get them to unlock it/release source code
That is textbook https://en.wikipedia.org/wiki/Logic_bomb and is very "jailable" - as if plain blackmail laws weren't enough.
Edit: someone already added Newag there :D
They better have some code around it that stops that from happening when the train is in service.
pretty sure even if they have source code version control, the "funny" patches are outside of the version control and applied on top before making a build.
That's almost certainly not the case. In general for vehicles, you're installing signed, verified images. Not only would it be a LOT harder to consistently apply OOVC patches that don't show up anywhere, having signed builds that don't match your source would be a worse red flag than just having the code in version control.
The code also allegedly bricked the train if "certain components had been replaced without a manufacturer-approved serial number," 404 Media reported.
If only commercial airliners had this too...
https://news.sky.com/story/fraud-officers-arrest-one-in-dawn...
Commercial airliners have other procedures, including - just like with trains in europe - complete unbundling of maintenance and repair.
Because it's pretty normal for an aircraft to be in use longer than the manufacturer exists.
Newag has said that "any remote intervention" is "virtually impossible."
That’s a very bold statement right there! Also, from my experience, most if not all companies who build automation/robotic systems they maintain “backdoors” of some sort, not to disable them remotely, but to quickly and efficiently access the software remotely upon request to troubleshoot or fix issues, it doesn’t make sense to fly to the client and sometimes the other half of the planet just find out it was a local IP mismatch or a system service is down, so saying “impossible” is not true.
Right to repair for anything we buy is essential and customers need the right to access software they have purchased.
Previous threads on same news:
We lament that circumventing DRM is legally perilous, but sabotaging your customer's property is illegal. That you were the one that sold that property makes no difference - you sold it, it's not yours anymore, despite corporate messaging otherwise.
The real question is whether or not we will get quality reporting from the courts! Hopefully someone goes to jail over this.
They will surely try that explanation too if the others fail. Currently their stance seems to be "there is no such code on the trains, and if there is, it's not from us".
It’s from those darn third party developers! ;)
Blame the hacker, wait for it.
Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.
You're talking about industrial PLCs. They're programmed using a-bit-more-fancy Scratch snappy blocks. There is no version control. The firmware contained paths embedded as strings, so we know that firmware for each model and customer was developed in a separate folder on disk. I wouldn't be surprised if they also had .zip files with backups of previous versions.
I'm intimately familiar with PLC programming, yes, you can do it the 1980's way but there are also plenty of environments that allow for modern version control.
https://www.google.com/search?q=version+control+plc+programm...
You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output (text) files. You may not be able to do an easy line-by-line comparison but you will have a commit log with helpful messages.
Look for 'engage in anti-competitive behavior' in the log message ;)
What about: rogue hackers maliciously squashed our whole repo into a single commit?
I don't doubt that's exactly what would happen, in fact I think that that rogue hacker is about to do his thing, quick, erase the backups!
Yeah.. I've just realized that while it's entertaining to watch how it unfolds or predict what can happen next, it's also sad, because pretty much everybody in the rail industry loses..
One of my business partners works for PKP it's very annoying to see this all unfold and in this particular way. Poland has so much potential, these idiots are ruining Polands image in ways that really matter.
But then again, as a Dutch person I have enough issues locally that I can't even complain...
Ugly times.
I have a friend that runs a business, where he hires Polish developers to do his coding.
He absolutely raves about them. It sounds like he's got some good coders.
Poland has a very strong technology and mathematics tradition that goes back decades. It's one of the reasons Poland has some strong feelings about their role in the breaking of the Enigma, for the longest time that was played down.
Working in security on the operating side (albeit not in Poland):
No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.
For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).
For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.
That's assuming we will get to the bottom of this. And I really hope we will. But I'm kind of concerned that it will all be wiped under the carpet.
What I meant is that I feel the trust among parties might go down industry-wide. In a sense you admitted that:
But then I can see it might help change things for the better across the board, as you nicely described. Thanks for the illuminating comment!
Temporarily lost ok. Better to let these manufactures do whatever they want.
If this goes on to criminal charges, then they're about to discover what amazing things a thorough digital forensics analysis can find out from their workstations.
Do you have more information about this sort of programming? I'd like to read about it.
IEC 61131-3
ah, thank you!
Google for PLC programming environment.
The IDE for these PLCs actually has VCS integration! It's SVN, but it's still better than nothing.
Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.
There is nothing wrong with SVN, it's just that Git allows for some workflows that are better suited to larger teams and more complex projects. But for your average PLC project with a team of 10 and one binary as the output it should be more than enough.
You likely won't see any 'feature branches' or frequent merges in this kind of environment.
Except merging things, and handling a lot of files...
There are lots of small things wrong with SVN. But it's indeed usable.
I'm aware of repos with a few million files in it that have been going since 2003 and not a single issue.
Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.
I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.
If it was developed anytime after 1990 (probably before) you will find plenty of programmers willing to be expert witnesses and tell the court that the company not having version control is gross incompetence, the only reason a company would do that would be so they can hide evidence of illegal actions. As such the court should impose punitary damages.
Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.
Oops, rogue hackers deleted our .zip files with the backups! :)
EDIT: BTW having no version control would be pretty telling on its own. It's a critical piece of software, that controls a train..
“Newag president Zbigniew Konieczek said that ‘no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software.’”
The ridiculousness of this defence makes it clear leadership was in the know.
Once this is proven false, they will blame the devs, just like VW did.
The dev. at VW was at least smart enough to keep the paper trail around. Lets hope the people working for this company did the same.
There's also a bunch of service people involved etc. (some "non-working" trains were sent back to NEWAG, and the people doing the maintenance had to know how to unlock them, etc.)
I wonder how they explain the "rogue dev" somehow clandestinely communicating the reset procedure to the manufacturer's shop.
I'm not sure that's the smoking gun.
IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.
And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.
The 2021-11-?? date was the day the train was scheduled to be serviced (so the intention of the code was to brick the code after the service). Only by random chance it didn't work, because the train was in maintenance a bit longer, and the code didn't trigger in January 2022 because of the date checking bug.
That is completely nuts and irresponsible beyond belief.
I guess they can't claim any copy write / drm violations if they also deny it was their code.
Riiight?! This is a really strong defense for the white hats. Either it’s your code and you are egregious slime balls doing illegal things, or “some third party interfered and hacked your trains like you said and we fixed it you’re welcome”.
It's the "you can't prove we did it" defense.
This gets penetrated by getting lower level employees to rat out their superiors.
https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne... - this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.
So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.
I hope evidence will be provided in court.
What lone developer says "let me find all the GPS coordinates of all the train repair shops and make the trains not work if they've been parked there for days"?
Don't underestimate these damned rouge developers!
I've heard of Khmer Rouge but not of Dev Rouge before.
Humor is so little appreciated these days.
Humor is prevalent in the internets various forums. It's well appreciated generally - easy upvotes on Reddit and elsewhere. But that also means that a serious forum with higher signal-to-noise ratio is something uniquely valuable.
We talk on HN a lot about ethics and programming and the importance of engineers refusing to develop unethical software.
But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.
Alternatively, if you're going to break the law, don't do that (break the law).
"My cat slept on the keyboard and accidentally developed and merged working advertisement feature into a release build of the new AC game. It was totally on accident, happens all the time." (c) Ubisoft
"I come to you now with apologies. This is the third time my cat has put paid mods into Skyrim, something for which he is very very sorry you are unhappy about."
Yeah, sounds like a secret 20% project :)
Management: "a 6-pack of beer for the brilliant dev who figures out a solution to this problem of us ..."
Is this really suggesting some developer, buried under tasks and putting extra hours, thought to implement this feature in his free time for months, slipped it through reviews and scans and pipeline deployments, for... what? Really? I rather prefer to think the comment was sarcasm.
Yes it was sarcasm, since it is common for management to shift the blame on individual engineers in these cases (see for example the Volkswagen emissions scandal and the Boeing MAX crash).
IIRC Volkswagen actually used the term “rogue engineer” when speaking to the press.
In VW's case there was some reason to believe that at first. There are reasons to test an engine in the lab without emissions controls, and so it is believable that a lazy engineer could put in code for the lab that also triggered in real world conditions.
Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.
As a lone developer there is absolutely no motivation to do this and hide this decision from management.
Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?
There is just no possibility where upper management is not involved in this.
100% agree. Doing this in secret without management's express approval means no bonus, working extra for no benefit, compromising your other tasks and exposing yourself to heavy liability for compromising the country's infrastructure. This is something that companies could get away with, but a single developers would not only lose their job, but risk going to jail.
Note that management's express approval could be verbal conversations that you cannot prove happened or not. With some work management can find some way to get a developer who has some other conflict of interest to do this: if your brother in law works for the maintenance yard situations then you have motive to ensure he has work.
If an employee did this without authorization, the company should still be liable because I do not think safety regulations exempt manufacture from responsibility of employee actions. I believe the opposite is true, that they must ensure safety despite rogue employees, with reviews, audits and such.
They need to show audits and that the code is obscure enough that it could be missed. I've seen IOCCC type things that do things I wouldn't catch in a code review. (IIRC there is a malicious code contest where the goal is to put backdoors in that others don't see in review, but I can't recall what it is)
There's been a decent amount of those in the wild, too. If you're in an environment without unicode auditing you can do all sorts of crazy things without IOCCC levels of code tricks too.
npm add block-repair-shop. It can happen so fast.
Guess. (Did I miss the sarcasm again? I always miss the sarcasm.)
His dog did it after eating the CS homework.
I suspect this is going to end all further sales of these trains outside of Poland. What railway will ever trust them again?