Dieselgate, but for trains – some heavyweight hardware hacking

My impression is that the quality of train firmware is generally not very good, and I hope that this scandal will lead to greater scrutiny. 3 years ago, Deutsche Bahn publicly complained of "grotesque" software problems with newly delivered Bombardier trains. For example, when train drivers changed the direction of travel, the train software would crash. It then took 1 hour to boot the train up again [0]. Switzerland had similar problems in 2018 [1].

As a computer scientist, I find this embarrassing. Just compare these modern trains to the old trains built in East Germany [2] during the 80ies that were pulling old West German carriages [3] from the 50ies here until recently. Minimal or no usage of digital electronics. No "boot times". They just worked. And if they didn't, the train driver usually knew where to hit the engine with a hammer to fix it. You cannot expect a train driver to hack into the train firmware and fire up gdb to find out why it doesn't move.

[0] https://www.sueddeutsche.de/wirtschaft/deutsche-bahn-ic-1.47...

[1] https://bahnblogstelle.com/33872/twindexx-swiss-express-soft...

[2] https://de.wikipedia.org/wiki/DR-Baureihe_243

[3] https://de.wikipedia.org/wiki/N-Wagen

Bad software is a symptom, not the cause.

What is the cause then?

Bad culture that views software as a necessary evil or afterthought rather than an important part of the product.

Same as industrial design then. You get the occasional Braun, Herman Miller or Apple, and a vast number of nondescript silver/beige/black boxes.

It's probably true of lots of aspects of product design - if it's not driven from the top, it's mediocre.

Yeah, unless the engineers are using the product themselves. People in general seem to take care of their own tools. Much harder to get them to look after a product they don’t use themselves.

Here it's more like the software - any software - is a problem. I agree with GP, and my experience confirms that adding software to something that used to work without it almost universally makes it worse in every aspect, understandability and repairability being just two major ones. On top of that, taking anything that run on old-school industrial/embedded firmware and replacing that with software using modern practices and stacks of the software industry, 100% makes the product go to shit.

That again is a problem of leadership and not of software.

That again is a problem of leadership and not of software.

SW has an input problem and a testability problem. On one hand, the inputs to the SW are not limited (iMessage happily accepts any image file) and testing is limited to some known inputs. Software vulnerability assesment (worst case analysis) is usually performed outside of the development process at very high costs and limited outcome.

I think the compensation given to software developers by companies that view software as their product has drawn many of the skilled software developers away from jobs that would have once grabbed them because of the fun factor. Companies that make things that contain software are not in markets prepared to pay 2 and 3 times what they were for software. What you are left with is people who are willing to accept that fun factor as the difference in TC, and people who couldn't get jobs that paid more. These are the people we have making most of our safety critical systems. Go look at software developer compensation at X vs spaceX. That is the market at work. Fun does count as TC, but you also end up with people who aren't good developers pivoting to engineer new processes and tools in these domains. They latch on to whatever fad full stack is just getting over a case of, and try to apply it to train firmware. It wouldn't surprise me to find out they are all about scrumfall and have 10x more text in jira than git. And they have restful apis, or service oriented architecture in a safety critical embedded system.

Facebook, Amazon, Netflix, Google, Twitter are not selling software but they are able to attract a lot of skilled developers

They all are selling software, with SaaS model.

You can compare this situation with Boeing. And issues they had with software of 737max.

Some of the problems here might have been logic problems by inept coders; however, the underlying theme of this scandal is corrupt management. Even the erroneous code was an explicit piece of fraud that almost certainly was done under order by someone in the management chain.

Speaking from experience in vehicle firmware. Typically, job requirements include hardware qualifications and web software isn't considered relevant experience. People with web backgrounds are in demand for adding telemetry and diagnostics data collection to products containing software, but those are separate components.

That is true. You can get few times more money as regular Spring Java developer making CRUD in some bodyshop than writing industrial software for local Polish company.

I think you are correct. Because the pay is so miserable the talent pool is mostly vba developing engineers from inside the company. Because of that they can’t hire good technical leads that know or can enforce good practices or design good architecture. The result is a giant mess of software in trains planes and automobiles

Companies that make things that contain software are not in markets prepared to pay 2 and 3 times what they were for software

The quality of SW has nothing to do with the pay. Notice that FAANG SW developers do not deliver safety critical SW.

There are more things to SW development than writing code.

Because this software is not made by software engineers, it's made by plc programmers, electric circuit designers and whoever did drift into the field.

Except for beckhoff to tc3 they haven't made it to object orientation yet, so the field is stuck as a whole in the blue screen mines of yore. Managing complexity with thin standard docs, no version control while the machines grow ever more complex sensor and actuator wise..

You can not treat modern machines like small embedded hobby devices - but the industry does.

Some outside-programmers make good money coming in and solving these yesterday's problems with proper software architecture and good c development practices. But the industries doesn't learn from this. Making software will forever not be a profession for them.

I'm not sure if you've ever used modern software. It's sometimes amazing just how unreliable it is. Web browsers crash every few weeks, windows is known for regularly needing a reboot, evince regularly crashes on me, you can't call 911 with some of cell phones, ... . This reminds me of https://danluu.com/everything-is-broken/ .

The clearest example of the difference of reliability is looking at public digital signage (on transit and elsewhere). If it's based on LED segments or something similarly basic (with old-school embedded software development) it will basically always work. New LCD Screens inside trains/busses and outside working with a modern software setup (using an OS, often with a pc architecture, quite often just displaying a website) are broken ~10%-20% of the time. Looking at (for example) busses, a large portion of the time the screen will either be blank, not display anything, old information or just wrong information. Going inside fast food restaurants with large LCDs for the menu, often something is broken, frozen or something else.

It is of course possible to make modern software more reliable. It's just much, much harder than making embedded software or PLC programming reliable. Software can be easily made more complex, but it's hard to make it non-complex or to wrap the complexity so it isn't an issue anymore. The ecosystem isn't set up for non-complexity.

I think to make software more reliable, you'd have to go back to the "waterfall" method of development.

If we went back to Dijkstra's notion of correctness by construction, then a specification for the program would be made, and then a programmer would prove their part of the code correct to the specification. They would write the precondition and postcondition of every effectful statement, document the invariant of every loop, and prove by induction that each loop does what it's supposed to do. Basically, annotate your program with Hiare triples. (There are books about how to do this). Then, extensive tests should be run for as much of rhe program as possible.

Nowadays, we have tools for this so that we don't actually have to write a proof by induction for every loop; instead, we have bounded model checkers. In theory, the manual proof writing could be isolated to the parts of the program whose properties a bounded model checker cannot verify.

However, it seems like this whole plan is infeasible unless regulations are written that enforce this onto the industry. It would make them a lot less productive, and therefore less profitable. The only benefit would be that software is more reliable. By necessity, it would have to become simpler, too. For instance, there's absolutely no way that web browsers like Chromium, with 38 million lines of code, will ever be verified, because they're too large and complex.

> ... they haven't made it to object orientation yet, ...

Not always a blessing and I've actually recently been thinking (e.g. in context of Lua) if object orientation is in most situations not better to avoid.

100% agree with this. IMO there are a few efforts to modernize PLC programming but I feel like they are still stuck in the 1990s software development. Take a look at Codesys, got Git support few years ago and in very bad shape. How do you test your code, in the field or buy another Codesys testing plugin....which is in rough shape.

The issue is as machines get way more complex this issue gets worse. Also there are generations of PLC devs that still want to stick with ladder logic. Huge fragmentation.

Because this software is not made by software engineers, it's made by plc programmers, electric circuit designers and whoever did drift into the field.

There are more "engineers" writing software for your car or a train than "engineers" at Microsoft, Google, Apple or Facebook.

I don't think that someone will be happy when driving with 100 km/h on a highway, the car will suddenly decide to restart itself. There are bugs everywhere where profits are put before engineering but calling those people names is not constructive. Especially when they use SW created by "engineers" which crash with no apparent reason when they are doing their work.

No "boot times". They just worked

Haha wait until you find out how TVs worked in the 70s and how fast it was to change the channel *sob*

Even in the 90s, you could just power it on and it would show image near-instantly. Warm-up time and channel switch time were all firmly under one second. With the exception of cable TV set-top boxes, which were separate devices and first to include the ridiculous boot times and delays, that still would seem blazingly fast compared to what we have today...

Sometimes low-tech is just better. Here in Finland we got Sr1 electric trains from the Soviet Union in the 70's, and after some renovations the model is likely to stay in use at least until 2030.

Simply of old designs is often a blessing as long as the drawing and documentation is readable and good. It can be hard to get replacement electronics for 1970s designs so sometimes you have to design new components but the functionality was relatively simple back then so it’s possible to build a 1:1 replacement

That is also my impression as well. The softwareization of trains has led to deep regressions in both basic reliability and interoperability/flexilibity. Many modern trains suffer from software issues for basic driving [0] and delays when getting the software approved [1]. But the loss of compatability is in my opinion the worst regression. Modern EMUs basically only work together with other EMUs of the same batch. Even the same model ordered by two different companies often don't work together and basically forget about trying to use EMUs of different companies or ordered over a decade apart together. Meanwhile pre-digital everything it was common to use e.g. trams of different generations together and rewire them to work with each other. Older train cars work together without issues, good luck trying to use an IC2 and a Railjet together (or a RailJet and ICE-L). Even certain locomotives and train cars would often only work with each other.

It is way harder for different computerized systems to work together due to the higher complexity and more obfuscation (a traditional logic circuitboard is often easily reverse engineered. Reverse engineering software is a very specialized task). This is also very noticeable in other sectors, where interoperability has become much worse due to moving to proprietary digital protocols.

This is in part due to the difficulty in getting software approved as compared to previous tech (due to software being so intransparent) but also because of truly lacking quality. One of the reasons Bombardier was so deep in trouble was bad software, even leading to a contract of over 40 ordered trains just being cancelled ([2]).

In my opinion building reliable (and understandable) software is way harder than building logic or even mechanical systems. I don't know what the solution is, but it's been a problem for a long time.

[0]: https://www.vrt.be/vrtnws/de/2013/02/12/belgische_bahn_storn... [1]: https://www.augsburger-allgemeine.de/augsburg/Neue-Zuege-auf... [2]: https://de.wikipedia.org/wiki/Bombardier_Talent_3#%C3%96BB

Except that this isn't really a story about poorly written software; it's a story about corrupt management. Further, if we look at Boeing's recent issues with the 737Max, it's the same thing. In both of these cases, the bad software was almost certainly ordered to be written by management acting fraudulently for profit. The one error that has been discussed in the article was a stupid mistake, quite possibly due to the logic conditions being made overly complicated in order to enable the fraud, but the recurrent theme of all of the real underlying issues found was intentional design malfeasance, not incompetence.

You could say that about most software where the fresher the framework the more glaring the holes - here's a recent post about it: "Software disenchantment" https://tonsky.me/blog/disenchantment/

I don't think fixing the software failure will improve DB's punctuality

In the case of Bombardier, I suspect contracting also contributes to the problem. The same for financial institutions.

But was this sabotage by an insider at the manufacturer, or something deliberate by the manufacturer?

There wasn't just one manufactured failure, but multiple different ones. Refusing to help would also point towards intentional malice. Why would you sell a product, then refuse to assist, unless you've intentionally designed the product to fail so only you would know how to make it work again?

To what end? So they can sell more trains? That makes no sense.

The train manufacturer, Newag, also competed in the tender to carry out the maintenance, but the manufacturer’s bid was about 750k USD higher and the tender was eventually won by SPS, which offered to carry out the maintenance of 11 trains for around 5.5 mln USD.

Just thinking outloud. But if you made it so your competitor couldn't fulfill their servicing contract, then the entity taking out the contract might just very well come to you to solve the problem. You might not win the contract on price, but win it by default because you made it impossible for anyone else to complete it.

That is until your scheme is uncovered because you left the GPS coordinates of your competitors workshops in your code.

More sanely (not to be confused with likely!) the courts will decide that since this is something only the OEM can do, it must done at no charge as part of normal warranty work.

These trains will be used for decades. Normal warranty wont cover anything of note.

Warranty should cover this - if the manufacture won't let it be fixed by someone else than in should be free.

Every once in a while there comes a point where the discussion of high-currency-shorthand pops up:

5.5 mln USD. U$5.5m? Not saying I'm more correct than anyone else, but the former seems outlandishly long.

mln is from the Polish original.

5.5 millidollars?

How does that make no sense ? That's the whole point of a business.

The idea that does not make sense is that this would increase train sales, not the idea that selling more trains would be good for business.

It seems like the trains were programmed to cease functioning if they spent more than 10 days at the GPS coordinates of maintenance shops not owned by the original manufacturer.

This would force the government to rely exclusively on that manufacturer to then fix these trains and perform all future maintenance.

They wanted to prevent third party repair services from being able to repair their trains, so that they could keep those maintenance contracts for themselves.

After sales support, as in spare parts and maintenance, is a big part of income for manufacturers of heavy equipment, as such machines run for a loong time given parts and maintenance. To me they really did not want to lose on 'subscription money' in the form of service contracts they missed out on. It came close to the operator coming back to them to fix the trains 3rd party seemingly couldn't.

The manufacturer lost the bidding process, so quite reasonably (if you look at it in a limited fashion) said "Fine, let SLS do the work, you're on your own".

Arsehole-ish, but not illegal. All the hidden lockouts on the other hand....

I don't think it's assholeish for someone who's not getting compensated in any way to not help out. It's a business. They have an active incentive to NOT help.

It's not about "not wanting to help". It's about placing logic bombs of "if vehicle is at this gps coordinates of a competitor, engage self-destruct". Hackers actually did extract such coordinates from train firmware.

unless we have the entirety of the context for this code and the 20,000 pages of service manuals, i do not accept at face value that it's this simple

Any kind of GPS coordinates, especially those of competitor facilities in the firmware of a train is proof positive that something really bad is going on.

Context and manuals are just so much smoke and fail to obscure the facts.

The hidden lockouts containing GPS coordinates of competitors' repair facilities should be more than enough to establish criminal intent (in my armchair non-lawyer opinion).

They knew that SLS would not be able to do it.

If the manufacturer did it, doesn't it still fit the definition? It's something like "deliberately causing something to fail", regardless of who does it.

While I believe intentions were malicious, it's very easy to argue that

1. it's not failing, it's disabling

2. it's a safety feature - "SPS can't safely maintain these trains, so we have a safety lock out if they attempt it"

3. there is a ton of stuff that works this way - even Harley Davidson motorcycles require authorized maintenance and the bike's computer won't accept repairs unless a proprietary tool is used

Newag was required by contract to provide accurate service manuals so that competitors could safely maintain the trains. This was not a "just take your car to dave, he knows some stuff". For SPS and other competitors this was like "you need to show every certification that exists and certify all your tools to prove that indeed you can service those cars, or you will be foreclosed due to fines". Plus, they were provided ALL service manuals, like 20k pages to follow to the letter.

i wouldnt be surprised if this info was somewhere in those 20k pages, and perhaps if the procedures were actually followed, stuff like GPS based lockouts wouldn't happen

Ah yes, on appendix 35 of section C, "do not store the train in your service yard specifically or it will stop working"

The article covers this, and says the information about the lockouts was not in the manufacturer provided manuals.

On #2, that's sabotage. Also, on #3, that's sabotage too.

The higher-up the decision went, the worse it is.

Considering that the "sabotage" was intended to bring the company extra revenue by having non-faulty parts replaced and by requiring maintenance to be carried out by them and never third parties, it aligns with the company's own interests too much so to say "some employee did this without authorization".

"Deliberate by manufacturer" 100%.

The scarier part is that had this happened in the United States, DMCA would likely have protected them from prosecution, and the government might be liable for damages.

Its insane how brazen this is. Code that 'bricks' the train locomotive if its gps coordinates remain with bounds of a competing repair facility for more than ten days! This is way beyond putting information barriers to repair, like undocumented interfaces or even crypto-signed firmware. This is actively malicious destruction of property. I don't know anything about the legal system in Poland, but I can't imagine how this gets by.

If an individual did this, they'd go to prison.

It's funny how, in the western world, as a company, you can commit crimes and take a pat on the wrist, but, as an individual, you get to jail for the same crimes.

Sadly, the general populace didn’t hire lobbyists to represent them. Our representatives were supposed to be built into the system, but that unfortunately made them part of the game, rather than some of the players.

Can’t you create an NGO that will collectively represent and lobby on behalf of the group, hiring lobbyists from membership fees and other fundraisers? Holy hell, maybe create a political party?

Me personally? No, I don’t think I have the connections, patience, or talent for that. If I did I’d probably do it for a big company instead, they pay better than “we the people,” I think.

rules for thee but not for me is not a western invention

It’s simpler than that. A rich company or individual can often avoid jail. A poor company can just fold. It’s the poor individuals who suffer.

Companies are made up of individuals. I'm all for holding everybody who contributed to this malware accountable.

I don't believe the Polish judicial systems has experience in dealing with corporate crime, especially of the tech-related kind. I'm a bit afraid of disappointment here.

Does it have experience in dealing with...sabotage? Specifically, a country that has a war on its eastern doorstep?

I mean - how is "let's mess with something on purpose so that trains won't run" NOT sabotage, since such time as railways exist?

Wait, what? You don’t think a country with a population of 41M has experience with corporate crime?

Unfortunately that is why fall guys were invented. I never liked the idea of punishing a company based on their revenue, but in this kind of case that is the only way to get the actual owners of the company to listen and punish the people actually responsible.

No, they would not. It would be entirely a civil matter that would be resolved in litigation.

Example prison terms below. I suppose in Poland sabotage might be handled differently from a legal perspective? I am admittedly not familiar with the Polish legal framework.

https://www.reuters.com/article/tech-usa-crime-computers-dc-...

https://apnews.com/general-news-97f379af27b94419a2b6a3620fd7...

https://www.cbsnews.com/sanfrancisco/news/citibank-sabotage-...

https://www.bleepingcomputer.com/news/security/siemens-contr...

https://www.csoonline.com/article/544764/it-wary-of-insider-...

EDIT: https://news.ycombinator.com/item?id=38571144

A contractor in the UK put a time-lock in the software he was contracted to write because he was concerned about non-payment. He didn't get paid and the software duly stopped working. He was successfully prosecuted under the Computer Misuse Act. He had some justification (unlike the Polish train manufacturer) but it didn't help him avoid prosecution. I've no idea what the law in Poland says.

Article 254a of the Polish Penal Code addresses the obstruction of railway operations and other critical infrastructure. Violating this law can result in a prison sentence ranging from 6 months to 8 years.

It doesn't matter whether the act was committed as part of a company's operations or as an individual's private endeavour.

To all software engineers: please refrain from engaging in criminal activities. If you are instructed to do something illegal, it is important to report it to the relevant authorities.

[delayed]

And if it is a big or even state company we need to save and ensure workplaces, or "hello dear lobbyist with that big suitcase!" :D

You could very seriously start a war by doing things like this.

It will be stuck in legal hell due to conflicts of interests. Trains already exist, and they need to work - but maintenance/repair companies cannot legally modify software of them due to copyrights. It's a catch22 situation.

I honestly hope that company will be fined to the oblivion, and for criminal charges for that, but i doubt it will happen.

Supposedly tho the maintenance company would want to sue to at least dispute their contractual penalties?

A day of train downtime in the workshop costs over 1000 USD in contractual penalties, and there are several trains stuck, so the tension level in the SPS is rising.

Also LSR, because evidently they were interested in holding a tender before and so likely don't want to be forced by Newag into overpriced maintenance contracts?

This is the kind of thing that will destroy a nation's manufacturing industry overnight.

Who in their right mind would buy kind of equipment from a Polish company knowing that this kind of nonsense is both widespread and that their legal system has no solution?

Hoestly, "Dieselgate" is not a fitting corollary for this travesty. This is considerably more sinister. Hopefully whatever happens from here will be an agent of change for the better.

Of course laws vary, and Polish copyright law might be completely crazy, but around here copyright only covers distribution of copies. It does not make it illegal to modify software that you own. It only limits distribution of copies of that software, modified or otherwise. If the owner of the train wants to modify the software then there is probably nothing stopping them.

In a properly functioning country the responsible persons should already be imprisoned. Some governmental agencies were aware of that for at least half a year, but failed to act. The fact that source code was not immediately dumped and analyzed is the evidence of malevolence, corruption and intentionally putting people's lives at risk.

Welcome to the dark side of Poland - where citizens don't matter.

Corruption is not just the dark side of Poland, but the entire west IMHO

The west is the least corrupt part of the world.

https://en.wikipedia.org/wiki/Corruption_Perceptions_Index#/...

Note from that wikipedia article

The Index only measures public sector corruption, ignoring the private sector. This, for instance, means the well-publicized Libor scandal, Odebrecht case and the VW emissions scandal are not counted as corrupt actions.

Because corruption is when state power is abused. When private companies do illegal shit it’s just a crime.

There is no requirement for it to be "state" power, please look up the definition.

Okay, whatever. Pretty sure they private sector corruption isn’t worse in the west than in other parts of the world but if you want to disagree I won’t be able to change your mind

perception != reality

Perception is a proxy for reality, given that you can’t measure the latter.

There are reasons for why we can assume that the west is the least corrupt area of the whole world.

It baffles me that people don't realize just how bad it is in non democratic countries. Russia has FSB extorting shop owners for protection money, and even an occasional assassination is nothing particularly interesting there. Chinese companies have party cells in management. Venezuela or many African countries need to hire foreign contractors (sometimes Western :) ) so that their heads of state and other VIPs do not get killed by their coworkers. The Red Sea has actual pirates. Lebanon failed to remove kilotons of explosive ammonia nitrate for years, until it eventually blew up the capital. I could go on and on, but you can see how this compares to "train company bricked a train and it's a major scandal".

The other 3 directions are corruption-free?

can't speak for the other continents but there is very little corruption in Antarctica, so I guess if you go South enough, it is actually better

I wanted to get numbers on this, but naturally it's not feasible to get accurate numbers on corruption happening. I found the Corruption Perceptions Index which seems to be the closest we have in quantization. By measuring perceptions of corruption, as opposed to corruption itself, the Index may simply be reinforcing existing stereotypes and cliches though.

But according to their results the "west" has the least perceived corruption.

https://en.wikipedia.org/wiki/Corruption_Perceptions_Index

Yes, the west is corrupt. The rest is worse though, and less ashamed of it.

"the responsible persons" ... hmmm. Who would that be?

The programmer who implemented the code? Do you think they thunk these tricks up? They was just following orders.

The manager of the programming team, who set these tricks as things that needed to be implemented? Again, just following orders.

The "Cxx" Title people who directed that there be "some protection" in some way that got implemented as what we see? Did they specify these measures? Did they say "it should break if serviced by a competitor?" Unlikely. Thye wouldn't know how to be that specific, probably.

Some middle manager, maybe a committee meeting, sketched out a "DRM" scheme with the specifics? What do you imagine that meeting looked like? "We've got a directive to secure the systems from outside tampering, what does that mean in terms of how the machine behaves?" Or does that bring us back down to the engineers again?

... the responsible part here isn't a person, its the company as a whole. Just as it took the collective efforts of everyone to make the train, it took their collective efforts to make it wrong.

Corporate Death Penalty; perhaps. make it plain that we will no longer tolerate sill shenanigans like this.

That would be at least everyone knowingly involved with who is a professional engineer.

Everyone in FTX except SBF should be innocents then. They was just following orders.

IMO responsible is enginer and everyone up management chain + possibly peers if engineer part of a team. And those people should have a trial.

Again, just following orders.

I seem to recall there was a trial in the forties of some relevance to Poland about this sort of thing.

"the responsible persons" ... hmmm. Who would that be?

But that is the thing. We do not know who is responsible without an investigation.

We don't need to guess. The local responsible agency should get a warrant and take a copy of their code repo and their internal comms. And then they need to spend the time (call in experts if needed) to figure out what happened and who was involved.

If it is normal code development you can find all the paperwork which documents the change. If they tried to disguise it, (which they might have, or might not) then that is some maffioso stuff and you take the tools police use to break up organised crime groups. You take a low level person who you can incriminate and you flip them. You show them that you have enough to send them to a prison for years and offer them the opportunity to cooperate.

They was just following orders.

In itself that’s not enough to be considered innocent.

Well, justice takes time and this is a complex novel case. I would rather have a system that is right than prematurely put innocent behind bars. However, if the allegations turn out to be true, which seems to have a decent probability, they could charge them with a criminal offence.

There is Article 254a in the Polish Penal Code. If you obstruct critical elements of infrastructure such as trains, you can face between 6 months to 8 years in prison.

Great advert for free and open source software.

As with dieselgate, this suggests you basically cannot trust anything containing software. Can't trust it to follow regulations. Can't trust it to do its job.

Can't trust the software. Can't trust the institutions that write the software.

All very "late stage capitalist software development".

Hell, even if governments are squeamish about requiring code to be fully open and public, they can still require the manufacturers to privately submit to the government all code that powers public infrastructure (like trains), to be made available to any relevant party upon request.

An organisation that is prepared to write "sabotage" software would have no problem deploying software that is different to the software they submit.

Compile the code yourself?

Right. Mandate that the software is delivered with CI pipeline running in the client's environment with 100% reproducible builds and verify checksums.

Doesn't mean it's not a step in the right direction. Any transparency is better than zero.

Implying that's an impossible obstacle. Reproducibility is a thing.

Make it so code needs to be reproducibly buildable. Only reproducibly buildable artifacts can be deployed on hardware. Document the whole process.

That would work only on paper. The financial interests involved are huge.

All the more reason for governments to insist.

code escrow in general should be much more common.

can still require the manufacturers to privately submit to the government all code

I wonder if companies purchasing trains could put code disclosure in the purchase contract? I wonder if, in aggregate, train purchasers or car purchasers could fund an independent code storage vault and pay a small premium to fund that code vault organization?

In other words, if purchasers wanted this and valued this, they would demand it in purchase contracts and fund it.

then you just need to bribe the code reviewer(s). open source is still the better answer, good luck bribing every member of the public who could potentially read public code.

I'm all for free and open source software, but what would you suggest here? That train operators will download code from the internet and install it on their trains?

Yes. Once it's signed by somebody accredited to review it for safe train use.

Clearly not. A reasonable expectation might be though that if you want to sell your multi million pound products to a captive public sector, you have to publish all the source code and the means to build the binaries.

The same way technical diagrams for roads, bridges and other public infrastructure are public.

In most OECD countries food needs to be labelled with a full list of ingredients.

Your GP can read scientific papers about the efficacy and risks of a new treatment.

(Yes, many papers are paywalled but that's irrelevant compared to secrecy)

Open source means just that - it doesn’t imply one sort of distribution mechanism over others.

It doesn't actually need to be open source. If they published binaries that would be enough to analyze.

I wonder if the solution to all these screwy engine controls (tampering with emissions testing, preventing 3rd party repairs, etc.) is to standardize the interfaces to these systems so they can be replaced.

Standardizing the outputs of the sensors would let us swap in and out various components to ensure the system is not cheating the regulators.

F1 does this. All teams are required to run the same, approved, ECU[1]. They can change certain mapping tables and such but it's a sealed unit and they can't replace the firmware.

[1]: https://wheelsports.co/formula-1s-standardised-ecu-explained...

This is quite interesting because you can imagine that lobbyists would argue that standardization would “stymie innovation.” If F1 does it why can’t you?

Indeed. Standardizing certain components may reduce some potential innovation, however I've long thought that the public sector would be better off buying modular systems with well-defined interfaces rather than the behemoths do-it-all oh-so-often fail.

At work we're a small team, providing a B2B application to perform a small, but very important task for our customers. We integrate with tons of other systems, at our largest customer we talk to 30 other systems. We're highly specialized and we rely on being good at exchanging data with other systems that are good at what they do.

This allows us to innovate and provide great value for our niche, while the other systems can focus on getting better at what they do, rather than implementing a half-assed solution because it's not their core focus.

One could easily argue that F1 hasn't innovated much in the last decade or so. The coolest stuff we get is clever aero and advantageous workarounds that get outlawed extremely fast.

Most of F1's "innovation" is around finding ways to beat the rules, not necessarily coming up with new technologies.

Or even just requiring the manufacturer to provide all source code to the customer and the tools to update/replace the software. Would be nice to get rid of the black holes that is firmware and allow for auditing.

It's a bit more than standardizing, since you must also remove the barriers to changing the software. And you don't need full standardization, just publicity.

But yes, it's basically it.

There are devices for automobiles that intercept sensor data and feed back fake data to the ECU to bypass emissions controls. It's a fairly simple to do.

I have a buddy with a WRX that absolutely should not pass smog, has no cats, big turbos, tune, etc, but it has no codes, passes every time without issue because the sensor data is synthetic that governs those things.

Dieselgate isn't a good comparison because in Dieselgate the equipment functioned normally from the user's point of view.

I think it is because the equipment changed dependent on context. In Dieselgate the cars changed their engine management when they got into a test cycle...

Dieselgate was about cheating environment sensors. This is more like DeereGate, locking out external service shops but even when you are supposed by law to allow them service (and even after providing them 20k page service manuals which they are supposed to follow to make appropriate service, but you lock them out anyway).

From my reading, they also seem to have seeded apparently random failures into the product, with a hidden reset key combo, even for those using them as support. Possibly to make themselves look good (our products may break randomly, but at least we fixed the "problem" quickly) going into the tendering process for support.

functioned normally? Intoxicating people with fumes is hardly what the user wants.

Some car users even do special modifications for "rolling coal", so that they can intoxicate other people with fumes.

And under test it even performed “too well”

I thought it was a good reference. In both cases, the manufacturer placed illicit, hidden code that that could (and probably should) get it in trouble with the law.

It's one thing to implement a secret handshake and underdocument some procedures to make your competitors look incompetent, but actively breaking your product when it's in your competitor's shop - that reqires some chutzpah.

https://news.ycombinator.com/item?id=38345858

That wasn’t intentional though.

Exactly, like that time they slowed down only chrome and every other browser was still fast. Oh wait that never ever happened.

Ooooops! Somebody put this delay here tooooootally by accident and nobody noticed it when shipping to production! Silly devs!

Totally unintentional, I'm sure.

*provable, you mean

Exactly what I immediately thought of as well.

Newag issued a statement since, denying all allegations and saying that it was their competition which "hired hackers to slander them".

I've met q3k because we used to work at the same company and briefly on a project together. Not the kind of person I would suspect of participating in a conspiracy of this sort and Newag's statement generally reads like "we didn't think we would get caught".

^I think this is being downvoted because of poor reading comprehension skills. Please note that the parent comment is in favor of the hacking group.

Thank you for pointing this out - I reread the post and can imagine now how someone would read it differently than I intended.

Unfortunately for Newag, other than in the court of public opinion, firmware deliveries count as written evidence.

While I haven't met the guys in this case, I am familiar with their work.

Additionally, I am fairly certain they are not stupid enough to not have kept detailed, forensic-quality records of their actions and whatever they dumped. Sure it may not stand up in court as evidence but it will be more than enough to show that they didn't pull this out of nowhere

Nevermind malware, not using seL4 should already be a crime in this context.

We have rovers on Mars and satellites and probably nuclear warheads using RTOS of all kinds and in cases even Linux, but sure seL4 is the only OS conceivable for those cases, obviously !

This is a case of fraud, industrial malfeasance and just plain dishonesty. The software component of the story and its security measures are not even at play. Sure they are probably shit (given the date parsing ...) but even FreeRTOS would make an amazing OS *IF USED PROPERLY *.

using RTOS of all kinds and in cases even Linux

Absolutely, and thus there's obvious room for improvement.

This is a case of fraud, industrial malfeasance and just plain dishonesty.

In practice, this amounts to critical infrastructure sabotage, which fits into terrorism.

If the train network experiences issues, the whole country is impacted.

That seems totally orthogonal; seL4 can run a program that checks GPS and sabotages the system just like anything else.

How is this different from companies like Apple or John Deere that DRM components and brick the device if repaired by "unauthorized" technicians?

(I think both are equally egregious personally, but I know there's a lot of support here for Apple, so I'm curious how people reconcile these. I don't want to make this a religious war about Apple, but those practices in general regardless of which company is doing it).

Is it the secrecy that makes it different? i.e. if the train company were honest about it then it would be ok?

Or is it the scale that matters? Trains are big and expensive, while phones are small and cheap, so it's ok? (that wouldn't work for John Deere but would for Apple)

I'm not a fan of Apple's practices, but there's some aggravating elements to this. Apple doesn't brick your device if it it spends time at a repair location, for instance. Apple also doesn't simulate failures on synthetic dates to force repair.

Is it the secrecy that makes it different? i.e. if the train company were honest about it then it would be ok?

IMHO mainly that and clearly those trains are required to be designed in such a way that they could be repaired by a third party (either by law or by contract based on how the situation is described).

Apple provides (nor is required) no such guarantees. Also it has more or less legitimate reasons for its design decision (making it harder to reuse stolen parts).

equally egregious

I certainly disagree almost completely. With Ape you know what you’re getting and can make an inform choice. Also it’s a completely different type of product. Trains have various regulatory, safety and maintenance requirements which are irrelevant for consumers devices. Screwing with the software controlling trains can literally kill people..

An angle you may not have considered is passenger safety.

Imagine if this happened to an airliner in flight: there'd be criminal charges for sure, not to mention huge damages and lawsuits from the families of the dead if some of the control systems locked up in mid-air.

Trains are not quite as susceptible to disaster arising in the course of operations as airliners, but a Newag Impuls 45WE runs at up to 160km/h in service with up to 218 people on board. (Their speed record is considerably higher.) A sudden breakdown in service is at a minimum going to cause timetable havoc and knock-on delays for other trains and at worse could lead to a mass casualty accident.

(John Deere tractors don't usually carry 200+ passengers and Apple computers don't usually get deployed in safety critical situations. So, different!)

it is hard to find an institution in Poland that has done anything beyond kindly expressing interest in the matter. We are not aware of any action taken either by the Office of Consumer and Competition Protection or by the Railway Transport Office,

That the worst part of all that.

When the companies see that this behavior is not punished, they'll basically need to implement their own versions of it to stay competitive!

It’s cute that you think they haven’t done that already…

The government anti-corruption office is formally investigating this now, which means almost certainly people will end up going to jail. The office of consumer protection doesn't have anywhere near the power these guys have.

I'm solely consuming EN content. And if it's from another country, it's only whats leaked by big media. It make me wonder how much good content could be translated.

I think you're saying, how much other good content is out there that I'm missing out on because I only read English, and it's a good point.

However, English has become the (now ironically named) lingua franca of, at least, the more educated parts of the world, and many people who are most comfortable in their native languages are still often translating their best work into English in order to see it more widely read. This is often the case with scientific papers, for example.

Perhaps England's biggest gift to the world was its language.

Worldwide colonialism wasn't exactly a "gift", but I must admit it has been advantageous to me, personally, for English to be as relatively universal as it has become as a result. ;-)

Seems like deliberate sabotage via software to force the costumer to buy the manufacturer’s services instead of 3rd party (cheaper) ones.

Curious to see the court’s decision.

There’s no question that it’s sabotage. The only thing left to prove is the culprit, which is with 99% the manufacturer (motive, means, opportunity) but obviously need to be established in a court who is responsible and criminally culpable.

The fact that lawmakers, courts and the public are lost in the tech is a problem, but surely this crime can be fitted into existing criminal code against sabotage… although the methods are “new” the crime itself is classic.

"Lawlessness is the condition in which your adversary refers you to a law he made."

Title is a bit misleading, because this *gate is not about faking ecology, and trying to pass certification in artificial conditions, as dieselgate was, but simulating fake failures instead. The company hardcoded algorithms that would report failures of parts that work correctly (like a compressor), if it detected that train has been repaired by another company (based on location readings), and stop the train from running

It's an example of fraudulent / malicious behavior found by decompiling industrial logic controllers, with incontrovertible evidence of illegality. Obviously no two situations are ever going to be the exact same, but I think it's clear why the analogy was made.

Dupe: https://news.ycombinator.com/item?id=38558120

No, it isn't. Six points with no comments don't count as already discussed.

This kind of thing reminds me of 737 max debacle.

Outside the obvious issue in this article I found the following statement horrendous:

it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested

Instead of having one public company mastering the art in its entirety everything is split with contractors. A good example of a successful way to do that (but slowly dying thanks to capitalism) is SNCF operating everything in a massive warehouse https://www.youtube.com/watch?v=SeRH2M2Z-ms

See also the discussion/comments of the previous post of a related article: https://news.ycombinator.com/item?id=38530885

A train manufactured by a Polish company suddenly broke down during maintenance. The experts were helpless – the train was fine, it just wouldn’t run. In a desperate last gasp, the Dragon Sector team was called in to help, and its members found wonders the train engineers had never dreamed of.