Seems like Boeing might not have the documentation the NTSB is requesting[1], even though they say they do keep that documentation. I wonder if Boeing doesn't want to admit that for some legal reason, maybe they don't have a lot of documentation they say they should have.
[1] https://twitter.com/David_Slotnick/status/176545419893241038...
Not sure if this was edited in later, but it's in the article:
In which case the NTSB will hopefully pin the blame on the lack of a functioning system of record that tracks such events.
Easy solution: Find the common manager of everyone that could have screwed the door up, and hold them responsible.
There are two companies and multiple teams involved. It sounds like this is a process gap. One company has a process that states to remove and reinstall the doors but doesn't mention checking torque. Second company takes over and their procedure states to check fitment but not torque.
Something like that is likely what's happening here.
That's quite plausible, but at some level in the hierarchy, there is someone who has the responsibility of not letting things fall through the cracks - if no-one else, then it is the CEO.
If outsourcing is a de facto means of avoiding responsibility, then guess what the consequences are.
This may be satisfying but does nothing to prevent a recurrence. Fixing the process is much more useful if you want safer aircraft.
Why should public money fix a process that will help Boeing shareholder value?
The executive leadership should lose their own (potential?) personal money over this incident. Until that happens, nothing will get fixed.
We don't know the specific process failure. While I appreciate your anger, it's a bit premature.
It is not sufficient, in itself, to prevent a recurrence, but adjusting the incentives to encourage preventative measures is a time-honored and generally effective way of getting results. It is more effective than shrugging one's shoulders and saying "It's so complicated! It can't be helped..."
Wouldn't the be the CEO?
Presumably you’d want the lowest common manager within the power structure.
Unfortunately that's still the CEO. The problems with Boeing are company wide.
The current one's only been CEO for four years; the structural problems presumably are older. Like, they probably _should_ consider losing the CEO, but that in itself is not going to fix anything.
The CEO has been on the board for much longer than 4 years though. And I agree that losing the CEO isn’t going to fix things. But putting leadership there that places safety above share value will. And the CEO is part of that leadership that needs to change.
Dave Calhoun was a member of the Board of Directors loooooooong before he was crowned CEO. Since 2009. Dave knows where all of the bodies are buried. In fact, he was on the board, and it’s lead director, when the 346 bodies of the Lion Air and Ethiopian passengers were buried. The man has blood on his hands.
You must be new here.
But that manager worked so hard to be the person who gets to decide who to blame!
Finding someone to blame isn’t the point. That’s for the influencers, politicians and armchair commentators. Engineers should seek to find solutions; the NTSB and FAA have a terrific track record in this respect.
Can you really do that though?
You have to trust your workers to not undo some random bolts here and there at some stage.
so its worth clarifying something because 'plug door' is a confusing term as it relates to the bolts. The issue here isn't about trusting workers to do things, its about systems and processes to ensure they know what the right thing to do is and they know how to do it correctly. You CAN (and SHOULD) use things like checklists to ensure people do each job correctly. There are tens of thousands of 'jobs' in an assembly like this - no one knows/remembers exactly how to do each one correctly, nor should a system expect rely on them doing so. Here's how that happens without anyone doing something 'random' and following their instructions as they understand them:
There are doors on an aircraft - you enter through them, they have inflatable slides attached. Then there are plugs - plugs that fill a space (this is one, there are a lot, basically any penetration of the pressure vessel). If there were a space where a door could be but isn't on this model, you need to plug it (hence, door plug or...plug door). In terms of assembly, maintenance, operation, etc. they have different functions and those functions have different documentation and traceability requirements. Doors are meant to be opened and closed - that is different from being removed (i.e., 'taking the door off its hinge'). That leaves us here:
* Doors can be opened and closed (this does not require documentation because it is part of the normal function)
* Doors can be removed, which is different from opened and closed (requires documentation because it is not part of its normal function, requires inspection after reinstallation)
* Plugs cannot be opened and closed (opening and closing them is not part of their normal function so...to whit...doing so is impossible within the vocabulary/concept of documentation rules, they are not designed for regular use)
* Plugs can only be REMOVED (documentation required, inspection required just like a door removal)
Where does that leave the door plug? is undoing the bolts and pushing it up off the stop blocks opening it or is it removal? In the minds of Boeing and the FAA - that is removal. In the minds of line workers, opened a door.
Suddenly, door plug and plug door become potentially different concepts. One is a plug for a door and is discussed and documented like a plug. The other is a door that serves the purpose of a plug...and is discussed and documented like a door. What seems to have happened is that workers opened the plug , which isn't 'possible' - meaning they 'removed' the plug and treated it like opening it. That meant it didn't trigger anything about a required set of procedures to do so, didn't trigger inspection, didn't trigger documentation, didn't trigger inspection, just got closed up and moved down the line.
I've dealt with a lot of these...a personal favorite was a company I worked with having someone who couldn't read be responsible for recording each serial number he stamped on each federally tracked part in our factory. He relied on his memory of the last serial number he did the day before.
I thought the plane eas delivered with the plugs missing because a contractor failed to install them.
It’s in the article (and every other article written about this debacle).
The plug was of course installed when the plane was delivered, otherwise there would have been a door-sized hole in the side of the plane when it was delivered.
The bolts of the plug were probably originally installed, but some rework was required in the area during final assembly, which required the plug to be removed and put back, and during that process the bolts were not reinstalled. It’s (lack of) documentation of that removal-and-reinstallation that are the subject of the article we are discussing here.
Lets be clear - I do not know the answer to this.
What I have read varies. Some say the door plug was installed and bolted. Some say the door plug was installed but not bolted. Some say the door plug wasn't installed - which I suspect from a lot of prior experience is a linguistic gap between engineers and reporters on installed but not bolted.
I remember reading but can't find at the moment something suggesting that inspecting the door plugs and the bolts was a required Boeing assembly line 'job' (think 'ticket'). But none of that matters if someone, as has been reported, later uninstalled the bolts to fix something else, without creating a record of doing so, because they misunderstood the specific procedure they were undertaking.
If they really do allow line workers to interpret federal regulations themselves, that's quite a problem. But I doubt this was a decision made by someone that turns a wrench.
But what if there's absolutely no documented process involving those bolts at all? Its perfectly reasonable to ask what the process is and who is responsible. Boeing acknowledges they don't have an answer here. Is it because they never had a process for those bolts to begin with? You can't document following a process (or not following one) when there is no process at all.
The process to do something undocumented could be to plan and write down what you are about to do, record the before during and after states file the documents.
Alternately the process should be to do nothing until the system studies the action and defines the process. But even then you have a proofing run where you probably should have extra documentation. And arguably in a high safety environment- this would be preferred.
Back upstream to the source of the problem, too much rework was already being required by the time the components got to the Boeing assembly plant (i.e. the Spirit techs onsite doing repairs).
In an environment where process exceptions are the norm, people begin to deprioritize handling exceptions properly. Everything can't be critical or emergency all the time.
And once that deprioritization happens, safety critical process exceptions begin being treated like all other exceptions. Because they're all exceptions, and there are so many.
And eventually normalization of out of bounds situations via repeated team exposure leads to catastrophe.
NASA lost two shuttles via the same organizational problem.
That’s all very true, but just a casual opinion, I think Boeing's process problem is more one driven by prioritization of profits with management corroding the safety culture vs nasa procedural breakdown missing risks because of routine complacency of many previous launches.
Then Boeing management is responsible for not realizing the process was missing. Peoples lives were at stake here.
I worked on airplanes in the long ago. You have to trust that people document every bit of their work and get an inspector to sign off on it where required and get in process inspections sometimes too depending on the work. Nobody who does undocumented work on an airplane should be employed.
If you have a culture in a highly regulated industry where workers can violate those regulations and you dont notice by intention or by inaction, then yes you can be held responsible.
How weaselly.
So... Maybe documentation exists. Maybe not. Either way, maybe they did not share it.
Shouldn't the reporting make it clear how much of a non-answer this is, or are they intentionally writing news reports laced with dry humor?
It’s a big deal because aviation is one of those highly regulated industries where the documentation not existing is it self a crisis level event. Like VW emissions scandal bad. That’s if it never caused a single incident, which it obviously has.
I fly a 40-year-old Cessna. If someone replaced a light bulb in 1982, I have a record of it.
" I wonder if Boeing doesn't want to admit that for some legal reason, maybe they don't have a lot of documentation they say they should have. "
This would be a bad legal strategy. Playing this through as a lawyer:
1. You will be asked to produce it in a lawsuit
2. It will come out it does not exist but should.
3. You will be absolutely and totally excoriated in the lawsuit, in the court of public opinion, and by your regulators.
4. You will prolong the news cycle of your malicious belligerence, and it may even spike higher when it comes out the docs don't exist but should.
5. You will be forced to settle the lawsuit at very high cost, and because you were totally belligerent, you will be regulated much harder, which has a higher-than-the-lawsuit cost most likely
Meanwhile, admitting you are missing shit you should not be, because the contractors played games (or even if it is just you):
1. You will be forced to settle the lawsuit at high cost
2. You will be less excoriated because you owned up to your failure, and because you look more like a bumbling company instead of belligerent and malicious.
3. You will probably be regulated more, but probably less than you did in the first scenario because of #2
The net expected value of the first strategy is much lower than the second.
They can't prevent themselves from losing or being excoriated, they can only try to make it have less cost.
Playing for time to restore faked documentation after the fact?
That feels like it would lead to jail time, which is an even worse option
It did not happen when they lied about the 737 Max problem even after a first crash, causing another hundred deaths, the CEO retired with a bonus of 60 million dollars, the current Boeing CEO was already part of the Board then, and will happen now because of some ... papirology?
"Boeing admits knowing of 737 Max problem" - https://www.bbc.com/news/business-48174797
"...Boeing has admitted that it knew about a problem with its 737 Max jets a year before the aircraft was involved in two fatal accidents, but took no action..."
Faking paperwork after the fact is unambiguous mens rea and “inadvertently made an alarm feature optional instead of standard” is not.
If you watched the Netflix documentary you will know the conduct was much worst than that. After the first accident, and before the second accident, the Boeing CEO went on CNBC saying the airplane was safe when they were already aware of the problem. To me it looks like cause for manslaughter charges at a minimum, but maybe that is why I am not a lawyer...
Got a source on them actually lying?
"In re The Boeing Co. Derivative Litig." - https://casetext.com/case/in-re-the-boeing-co-derivative-lit...
"Downfall: The Case Against Boeing" - https://www.imdb.com/title/tt11893274/
I am not aware of a single time that's ever happened.
which of the two scenarios captures the most value for lawyers involved?
Speaking as a lawyer, we’re not incentivized to advise a course of action whose primary purpose would be to line our own pockets. Doing so would cause us to lose trust in our clients and could possibly subject us to disciplinary action. We are expected to act in our clients’ best interest, not our own.
I’m obviously not Boeing’s attorney, but given the industry they’re in and the amount of scrutiny they’re under, they don’t need their lawyers to fabricate situations that drive up their billables. They’re quite busy enough already.
There must be someone somewhere that is incentivised to take course they are taking. If it's not the lawyers, who is it and what is their incentive?
Executives still trying to find out if there’s a convenient scapegoat they missed?
This sounds like a case of Schrödinger's documentation: it may or may not exist. I think Boeing is still looking for it and doesn't want to admit they don't have it until they're absolutely, positively certain they don't.
Time is running out, and the situation doesn't make them look good, but it seems they're not ready to abandon hope just yet.
I can imagine that for each plane could exist millions of documents, hidden away on different servers. Best-case scenario they could claim that a subcontractor was order to create it and lost it.
From experience I saw a company where people would have single copies of documents stored on hard drives, personal cloud drives, memory sticks, etc. They were always able to find what they needed, but there were a few close calls.
This is where being under so much pressure to produce the documentation, somebody ends up magic-ing it into existence. I wonder if they even have a chain of custody that would highlight that forms, etc, were retroactively created?
I am an investigative reporter, and I take the same approach toward public records. If it ever existed in digital form, there is almost always a stray copy somewhere — regardless of the agency’s records retention schedule. The challenge is figuring out where it is.
Just to point out that, despite the natural state of documentation being a superposition of "exists" and "doesn't exist", for that specific one there is a relative short (hundreds of pages) manual that says in no misleading terms exactly where the documents must be.
Just speaking from my experience here: I worked for a company that was comparable in role to Boeing. I was prohibited from retaining any documentation about my work that was not court ordered. Files, notes, emails, scribbles on a napkin all of it had to be destroyed.
Having a friend who has represented whistleblowers, etc, yes, this happens.
Sometimes for good reason (you are on a classified project, etc). Sometimes for bad reason (we don't want anyone to be able to prove anything).
The latter is something bad lawyers suggest. It basically never works to get the company off the hook. I'm not going to say "never never", but most of the time it leads to a situation where it's easy to convince a jury you are malicious and evil.
Even without a jury, i will simply put your peers who don't do that on the stand and make you look like you have a non-standard policy of hiding things.
Or i'll find the human who asked why. Because in any company of any size, people eventually ask each other why they are being asked to do this, and someone will say the wrong thing ("You know why we do this"). Most of them will not lie for the company because they don't buy it either.
etc
This sort of hiding strategy goes wrong in many many ways, and only goes "right" in a remarkably small set of circumstances.
Step 3 on your first scenario takes years to happen. More than enough for rich people to cash out their gains and move away.
Sure, that's fair.
I would say 'that is an unethical but 'reasonable' strategy for a rich CEO or whatever as an individual' (where reasonable here is 'probably would work' not 'morally sound').
It is an unreasonable one for the company as the company.
"That's correct, Senator. We don't know," Homendy replied. "And it's not for lack of trying."
They straight up said, we don't know.
And we don't know what we don't know, Senator.
Feel like a Cheney "known unknowns and unknown unknowns" meme would be fitting here.
I always felt bad for Rumsfeld (not Cheney) being the target of so much mockery for that comment. I disliked him style and decisions as Secretary of Defense, but that was one of the smartest and most realistic things he said. It's amazing to me how many people derided it as meaningless garbage when it self-evidently true and provided a worthwhile insight into the complexities of strategic decision-making.
He was lying about weapons of mass destruction.
The context is that he'd previously said we had to invade Iraq because of the imminent threat of the weapons of mass destruction.
So he's using a trivially true and non-interesting statement about "unknown unknowns" to evade questions about previous statements where he claimed to know something.
In other words, it was garbage response. He deserves mockery, scorn and a criminal trial.
I’m not sure that’s quite accurate, although it’s not that far off either. The full context is here:
https://web.archive.org/web/20160406235718/http://archive.de...
His “unknown unknowns” response was, to me, a basically fair if evasive response to whether there is any evidence of a direct link between Baghdad and terrorists as far as supplying WMDs. He essentially said “we don’t know what we don’t know.”
To me the more damning answer is the follow up later in the conference, where he strongly implied that he has evidence but can’t disclose it. That just misled the public and is far worse than the “unknown unknowns” comment in my view.
He'd already said for months previous that he knew Iraq had WMDs and were planning to supply terrorists.
Rumsfeld
McNamara
"known unknowns" and "unknown unknowns" are concepts that came out from the project management world. It's not a thing that Rumsfeld invented.
Homendy is NTSB, not Boeing. As far as I can tell, Boeing has not admitted top not knowing, nor have they provided the documentation.
I recall reading (probably a Twitter thread) a while back that showed entries in one Spirit Aero work logging system that showed the plug had been unbolted, and that there was no corresponding entry in the QA system that's required whenever a plug is removed. My gut feel its that someone was rule-bending, and they probably unbolted and slid the plug out of the way of the rivet holes they needed to fix, but didn't want to admit to "opening" the plug and having the QA audit records and self justified that they "didn't actually open it" when they unbolted and moved it (and failed to rebolt it when they put it back).
This thread seems pretty credible and seemingly consistent with what little official information is available
see comments by "throwawayboeingN704AL"
I can't find any such comments (nor the user) unfortunately. What did they say?
It's an anonymous comment posted here: https://leehamnews.com/2024/01/15/unplanned-removal-installa... . It was posted before any official information was released and seems to track with information that was subsequently released.
Thanks - that's exactly what I was remembering.
(Terminology as described further upthread at that link: CMES is an official record system for the plane. SAT is described as "Like Slack" and not an official record of the plane and its maintenance)
"finally we get to the damning entry which reads something along the lines of “coordinating with the doors team to determine if the door will have to be removed entirely, or just opened. If it is removed then a Removal will have to be written.” Note: a Removal is a type of record in CMES that requires formal sign off from QA that the airplane been restored to drawing requirements.
If you have been paying attention to this situation closely, you may be able to spot the critical error: regardless of whether the door is simply opened or removed entirely, the 4 retaining bolts that keep it from sliding off of the door stops have to be pulled out. A removal should be written in either case for QA to verify install, but as it turns out, someone (exactly who will be a fun question for investigators) decides that the door only needs to be opened, and no formal Removal is generated in CMES (the reason for which is unclear, and a major process failure). Therefore, in the official build records of the airplane, a pressure seal that cannot be accessed without opening the door (and thereby removing retaining bolts) is documented as being replaced, but the door is never officially opened and thus no QA inspection is required. This entire sequence is documented in the SAT, and the nonconformance records in CMES address the damaged rivets and pressure seal, but at no point is the verification job reopened, or is any record of removed retention bolts created, despite it this being a physical impossibility."
I wonder if the system recording the actions could have knowledge of these types of dependencies between actions, so it could refuse to record a log which is physically impossible (replaced a seal that — due to a plug not being opened — you cannot physically access).
It's not clear to me - was the seal replacement documented in CMES?
Is there not some dependency tree such that a cascade of sign-offs are required when one thing is changed? Clearly, changing the seal has dependencies which should automatically be flagged.
Search for that user in the comments here;
https://leehamnews.com/2024/01/15/unplanned-removal-installa...
She is the NTSB chair, not Boeing.
That does seem most likely. The putative insider blog post about the production incident essentially said the same thing: the contractors/managers were exploiting a loophole to avoid recording the work in such a way that it would trigger a required review step (or somesuch, I forget the details). Not a big leap from there to "didn't get recorded at all", and from there to "lots of other stuff isn't recorded either".
Boeing is in so, so much trouble here.
Their situation feels analogous to my memories of the initial offshoring boom in tech, when you could pitch plans that didn't factor time zone or QA into the budget and nobody cared.
Can they be subpoenaed? Then they will be forced to comply, or dump all the dirty laundry.
If this is true, this is pretty bad. An audit paper trail, at the very least, should be necessary.
Interesting to see how Boeing seems to be operating more and more like a US hedge fund (or any other criminal organization for that matter).