Pentagon fails audit for sixth year in a row

Incentives only work on low-skilled labor that is quantifiable. Eg. more money for flipping more burgers. It is not a settled science paying people a lot more will automagically solve problems. People crave Autonomy, Mastery & Purpose: https://www.youtube.com/watch?v=rbR2V1UeB_A

What is definitely going to happen is you will attract the cowboys who have convinced themselves and can convince you to pay top dollar. They will come up with lets do kubernetes + service-mesh + multi-cloud hybrid + blockchain + crypto + AI/MI on this COBOL ... They will take top dollar, make a bigger pile of mess, sell it as a success story and move on make bigger messes.

Don't they have caps and levels for federal employee salaries? I agree that raising the salary would decrease the issues. I think that they'd need exceptions to account for these increases though.

Cobol cowboys are a thing. They make substantially more than 250k. They’re competing with Wall St though, so hard to just pay more.

Fed govt migrations are ALWAYS shitshows. The army alone has like 20 individual email services. The pentagon is to org complexity what big tech is to technical complexity. It’s turtles all the way down.

This is literally one of the largest bureaucratic challenges on earth. There’s no simple fixes

That could work if the existing code is actually up-to-date, correct, and if you have knowledgeable users of the systems backed by this code. The part about 'code corruption' makes me thing this is not the case.

Worked at a big bank migrating some old code to new stack one time:

  - Original developers are gone.
  - Existing users don't understand the systems outside of their own narrow use-cases.
  - Code which was committed to repositories didn't match the actual running code, because people would go into production servers, manually edit the code there, in place, not document or commit the changes to repo. There were like 20 years of adhoc changes made in this way.

Anyway, it was a hilarious shit-show, but kinda fun at the same time (if you're able to ignore incompetent leadership pressure - a skill I mastered during this time).

Starting pay for cobol in finance was around 350k last time I glanced. Nobody wants to rewrite a working system which will likely be much more expensive than finding one person to maintain the project.

Part of the problem is, you tend to need cobol experts to actually have any hope of successfully transitioning to another technology, and if we had enough cobol experts would look at the cost of migrating and say, “who needs to, we have a bunch of cobol developers?”

Start paying 250k a year for cobol developers and the problem fixes itself really fast.

Do you think they aren't? I don't know any now, but the last one I did was making more than that 15 years ago.

Old crufty legacy systems in a big bureaucracy are many peoples idea of hell. Empirically speaking, good pay and job security aren't enough to have people knocking down your door for that kind of work if they have other decent options.

This is a problem not just in software. Today, everyone wants a senior <insert job role> for the price of a junior <insert job role>. So many problems would be solved orders of magnitude more soon if people were properly compensated. Instead, most things in the United States are running on economic fumes.

In the last part of the series from 2013 we have a chart that shows the Pentagon had spent over $10b modernizing their accounting systems (that was through 2013, no idea how much that number has grown.) I can't imagine anyone would miss a few million of that going to COBOL developers. The money scale for Pentagon projects is mind-boggling to me.

I can build you a team using today's (quality: average B2B startup) programmers and today's methodologies that will write software for 40 years.

It will just cost twice as much as the team that iterates quickly. It will cost more because I will spend more money on business analysis up front. I will spend more money writing automated tests. Most importantly, I will maintain my own toolchain and libraries or pay someone to do so. Every piece of software deployed will have someone accountable to my business to maintain it.

It turns out that this is expensive. It's no more expensive than maintaining the COBOL code, mind you, and would likely give you better results than the equivalent COBOL code. Most companies will take their chances with less expensive software because that's what their competitors do.

About a decade ago, I saw one of the major systems you talk about up close. While they spent that amount of money for mission-critical systems, they didn't do nearly as good of a job on the average software in the system. I'm not convinced that any business is ready to go back to the battle hardened days.

you can't say the code base isn't battle-hardened

Your statement is undermined by the fact that the system has failed its sixth consecutive audit. If it's not hardened against failing an audit, I'm not really sure what it could possibly be hardened against.

COBOL in 60s was written with a lot more focus on lasting in general

This is the same code that uses two-digit years and doesn't support lowercase letters?

Of course, devil's advocate would point out COBOL in 60s was written with a lot more focus on lasting in general. They didn't largely have the same general attitude of 'we iterate quickly with our script kiddies' as is prevalent today.

I worked on enterprise cobol for nearly 20 years and can generally attest that cobol code from the 60s is absolutely nothing at all what you’re describing.

It’s brittle and constantly breaks, requiring 24/7 on call support for people to deal with issues so it doesn’t hold up the next days work.

It was not written with any semblance of engineering prowess. Most cobol written in the 60s was written by business managers, not programmers.

The reason it stays running isn’t because it’s battle hardened. It’s because nobody in their right mind would ever pay to untangle the unfettered mess.

Having worked on a team responsible for maintaining COBOL code, if from the 80s and not the 60s, it may "last," but it's also a rats nest of dependencies that's a pain in the ass to test, and part of the problem of moving away from it is 40 years of undocumented business logic baked into it over the years.

Also throwing out there that COBOL was a language designed by the DOD specifically for managing it's bureaucracy.

It was a naval research project headed by Admiral Grace Hopper to program with words rather than formulas like earlier "high-level" "autocoders", enabling secretaries to transfer their business logic into computers. It actually intentionally has a lot of the same structure as a recipe, hoping that this would allow easier uptake by secretaries.

This just sounds like survivorship bias to me. I am sure there is a lot of junk COBOL that killed the companies that relied on it. Nobody cares today because the business is gone.

Something that's interesting to think about is that because the cost of changing COBOL is so high now, only the most important change requests actually get implemented. Compare that to a team that can develop and ship a change the same day; their efficiency leads to more requests that end up being bad ideas. The Pentagon has designed a system to ensure that only the most well-thought-out ideas make it into production, they just went about it in a very strange way.

Even when I started 12 years ago in Java/JSF/Oracle stack there seemed to be way more focus on durability and longevity. The system I started on was architected in a way that made it easy to modify and integrate with other system. It worked well. Eventually it's scope creeped, which would necessitate modernization. But it was very maintainable and lasted years.

Now the systems I work on are hot garbage because the tech is constantly needing upgraded versions, changes to the code are onerous because of the way the system was designed, we are constantly having data issues for various reasons, and people jump teams every 2 years so there's no continuity.

I'm convinced it's a leadership issue. Push for speed and you'll get it. It will cost you in other areas.

It'd be battle hardened to 1960s standards, payroll has changed a _lot_ since then. And if they can't accurately describe exactly how certain calculations are made (or have to do some of those manually outside of the program after it runs), that would be a huge problem in an audit.

This is why big companies use Oracle or SAP for their ERP, auditors are very familiar with it and the way it functions is well-known and understood.

If my experience with an unemployment system is a guide, the COBOL is fine. The scaffolding around it, different story.

I can imagine the opposite.

The hard part of accounting systems is doing the requirements analysis (including interviewing humans who give inaccurate or incomplete answers) and writing detailed functional specifications which account for every possible edge case. No LLM can do that.

Once you have the detailed specifications the coding is relatively easy. Some of that can be partially automated using AI CASE tools, but that only gives a marginal improvement to overall project productivity.

Are there any SA tools, that would understand COBOL?

Not all the RAM on earth would be enough to process that shit

Debt is also how economies and a lot of companies grow

The American dream is to "own" a house, not to own a mortgage.

Has the number ever decreased? As in, have they actually paid things off? Without actual numbers, U.S feels increasingly like a high-school bully who borrows money and then threatens with violence when that money is wanted back.

If you are quite wealthy there are lots of ways to borrow money and it’s all good!

If you have a negative net worth (more debt than assets) you're generally not considered a wealthy person.

Most explanations I've seen boil down to to "The US (or any other sovereign nation with control over their own currency) can print their way out of debt, it's literally impossible for the US to default."

Its impossible for the US to be forced to default.

Its quite possible for the US to make a political choice to default, and it has come very close to doing so.

The Russian rouble was not free floating prior to the crisis. They tried to defy the iron triangle and paid for it. This is not a problem for the USA, since other currencies are denominated in terms of the dollar.

The reason it would be impossible for the US to default is that the US is not significantly in debt. Most of the federal debt is held domestically, meaning by the US. You can't be in debt to yourself.

Sorry, but 'Fallacy of Composition' is just a theory. (Disclaimer: I happen to disagree with it.)

Ultimately the way these ideas about economic theory are approached is a lot more philosophical/religious-based (for the Atheists, please read that 'strong belief-based') and a lot less math based than some would be willing to admit. Honestly the math itself is usually statistical, which in turn is a lot more difficult to comprehend and analyze properly.

But imagine if they were. Wars and other destructive behaviors would be a lot more difficult

This only becomes a concern when total debt exceeds some multiple of GDP. The exact limit will vary based on many factors including interest rates, economic growth rate, demographics, and tax compliance. Once that limit is breached it tends to cause a fiscal and political crisis within a few years.

This is the kind of theory that, without a concrete model of how the limit varies (or even with one, if it involves much statsitical variability) is nonfalsifiable, either because the probability of a crisis in actual concrete conditions is unknown or the range of variability is too wide to make strong conclusions given the small-n problem with real-world conditions, but at the same time becomes very easy to rationalize almost any real-world conditions as fitting.

Barring difficult to foresee circumstances like an expensive medical procedure, a debilitating injury/illness, or a devastating lawsuit, most of the circumstances that lead to a household going bankrupt are directly attributable to mismanagement.

The numbers are just outside comprehension. We can comprehend a dollar, or a hundred dollars, in the hand. We may be able to imagine (or have seen) a room full of a couple tens of thousands of dollars. But to extrapolate one, to 32 trillion of the same...

If a 4 GHz processor core executes an instruction every clock cycle, it would still take two hours to iterate that instruction 32 trillion times.

If you lined up 32 trillion inches, it would be over 505 million miles.

If a dollar bill is 0.1 millimeters thick, and you stacked 32 trillion of the bills on top of each other, the stack would be almost 2 million miles high.

Etc

By the way, we went past $33 trillion in September, and reached $33.5 20 days later. Only a half trillion added in 20 days.

The attacks directly on the national debt are a political strawman, meant to distract from the real issues about where revenue comes from and how it is spent.

No, I assure you that those of us concerned about the national debt are actually concerned about it.

Equally wild is when "fiscal conservatives" complain about spending, it's only on social programs they disagree with.

Fraud and/or waste should be a subject of concern regardless of the program that experiences it.

When did Eisenhower warn about failed audits, or something like them?

Eisenhower might have warned about it but he apparently never asked Congress to appropriate funds for modernizing the Department of Defense accounting systems. Had he done so they wouldn't be such a mess today.

IRS exists to give those organizations money, so they’re more of a client and you’re more of a product.

You're going to make the excuse that your company's accounting mess is because you founded it before the IRS existed?

Good... Luck with that.

What are you proposing exactly? Hire more accountants? Shut down the military until they pass an audit?

The DoD is Congress' baby. They will move heaven and earth to make sure the defense industry gets paid.

The failed audits have no impact on their budget the following year.

And what has failing audits led to in the past?

It's a signicant issue in Congress

What are you basing that on?

It should be. But its not and, likely, never will be.

Heh, most of the DoD is. But DoD also has non Title10/PPBE-programmed funding (foreign military sales, contingency and humanitarian operations funds, MWR funds and incomes, support-to-other-agency (e.g. the US Navy NAVWAR organization provides SUBSTANTIAL IT and cyber expertise to non-DoD entities and other federal agencies such as the FBI and DHS).

The DoD is like the End Game Boss on legal and financial edge cases.

That's not how authority works. Whatever you would choose to implement, you will need to have enough buy in from those middle managers and fiefdoms otherwise you will end up with heel-dragging or, worse, malicious compliance. If you've been in any kind of organization that's tried to impose completely top-down edicts, you should have seen the kind of destruction of morale it leaves in its wake.

Your analysis of the problems is interesting. Do you work in DoD?

IME such internal politics is unavoidable in large organizations. Nobody likes it but it's the ocean in which everyone must swim. You can alter it to some degree, but there's no way for a structure that large to work without it. How do you manage million(s?) of people without middle management?

Also, it's not Meta. It's an organization that is responsible for life and death of hundreds of millions or billions, and national and global security, so control is essential. As a simple example, you can't lose track of the grenade launchers or nuclear weapons and have them end up in the wrong hands.

Hell, it would probably take more than four years just to understand the problems the various organizations are trying to solve. If you rush that, you "solution" is bound to miss important nuances that make things worse rather than better.

Sure, you could assemble a team with that experience to short-cut some of that. But you'll likely need a long time to just build the trust necessary to get the right information. You're an outsider and will likely be treated as such.

One data structure, one solution, for the entire problem seems like an unlikely path. Probably I would see how to break it into smaller, individual, more manageable audits, which seems to be what they did.

Yeah, it just seems like we've spent an inordinate amount of time on things that would either (1) save what amounts to peanuts in the context of the Army and DoD budgets or (2) not save any money at all.

I guess the primary purpose of the audit is to make all the numbers add up, and maybe that can inform cost-saving decisions down the road, like you said about warehouse storage space(?).

I'd rather see a number of "elephant in the room" programs ruthlessly cut, or processes changed, to start saving many billions of dollars now.

Stuff is cheap... there's been immense value in having tons of old cheap assets to throw into Ukraine. And even if it isn't dirt-cheap, there's value in maintaining industrial military production capacity.

Corporations got incredibly fragile supply chains by optimizing for JIT asset delivery, let's not repeat the mistake with national defense. I don't want the US to run out of vehicles, artillery shells, and spare uniforms 2 weeks into a real war.

Large fire extinguishers are high-pressure cylinders that require high-precision valves. There are fewer than 10 manufacturers in the world that can do it - half of them are in the US and the DoD will spend what it takes to keep it that way.

You wouldn't want to end up like the EU and not have a single manufacturer that can do it, or (since Brexit) not have any that are legally allowed to import them. It's a good thing they can be refurbished when used, and that the authorities look the other way when a shipment arrives from a 3rd party...

When you measure it like that... something really tingles and puts it into perspective. That also shows how much progress we make with how little money. To put it in units I know:

I work on a team that operates ~the only climate research station in Greenland, the fastest melting piece of ice on earth. The total cost of our project is ~10e-7 of the money they lost. You could build, operate, and employ ~10,000,000 scientists at 1,050,000 climate research stations with that kind of money.

Let's just say our forecasts and climate projections would be pretty spot on with that sort of funding.