return to table of content

Innovation heroes are a sign of a dysfunctional organization

ecshafer
38 replies
1d1h

My goto example of a dysfunctional and bureaucratic organization is an example I saw in a finance company I worked for.

I was a on team that we were doing a major project. We basically ran Kanban but had to run "sprints" so we chose 4 week sprints to get it out of the way and we put everything on the board we had to get done that month to stay on track. Our pipeline was setup in such a way that you were required to have a jira ticket to push a commit. We were really crushing our timeline, doing 2x the work we were expected, a really great team honestly. But we opened up bugs and additional features in the middle of our sprints to track our work as we did it.

What this amounted to was maybe us saying we have to do 40 tickets this month, and wed be closing like 80. Everyone should have been thrilled by this development! Well we had a sprint review where some "Senior Project Manager" That wasn't really affiliated with our project but was some manager higher up was mad that us opening up new tickets mid sprint and closing them was ruining the org level burn down charts and expected delivery. They wouldn't give up on this, and said it was our fault for not estimating better (which sure, but we were beating expectations!). So we did the reasonable thing and improved our estimates! No of course not, we doubled our number of tickets and filled in "placeholder" on half of them, and used them as needed then closed everything out at the end of the sprint, where we were congratulated by everyone for our phenomonal estimation.

constantcrying
11 replies
1d1h

No of course not, we doubled our number of tickets and filled in "placeholder" on half of them, and used them as needed then closed everything out at the end of the sprint, where we were congratulated by everyone for our phenomonal estimation.

Are you sure this is a negative example? You even have a work around which makes everyone happy.

This isn't what a dysfunctional organization looks like.

loloquwowndueo
8 replies
1d1h

It’s called gaming the metrics and it’s totally a sign of dysfunction at the org level.

The team that decided to do this to keep the metrics happy rocks, though :)

constantcrying
5 replies
1d

It’s called gaming the metrics and it’s totally a sign of dysfunction at the org level.

Have you seen actually dysfunctional organization?

Millions being burned over ridiculous mismanagement. Bureaucracy which takes up vast amounts of time. Tracking of completely meaningless metrics. An IT department which needs to be "worked around". Managers who's only interaction point is them asking you what they should write into their MS Project document. Constant shifting away of responsibilities. People making completely unqualified decisions over the heads of the actual people being concerned. Inter department conflicts.

Really I could go on, but a bit of massaging metrics (which are totally irrelevant anyway and their trackers can be safely fired) is not particularly bad.

sodapopcan
3 replies
1d

The person you are responded to said it's a sign of dysfunctional org, not that it by itself make an org dysfunctional. There is a huge difference.

constantcrying
2 replies
1d

The person you are responded to said it's a sign of dysfunctional org

He said it was his go to example of dysfunction.

sodapopcan
0 replies
23h38m

Ah GP did ya, fair enough. I assume a bit of hyperbole but sure. I would infer that there is a lot more going on than just some guy worried about if some guy is worried about individual teams' metrics getting messed up but I don't know the whole story.

loloquwowndueo
0 replies
22h23m

I did not. Read more carefully. It’s “totally” a sign but it’s neither the worst nor the only one.

Have you seen actually dysfunctional organization?

I have. I quit because of it. Imposed metrics and untold amounts of time gaming them to avoid losing one’s job was just one part of it. Most of the ones you mentioned in your first reply were also there.

pyrale
0 replies
21h22m

Have you seen actually dysfunctional organization?

I have yet to see an organization that is not dysfunctional.

Some dysfunctions I can live with, though. I agree with you, if gaming the metrics wastes a reasonable time (say, 5 minutes a week), it's definitely not the worst.

sodapopcan
0 replies
1d

Exactly. Org-level metrics are a sign distrust and micromanaging. It's useful for a team to look at their own metrics so they can understand what they are capable of delivering, but no one else should care. That they can reliably and consistently deliver a satisfactory amount of work should be the only "metric" they are judged on.

snapetom
0 replies
1d

As an engineer recently turned technical product manager, I think this is one of the values I've provided. I focus on feature deliverables and meeting timelines. How you get there is sausage making that doesn't need to be over-emphasized.

There are too many ways to game the metrics. I've done it, and I'm constantly impressed at seeing how innovative colleagues do it. If you focus so much on metrics during the development process, you're going to just add useless overhead and piss off developers. My boss comes from a sales background, and thankfully he gets it despite pressure from other senior leaders.

danaris
1 replies
1d

It's still dysfunction even if the group in this case was able to finagle a happy ending to this specific story.

Charts and processes are there to serve people, not the other way around. If a particular group is clearly doing a great job, but the way your metrics are set up claims they're causing problems, then the problem is in your metrics.

The organization is dysfunctional because a higher-level manager is able to enforce their particular view of how these metrics are supposed to work at the expense of actually productive groups underneath them.

Note that if the GP had actually done what they were told, they would simply have not worked on all those bugs that came in mid-sprint until the next one.

constantcrying
0 replies
1d

My point was just that the example is a very low bar for dysfunction.

riskable
8 replies
1d

I also work at a large financial institution and have had many similar situations. Fortunately, I'm the one in charge (team lead of sorts) and I have a pretty standard response to such "high level" nonsense:

"Your inability to adequately track my team's weekly or monthly performance is not my problem."

Every "project" has plans, deliverables, and due dates and those are the ultimate arbiters of a team's performance. Not the weekly/monthly statistics! If we open 10 or 10,000 tickets it makes no difference. It's entirely arbitrary and only carries meaning for the team in question (not upper management).

Like some high-level manager/PM is going to be able to make any difference whatsoever on some software development task by watching weekly Jira ticket statistics. Sounds like they're giving themselves busywork to justify their role. Because having fancy charts and statistics at meetings of managers makes you look like you know what you're doing (LOL).

gspencley
4 replies
23h53m

"Your inability to adequately track my team's weekly or monthly performance is not my problem."

I'm an engineer and I can certainly understand and empathize with where that sentiment comes from.

However, when people say things like this the first thought that goes through my head is "there is a culture problem."

That sentiment underscores an adversarial relationship between teams and leadership/management. That adversarial relationship should never exist (it does all too often, but it shouldn't).

Tracking timelines and deliverables is something that requires communication. That communication can be automated, but it's not up to leadership or management to implement that automation. They are not the engineers.

So if the process that is in place, which has worked for them despite inefficiencies (which they may not be aware of) is suddenly disrupted then no, it is not only "their problem." Some team went and did something differently than how things are usually done. The team [rightfully] recognizes that it is an improvement, but it was unsolicited and the communication / warning of the upcoming change was likely lacking.

Companies are called "companies" for a reason. They involve multiple people with varying skill-sets, responsibilities, understanding of how things do and should work, they have their own pressures and reporting structures (they need to hand things over to their management who expects a certain status quo as well) and most people have a default low tolerance for change.

This is no one person's fault. The company culture needs to facilitate iteration, improvement and innovation.

ryandrake
1 replies
23h23m

This is a great, experienced reply. All reasonable companies operate somewhere between the extremes: Tracking a project minute-by-minute is extreme. "Kicking off a project one day, and then returning once on the due date to make sure it is done" is an extreme, too. What's an appropriate amount of periodic tracking is a negotiation among all the stakeholders, many of whom need to plan their own work: leadership/execs, marketing, sales, r&d, and many more. Some companies arrived at "weekly tracking" as a happy median, others have longer or shorter periods.

So we've hopefully established that all projects need at least some kind of periodic tracking. So then, how do we do it? As a project manager myself, my preference is to have some kind of automated metric/metrics that I can pull myself and not have to bother engineers directly about. "Ticket count" might not be a good one, but the team should, together, find that good one. My management expects progress updates in some other format (often E-mail or silly slide decks), and I'd love to automate these, too. But, if we can't automate status updates, then I'm not going to just not get them--I'm going to do it the annoying way, by checking in and watching tickets and looking at git logs and manually "pinging" for information. Yuck!

GP's "Your inability to adequately track my team's weekly or monthly performance is not my problem." quip is only partially true. No, it's not strictly their job to write your project manager's reports for them, but it's also not appropriate to block them. As a team lead, part of their job is to partner with others, and sorry, but that includes the folks watching the clock and looking out for slippage and risks.

neutronicus
0 replies
21h50m

Yes, and one of the most pressing reasons to track a project is figuring out whether you need to move the due date.

I work at a product company, and "whether the project is complete by the due date" is not the end-all, be-all, for us. If the project is done, eventually, and good, our customers get value! We do, though, need to do things like market the new feature, produce training materials for it, etc, and these efforts need to be synchronized with the the feature actually getting done.

Assessing team performance is honestly not the main goal here! It's coordinating with everyone else in the company who can't act until the project is complete.

tkiolp4
0 replies
19h28m

You are right. Unfortunately, most companies out there do not operate as desired, hence the adversarial approach. I would even go further and say that if one lands in such companies (the not good ones, this is, the majority) with a non-adversarial approach, oh boy, you’re gonna be eaten (specially true for juniors)

fifticon
0 replies
23h41m

the management where I work, has the known strategic objective of eventually outsourcing all our development jobs to our indian facility. However, they prefer us to not leave our jobs before time, to ensure an orderly transition that wont lose us customers. In which universe are we not supposed to have an adversarial relationship? Our role in this is that our former ownerboss sold us to one of the big vertical software graveyards when he retired.

parkerhiggins
1 replies
13h59m

outcomes > outputs

It’s proven time and time again by DX programs (S.P.A.C.E. etc) that focus time is really the only KPI that produces outcomes.

gmfawcett
0 replies
4h38m

Reference, please?

roenxi
0 replies
16h47m

ecshafer's response seemed more productive. He was effectively providing an explicit maintenance budget in his estimates, which in practice seems to be exactly what Mr. Senior Project Manager wanted. No real cost to the team. Everyone a winner.

apozem
4 replies
1d1h

That is amazing. I worked with a (different?) finance company that did the same thing - every team was judged on their burndown charts.

From what I saw, it did no good. Teams simply did not use tickets to track work. If a story surfed sprints, its “ticket” was closed at the end of one sprint and a new one was opened for next. If a priority bug came in mid-sprint, we simply worked on it without a ticket.

cruffle_duffle
1 replies
1d

If a story surfed sprints, its “ticket” was closed at the end of one sprint and a new one was opened for next.

LOL, just joined a team that seems to be doing exactly what you describe. You start attaching performance incentives to how tickets are estimated and closed... people are gonna game the fuck out of them.

morkalork
0 replies
17h25m

I used to work somewhere that different bug severity levels were a KPI. One day after opening some prod/blocker/whatever level bug, I got a slack message from some useless agile coach arguing over it. After that, I created one kind of ticket in Jira. Fuck it. And when I talked to another senior engineer on another team it turns out they had the same experience and came to the same conclusion.

sailorganymede
0 replies
1d

Strange… The finance company I work at also does this. And it’s so strange. I feel like they value Jira tickets more than actual work being done. It’s even funnier cause we get asked to close work when we want to go into prod and told to create new tickets to put our work in to prod. which kind of is annoying.

ryandrake
0 replies
23h13m

Careful, though, because you may encounter someone who can sniff out these tricks.

I worked at a place where, like many companies, tracked the bug burndown as one of the measures of project risk. So, the closer you got to the release, the fewer bugs were expected to be open. Having many bugs open early in the project was fine and encouraged. Having few bugs open towards the end was seen as On-track, and having many bugs open towards the end was a red flag, where the program might want to either step in and help organize things or at least raise the risk with higher management. This is a pretty standard scenario you get at a lot of software companies.

Well, one of the teams realized that if they just kept their bug count always at zero, they'd never be bothered by those nasty, annoying project managers. So that's what they did. They were constantly in crisis, fixing problems and committing code, but their bug counts always looked great. A trivial bug tracker query revealed what they were doing: They'd just keep a side bug tracker in a doc rather than filing them in the real tracker, and then seconds before committing the code, they'd quick create a bug and use that as the required bug in the commit message.

So, the following week, these "ninja bugs" were added to the automated reports, and that behavior ended quickly.

siva7
2 replies
1d

Everyone should have been thrilled by this development! Well we had a sprint review where some "Senior Project Manager" That wasn't really affiliated with our project but was some manager higher up was mad that us opening up new tickets mid sprint and closing them was ruining the org level burn down charts and expected delivery.

Why should they be thrilled that you were running on Kanban but claiming to do otherwise and therefore ruining the burn down charts?

gopher_space
0 replies
23h42m

Am I working for The Tacoma Chart Co. or are we developing actual products here?

ecshafer
0 replies
1d

This was a team of 10 people, so costing the company $2M+ a year. How is halving the timeline on a very major, mission critical project not a good thing?

Some appendage management structure in the organization having charts and reports looking a certain way is secondary to how much value is being produced by a team of expensive engineers.

tokyolights2
1 replies
1d1h

I have seen this multiple times before. Management chastising good work because some externality of that work shows up on a graph that they have to talk about at some weekly meeting. It doesn't even have to make the graph look bad, just stick out.

aiisjustanif
0 replies
21h37m

Could have spent that time more wisely by making a better graph that accounts for this…

nonameiguess
1 replies
23h33m

I experienced something like this for many years across several of the big five defense contractors and smaller SBIR contractors working for the US DoD and IC. As someone who 20 years ago had a slightly better understanding of how computers and networks actually work than an average developer and was comfortable at the command line in an era when juniors increasingly couldn't leave their IDE without becoming hopelessly lost, as DevOps, SRE, and platform engineering started to become things, my career drifted in that direction.

The problem being on teams like that was always the same. You're responsible for developing software products of some sort, but they're software that runs, tests, or delivers other software or even orchestrates the operations of an entire environment shared by many different applications. This inevitably means a whole lot of your work is the classic incident response/post-mortem of operations, plus some level of customer support given to other development teams because internal platform teams never get a separate support organization. To this day, the government has no idea how to handle this in light of how DFARS (administrative law governing acquisition) works.

Every labor hour a contractor charges has to be tied to a specific line of accounting which is itself tied to some unit of planned work tracked in an issue tracker or project management system of some sort. Most of the time, this is an Epic in Jira. This is logical insofar as you consider the intention of acquisition as a category of appropriations bill. A budget proposal with line items tied to measurable product features is presented to and approved by Congress. You have to demonstrate you're spending the money they gave you on what they approved you to spend it on, because per the Constitution, that's how power of the purse works. The executive branch can't just do whatever it wants.

But it falls apart as contractor labor increasingly replaces federal civil servants. When your job is anywhere from half to all running and maintaining an operational system, that isn't really acquisitions any more. It's operations and maintenance, which is an entirely separate appropriation. Soldiers manning a defensive perimeter have no idea when they're going to be attacked or how much work they'll be putting in. Software operations is effectively just the less lethal civilian equivalent of that. The government arguably even recognizes this in many ways. Typically, running and maintaining the production system is a separate contract from the development contract and it uses separate work tracking systems, and if someone spends 8 hours watching a screen while producing no quantifiable work outputs, so be it. They charge 8 hours because that's what the contract says they're supposed to do.

But as we're increasingly expected to be modern software organizations with things like CI/CD, test and staging environments, and you inevitably need to run at least some of your own development infrastructure, well now what? You have a lot of people whose job is the same. Be on constant lookout and respond as needed, but now it's part of a development contract and they need to have quantifiable work outputs that be tied to a budgetary line item with an associated product feature.

So we convolute nonsense out of thin air like cloning the same monthly "Support" Epic in Jira, month after month. "As a developer, I want my tooling to work so I can do my job and deliver value to the government." Plug in some SWAG number vaguely guessed at based on how much of this kind of work you ended up doing last year. Every 90 days, play planning poker with it. LARP a product team even though that isn't what you are.

OldGuyInTheClub
0 replies
20h31m

Excellent and, to me, underappreciated points on HN. Now, expanding into the hardware world of A&D brings different levels of agony. If there's development risk, there's no good way to write a schedule that both accounts for it AND consistent with absolutely mandatory "Earned Value Management" where complex tasks have to be shredded into tiny 1-3 week chunks with metrics that the accounting team can track and report. If you need to fix a broken piece of equipment, if you need to buy a part, you're at the mercy of a procurement system that exists to survive audits and wants to know why the break or purchase wasn't in the plan. If the problem requires evaluating Plans A, B, and C, you have to pick one as the best, baseline, and execute it. Then if you need to consider the other two options that everyone knew up-front were possible, you get to explain why The One Chosen Plan didn't work. "You signed up for it."

The only way I've seen around it is a Risk/Opportunity Management system where risks/opportunities to whatever the baseline plan might be are enumerated, assessed, planned for, and maintained for when things go sideways or when there's a chance to save schedule and/or cost. But, that takes a VERY gifted _leader_ to run at the top level and those are rarer than hen's teeth. Ideally when a risk becomes a problem or an opportunity presents itself, the contingency plan is implemented. When it works, it is great and I've seen tough, ambiguous projects come in ahead of schedule and under budget. The common theme was one guy running all those projects and he's now retired. Usually the result is that the person reporting the problem gets stomped upon by a _manager_ who runs a cargo-cult process that s/he doesn't really understand.

Earned Value Management may be fine and dandy when development risk is done, the transition to manufacturing is complete, and the manufacturing process is relatively stable and the perturbations are small. Of course, that's when the contract SHOULD be Firm Fixed Price and not tracked by an army of accountants on both sides of the contract.

imtringued
0 replies
9h34m

we doubled our number of tickets and filled in "placeholder" on half of them, and used them as needed then closed everything out at the end of the sprint

Seems like someone has become aware of the power of liquidity and delayed decision making giving one the ability to respond to future information.

hinkley
0 replies
20h30m

It's funny that my anecdote also comes from a kanban team, but amongst mostly waterfall people and one Scrum nutjob.

When you are showing people up they throw the book at you. Yes you're getting a lot done but you're cutting corners so that's cheating! In an org that is perpetually behind schedule they rely on one team being a scapegoat for why all the other teams are behind schedule, and when your team has never been more than 10 days behind it stings.

One of my proudest moments on that project was when I figured out a judo move to meet (and exceed) the documentation requirements for our part of the project with less than 3 man hours per milestone after I got it working (and half of that was typesetting fixes by one of our tech writers). I got the impression that the person who called it out was really hoping it would cost us 20-60 hours per milestone. I think I spent about 30 hours total, including the initial set up time for the tools and templates. So their gambit only worked for a single milestone and then it was back to the races.

araes
0 replies
1h35m

I like how the response was "falsify the metric data" because an ape in a suit somewhere demanded a "correct" metric. And were rewarded for their falsification. There's an Aesop's Fables story in there somewhere. I wonder how many metrics people depend on, like jobs data, get the "change the data" to suit ape in suit's needs?

FireBeyond
0 replies
1d1h

I am dealing with this as a Senior Product Manager, explaining to execs, and defending my engineers when they started introducing metrics, and one of the keys was "Planned points delivered".

My team gets and triages all the escalated bugs.

It's not accurate, not fair, and demoralizing for that team to be "red" because the metrics look like: "Story Points Planned: 30. Planned points delivered: 5." when they delivered bug fixes on what amounted to 20-30 unplanned points.

"If they have to do that work, they need to get credit for it, planned or not. You can't have work that is required but counts for nothing in KPIs".

The issue of HOW MUCH unplanned work is being required is a separate and valid discussion, but not in the realm of "engineering cadence/velocity".

Aurornis
0 replies
23h37m

we had a sprint review where some "Senior Project Manager" That wasn't really affiliated with our project but was some manager higher up was mad that us opening up new tickets mid sprint and closing them was ruining the org level burn down charts and expected delivery.

This is giving me flashbacks of a company I worked for (for a very short period of time).

For some reason, they decided that planning accuracy was the most important metric for the software organization. This was the headline metric they talked about in executive meetings and it was primarily how the program managers' bonus structure was determined.

So everything in the company revolved around planning accuracy. Opening new tickets within a sprint was strongly discouraged. Doing work from the next sprint was heavily discouraged. Trying to take on big projects within a sprint was heavily discouraged, because if you couldn't 100% guarantee that it would be done by the end of the sprint it posed an existential threat to the Program Managers' charts, and therefore their bonuses.

Program Managers wouldn't come out and say any of this, of course. They knew the situation was at odds with delivering software quickly. However, if you deviated from the plan they would pull you into meeting after meeting for hours and hours to try to keep you in line.

If you opened a new ticket mid-sprint, you'd get pulled into meetings with Program Managers to justify it. They'd argue and debate and cajole you into deleting the ticket and rolling it into something halfway related. They'd CC your manager on 5 different e-mails and check in multiple times a day to make sure you'd fallen in line. It was hell.

Weirdest part to me was how many people around me seemed to enjoy that structure. They recognized the game and gladly played along, delivering a couple hours of work each day and then sitting in meetings to talk about it for the rest of the time.

It all came crashing down about 18 months in, when management brought in someone who actually understood software development and started actually looking into what people were doing. I was gone by then, but they went slash and burn on the remaining department. They cut it down to 1/5th of the size and started delivering, as far as I can tell, the same amount of work.

Jun8
14 replies
1d3h

This a million times! I’ve been in the position of the “innovation hero” but also was unfortunate enough to work on “innovation pipeline” implementations within a large company. These never work because:

1. (99% of) employees don’t care. They have their own job to do and working on other things is an eyebrow raise from their manager (see 3). And what’s the e benefit? Mostly it’s a pat on the back or some “points” in the company award system that you can use to buy shitty merch at the end of the year.

2. Executive leadership doesn’t care because for the most part they don’t trust their own tech team to innovate. Why take the risk when you can buy a startup which comes with a bona fide certificate of innovation.

3. But the real problem (as commonly identified) is middle mgmt, who not only don’t care but are generally hostile to nonstandard work. The reasons for this are complex, partly it’s the aging manager suffering from Peter Principle, partly it’s the fear of negative pushback from senior leadership.

PS: Steve is amazed, but 10 months for the sort of setup he describes which includes HW buy and setup, in a Big Corp is very fast.

cogman10
12 replies
1d2h

This all, imo, is simply a trust problem primarily from leadership. Leadership does not trust the grunts to do productive work. So in order to make sure productive work is done, they build elaborate systems of cases, reviews, meetings, planning, scheduling, fighting, readjusting when the schedules are invariably missed, and finger pointing. All almost always completely devoid of input from the grunts.

The middle management problem is they are right in the worst place possible. They are removed from the actual work being done so they don't know what it actually takes to do anything and they are blamed for things not accomplished. Further, they are rewarded for every little stupid thing done. It hyperintensities them to do lots of small safe initiatives and vehemently oppose anything with any sort of risk. All while being almost completely disconnected from what actually needs to be done.

This all leads to a culture meant to squash innovation. Middle management isn't rewarded for implementing a grunt's idea, they are rewarded for delivering CEO initiatives. Anything that takes time away from that is seen as waste.

foobarian
10 replies
1d1h

Leadership does not trust the grunts to do productive work

Not only that, but unfortunately they are usually right. Without oversight the in-house team has high likelihood of building NIH spaghetti, which causes more problems down the line. To avoid the negative outcome leadership needs to be technically competent and resourced, and that's the other side of the coin - usually they don't have the expertise so in a way they also do not trust themselves to lead the project to a positive outcome.

throwaway82931
6 replies
1d

they are usually right.

This message encapsulates why so many software jobs are terrible. Put your heart and soul into doing your best, earnestly combat NIH and pursue meaningful productivity, and _still_ the culture is such that at many companies, there will never be trust because the prevailing culture is that management is "usually right" that grunts can't be trusted.

I've worked for good bosses that aren't like this, but they're hard to find.

foobarian
3 replies
23h35m

Clean up enough messes left behind by grunts that were trusted by management and you will be cynical too :-)

throwaway82931
0 replies
14h37m

For what it's worth, I'm actually a senior IC who's cleaned up plenty of messes over decades and who enjoys migrating legacy systems. In my experience, it's the management-level architecture astronauts who don't understand loose coupling who do the most harm. The damage done by low level individuals can be contained if they're working on properly isolated subsystems (which of course requires competent management to set up).

And I'm cynical all right. I used to believe that I could be part of a "we're all in this together" team. But I've realized how rare that is after bad experiences at multiple companies where management sees an antagonistic relationship with engineering as inevitable — because they agree with you that "grunts can't be trusted".

oopsallmagic
0 replies
18h45m

Concur. It's an unfortunate race to the bottom which is just indicative of the dilution of ability in this godforsaken industry. Why can't a team of grunts manage a basic CRUD web app? That's what's so frustrating: most of these problems aren't even _hard_, and yet devs screw them up anyway.

Signed, a greybeard sick of cleaning up messes.

aiisjustanif
0 replies
21h29m

I still blame the management here for not building a team where quality and foresight matter enough to avoid large messes.

Trust from management isn’t all that is needed for good work to be accomplished, good managers of software development also should not shockingly know good software development.

I swear software and tech related roles are one of the few places where the managers don’t also know how to do the job of their reports.

Too
1 replies
12h44m

What's difficult is that usually half the team of grunts can be trusted to run their own race, the other half can be completely clueless and need explicit guidance, even if they are decent programmers otherwise.

If you were to allow everyone to innovate freely, some would spend several weeks learning the latest shiny fad and creating things of no or negative value, like using AI to generate commit messages or adding service mesh to your single container kubernetes cluster. The disconnect from business value and writing code for the sake of writing code can be astonishing. Conversely, if you let loose one smart guy in the team, it may seem unfair to the rest of the team.

A good leader is the one who balances the need and utilizes the best of both these sides.

throwaway82931
0 replies
7h16m

You’d think from this thread that the messes inevitably originate from ”grunts” wandering off the reservation. In my experience the source has often been “innovation” imposed from above, in the form of buzzword-driven development championed by semi-technical leadership enamored of this poisonous “grunts can’t be trusted” ideology.

danaris
2 replies
1d

Without oversight the in-house team has high likelihood of building NIH spaghetti

I think when you see this you need to start digging deeper and questioning why this is happening.

Is it because the "grunts" are genuinely bad at their jobs? If this is the case, then who hired them?

Or is it because they have been conditioned to believe that if they ask for permission to use an outside tool/library/etc, they will be told "no, we don't have the budget for that" or "that has to go through 12 layers of approval" or "great idea! we'll get it into a committee to talk about the best way to implement it and get back to you (in 6-12 months)"?

In other words: Are they building NIH spaghetti not because they lack oversight, but because they have too much, that hampers them from actually doing their damn jobs?

cruffle_duffle
0 replies
22h57m

Don’t forget incentive structures that award developers for “org wide impact”. Often the way to do that is to roll your own crap instead of use off the shelf stuff. This is how you wind up with developers creating their own key-value databases or billing systems… it sounds much better to write these from scratch.

Terr_
0 replies
21h7m

I'd like to add that NIH spaghetti sometimes comes from customer relationships rather than from exuberant developers.

The important customer wants just one tiny convenience-feature that fits their use case and it sorta makes sense... Which somehow keeps scope-creeping over time into a cancerous unplanned product which is expensive to support.

With the benefit of hindsight, you realize it's something the customer should have bought directly for themselves from a completely separate and more-qualified vendor. However they either didn't realize what they wanted or they were able to trick you into building an over-specialized product for their use case--for much cheaper than if they had hired a contractor to customize another better offering.

danaris
0 replies
1d1h

This all, imo, is simply a trust problem primarily from leadership. Leadership does not trust the grunts to do productive work.

Which, IME, stems from a deep-seated classism that sees "grunts" today as being essentially no different than assembly-line workers in the Industrial Revolution: you're just a pair of hands who not only doesn't know enough to make changes in the process, you shouldn't even think of it, because it's not your place.

In this worldview, it's managers (and up) who have the education, intelligence, and breeding to know best, and lowly workers just need to shut up and do what they're told.

...This attitude is also responsible for a lot of other really destructive problems in the modern world of work.

jmull
0 replies
1d1h

Mostly agree, but this

aging manager suffering from Peter Principle, partly it’s the fear of negative pushback from senior leadership

is not typically the reason, in my experience. I've seen a lack of full comprehension on the part of the team pushing the innovation as to the actual benefit and cost of the innovation to all the affected teams. And as a result, the lack of a plan to address those issues...

I've seen a lot of incomplete innovations, where the benefit is real and useful, but the plan leaves various concerns of different teams unaddressed -- no doubt due to the innovation team being unaware. Strangely, the innovation team often wants to push forward anyway, which is not good since the plan is basically unworkable with critical issues unresolved (I can tell they kind of think the issues aren't critical but only because they don't understand them -- leading with ignorance when the processes and products actually have to work at the end is always doomed to fail.)

The most common way plans are incomplete that I've seen is when they don't account for the schedule. The plan will take X time away from other work to implement, but the schedule for delivery of that other work isn't moved back X, nor are there other compensating measures. That's an unworkable plan, and any half-decent manager will push back on it.

(Schedule impact is usually a tough one... at least at larger places, in my experience, the high-level delivery schedule is negotiated at a high level, and it hard to change for political reasons. That means time for any innovation plans has to be included from the start. Yet slack in any schedule tends to get gobbled up at the team level or below, addressing their concerns -- who doesn't have tons of technical debt they are dying to resolve? There's a way to handle this, but it has to be planned for and done at a high level, and done correctly. The fruits of any innovation team that doesn't have this are gong to be minimal.)

tredigi
11 replies
1d3h

I agree to the overall tone, but there are also counter points.

One of them is the Google example. To get promoted beyond a certain level, you must have brought some new product over the finish line. Result? They have so many new things happening all the time, all of them suck, and then just move on to the next. Eg how many chat products do they need to invent before they settle on one and let it mature?

adverbly
4 replies
1d2h

To get promoted beyond a certain level, you must have brought some new product over the finish line

This always confused me. It looked from the outside like Google does so many things right on the innovation front, but after some early success they have had a rough streak.

I'd argue that one thing that Google is doing wrong is gatekeeping promotions based on (overly) well-defined criteria such as new products. Goodhart's Law applies to this situation: you're sure to see lots of new products if its highly rewarded - a lot more than you'd see naturally. As the author mentioned - this might still be desirable depending on the market conditions, but there is a lot more to this discussion, and it's not entirely clear that Google has it wrong.

I'd argue that they emphasize comparability of assessment results(e.g. being able to quantify someone's output like a percentage grade in a course) over the actual relevancy of the assessment criteria/work/kpi to the company's bottom line(e.g. does the course's test actually prepare students for the real world). This probably comes as a by-product of the organization's heavily academic-focused staff - so it might actually be the best culture choice for them given that context - but it might also lose to companies that can successfully put a bigger weighting on the right "intangibles".

immibis
3 replies
20h33m

You would think that. I would think that. But can you actually name more than a handful of things Google created that were good? Because I can't.

There's search, obviously. Gmail, if that wasn't an acquisition.... What else?

mdorazio
1 replies
20h3m

Ones I personally use because they are/were good: Maps. Android (+ Auto). Chrome. Docs/Sheets. Translate.

The issue is that most of these originated a long time ago and recent Google creations, like Bard/Gemini, tend to be mediocre copies of other things.

saulpw
0 replies
19h22m

Google Maps: acquired in 2004

Android: acquired in 2005

Google Sheets: acquired in 2006

Translate: launched in 2006

Chrome: launched in 2008

So only 2/5 of the ones you mentioned were started in-house, and all were pre-2010 (pre-Sundar) creations.

edit: Sundar joined Google in 2004 and apparently led Maps, Chrome, and eventually Android before becoming CEO. So "pre-Sundar" is technically incorrect.

adverbly
0 replies
1h19m

You're kidding right? Maps, android, camera, calendar, hangout, pay/wallet, docs, sheets, slides, voice to text stuff, YouTube, chrome/debugger, kubernetes, grpc, Gmail, music, ads... Pretty ubiquitous suite of products both consumer facing and business facing... I don't think you can question that Google has been successful at innovation. You can question if they could be doing it better but impossible to say they aren't having some success doing what they're doing...

laidoffamazon
2 replies
1d2h

The promotion thing seems severely overstated. At the higher levels for IC and management this is basically how all tech companies that build products are run. But you don't see this said about Microsoft or Amazon, even though they also have hundreds of new features and discrete new products per year.

My theory for why Google is different remains unpopular, however.

p_j_w
1 replies
1d1h

My theory for why Google is different remains unpopular, however.

May we hear it?

laidoffamazon
0 replies
1d

I think it's because Google has created an insular, navel-gazing culture that is excessively engineer driven rather than customer driven, to the extent of thinking people not at Google are just inferior.

nutrie
0 replies
1d2h

100 %. There's a sweet spot. I worked for both, start-ups and huge corporations. It's not one or the other, you want the right combination of maturity and fresh attitude. We often don't realize it can take years to steer back to find the balance again, but not oversteer, which is usually the default :) Just as with any other organism.

loceng
0 replies
1d1h

So that will result in an "innovation hero" improving on Google's model so the fundamentals don't create such waste, no?

dekhn
0 replies
1d1h

beyond just launching a product, the launch has to have some sort of "impact" (at least, this was true in the time period where I was trying to get promoted, roughly 2010-2013). Something that is not perceived as impactful by the promo committee is likely not going to count towards promotion. It's the job of the employee and manager to document the "impact".

If your manager is a director or higher, they can appeal the promo denial and an appeals committee can be manipulated into giving a promotion. That's what happened to me- promo saw no impact to my launch. Then my director went to appeals and basically said "promote him, he's doing good work".

Everything about Google messed up my expectations and planning around career. To work anywhere else (a startup, or a pharma) I had to unlearn all the bad habits of self-promotion and cookie-licking and impact-demonstration.

Of course many people joke the best way to get promoted at Google is to leave, get promoted elsewhere, and return to Google at a higher level (using all your newly learned negotiation skills).

setgree
9 replies
1d3h

I favor the general point here, but the leading anecdote doesn’t really fit with the lessons learned. A government organization generally does not need an innovation doctrine to avoid being outcompeted because they are a monopoly provider. They maintain that monopoly through force.

If you want to get a government agency to perform better, fix the incentives.

tredigi
2 replies
1d3h

Doesn't the leading anecdote give an example of a (dis)incentive that needs fixing? That it takes 10 months and lots of head-banging and then you get a $100 bonus, that would certainly disincentivise me to do any type of innovation.

setgree
0 replies
1d2h

It does, but the last lesson learned is:

All large organizations – both government and corporate—need an innovation doctrine or else risk being outpaced by competitors.

I don’t think the anecdote fits conclusion. The reasons to innovate are many, but being outcompeted is not that salient to, say, an IT person in some agency.

renewiltord
0 replies
1d3h

POSIWID implies that unchange is the desire of the government. People desire unchange and this is a tool they use to try to force it on everyone.

toomuchtodo
2 replies
1d3h

Incentives matter, certainly, but you must suss out what those incentives are and what they need to be to arrive at the desired outcome. Look no further than the US Digital Service and 18F (within GSA). You are not paid top dollar, but you are put in front of meaningful work and enabled to deliver (although that in itself is an incentive for the practitioners recruited). From the bottom of the impact report I cite: "We need you. Let’s help millions of people together." right above the Call to Action to apply.

The USDS is enabled to succeed in this mission through the support of the Executive Office of the President, the equivalent of corporate executive sponsorship. Culture comes from the top.

https://www.usds.gov/impact-report/2024/

https://www.usds.gov/impact-report/2024/by-the-numbers/

(full disclosure: went through a USDS interview cycle and was extended an offer, no other affiliation)

merely-unlikely
1 replies
1d2h

The USDS is enabled to succeed in this mission through the support of the Executive Office of the President, the equivalent of corporate executive sponsorship. Culture comes from the top.

That sentence is the essence of generic corpo-speak. Doesn't really mean anything without specifics.

toomuchtodo
0 replies
1d2h

Someone has the authority, budget, competency, and stamina to make change happen. I spend quite a bit of my time speaking with execs, my apologies.

kurthr
0 replies
1d2h

Well, except when they do.

Can you really imagine a military that didn't have and celebrate heroes?

If a large organization can't have those that sacrifice themselves and break rules to achieve the greater goal, then they're unlikely to succeed against a significant foe. At the same time communication within large organizations is challenging and leadership is unlikely to know what the challenges at the front line are. Fostering all innovation is as likely to lead to regularly scheduled mediocre "improvements" and "features" that nobody really wants in order to meet whatever metric is in place (to the detriment of what is not explicitly measured).

The argument of the article seems to be, just be so good and well directed by both senior and middle managers that exactly the "right" innovations occur within the process. That likely means those in the trenches aren't getting what they need.

The most effective rapid innovation method I've seen (though painful and challenging to implement) is having separate teams competing to several performance milestones (which allow more generic goals and targeted metrics). At the milestones they share their results and innovations, which competing teams can then use/combine to compete against them. Management needs real goals with known tradeoffs, 2-3x more people than a single team, and it's stressful hitting deadlines knowing that failure is an option. The failure mode is putting all the "best" people on one team which is supposed to win, though I've seen even that get broken by a team of underdog "heroes" who embarrassed the chosen team, and luckily senior management rewarded that.

It's similar to "red" vs "blue" pen-testing or wargaming, but you can have more than 2 teams and the goals can be aligned against the status quo (sometimes a tweaked current solution is the winner).

betenoire
0 replies
1d2h

I immediately thought of the government. I interned for the transportation department. I was one of these "innovation heros". It was a budget thing as explained to me. If I didn't have work to do, they told me to do homework, I wasn't even allowed to help people on things outside of my department, since they'd have to report on that in budgeting details and be accountable of it to the taxpayer.

MeetingsBrowser
0 replies
1d3h

A government organization generally does not need an innovation doctrine to avoid being outcompeted because they face no meaningful competition

Maybe true for some government organizations, but not true on the whole.

Governments spend billions (and sometimes trillions) on innovation to compete with each other. Collectively, government programs probably account for 99.99% of all "innovation" spending.

And they have the highest stakes when it comes to being "outcompeted".

shermantanktop
4 replies
1d3h

Reminds me of the Drucker quote about “culture eats strategy for breakfast.”

The problem is that large organizations naturally drift toward inertia and ossification. If a forward-looking leader wants the culture to change, they are faced with a conundrum:

- use a strategy like this (e.g. some top-down “innovation center” approved at the C level) which reinforces the rigidity and process-oriented thinking that needs to change, or

- create an insurgent skunkworks group that hopes to prove a different approach via undeniable results. This usually ends with back-alley knives getting unsheathed.

Pet_Ant
1 replies
1d3h

Innovation means potential for failure and waste. The larger the company the more it's investors want it for steady predictable returns. Even shrinking returns as long as they are steady and foreseeable. A large company becomes about control. Look at summer blockbusters. They know the demographics that will see it and have a pretty good (not absolute) idea of how much they can make.

This is all by design.

shermantanktop
0 replies
1d2h

This can be by design, sure. But I think it is also an inexorable tendency for people in large groups. Once your company is 10x Dunbar’s number, it is very hard for “we” to mean the entire company.

Angostura
1 replies
1d2h

I think there is possibly a third way.

I’m currently working with the NHS in a Quality Improvement role in a hospital. The team exists to give grass-roots folks the tools they need to make change happen when they spot something in the system which is less than ideal. We offer training, help with setting aims, running projects, finding stakeholders, analysing data etc. These aren’t huge system transformation projects, in fact some of them can be quite small - but as a way of working it feels quite effective.

withinboredom
0 replies
1d1h

in fact some of them can be quite small

The biggest changes always start with someone taking one small step towards their goal.

Some of the biggest software (used by billions of people) I've ever worked on started as a simple script made by a teenager.

nathan_compton
4 replies
1d3h

God yes please give us an "innovation doctrine."

This is certainly an interesting article to read about but I'm not sure his suggestions or analysis are that substantive. Complex institutions develop rules to simplify decision making and streamline information flows. They choke otherwise. "Develop an innovation doctrine" isn't really effective advice.

Mathnerd314
2 replies
1d3h

There are various articles on developing a culture of innovation, e.g. https://hbr.org/2019/01/the-hard-truth-about-innovative-cult.... Probably some books too. Even ChatGPT probably has decent advice. Management is not technically complex, it just requires putting in the work. But of course it is not easy, e.g. the first advice in the HBR article is to fire incompetent people, whereas the example here was government where firing incompetent people is notoriously hard.

nathan_compton
1 replies
1d1h

chatGPT probably has decent advice

If chatGPT has anything other than banal platitudes, I'll eat my hat.

Mathnerd314
0 replies
23h55m

It seems... practical? But not really rocking the boat. Just using Jira though seems like a big step for a government organization. I wouldn't eat my hat but I'd maybe nibble on it and play with the prompts more.https://chatgpt.com/share/13585d1b-a781-49fc-aa42-a7322b6f32...

OldGuyInTheClub
0 replies
1d1h

I agree. I am in aerospace. When times are good, there's money to bring in speakers like this every couple of months to talk to our management. We get the slides afterward and they are refreshingly free of content. All of them essentially repeat the cliche, "Think outside the box!"

We even got one of those Innovation Forums where people could propose ideas, get them upvoted, with the promise of tchotchkes at the end for success. Not a single one got funded. Every efficiency improvement, every streamlining suggestion affects someone's budget and headcount. When that person has to approve, nothing will happen. And, in this industry, a lot of rules are actually law. They have to be followed.

But, the consultants and professional keynote speakers seem to be making good money off of it.

chuckadams
3 replies
1d3h

"Innovation Doctrine" sounds like something you put next to your Mission Statement. How about fostering an internal discussion board where employees can pitch ideas and get hooked up with others who know how to implement them? If you need guard rails around the anarchy, then you can tie action items to a ticket tracking system that's readable by the whole company. I can file a bug in JIRA against any product my company makes, why not the company itself?

MeetingsBrowser
2 replies
1d3h

Because in large companies, any helpful suggestions get lost in the noise. Everyone has opinions on how things could be done better, but very few of them are good.

justin_oaks
0 replies
1d

I remember a company I worked at that an internal website for suggesting ideas. The problem is that the management never looked at it. And because of that, employees stopped posting to it. And that was that.

chuckadams
0 replies
1d2h

The discussion forum still has to be managed (I said "anarchy" facetiously) and at least minimal standards of professionalism would still have to apply. And if the board still devolves into a swamp of griping and bickering, then well, they can nuke it and at least say they tried.

Most ideas are indeed crap, and the good ones have to be picked out. But reducing the total number of ideas doesn't raise the percentage of good ones.

bearjaws
3 replies
1d3h

Why Innovation Heroes are a Sign of a Dysfunctional Organization

Because often you can solve 99% of companies problems with boring software.

I am reminded of this blog post from earlier this week:

Most organizations cannot ship the most basic applications imaginable with any consistency, and you're out here saying that the best way to remain competitive is to roll out experimental technology that is an order of magnitude more sophisticated than anything else your I.T department runs,

https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...

verisimilidude
0 replies
1d2h

"Innovation" in this context does not mean cutting-edge technology. It just means changing processes to deliver better results. The tech is often the easy part, and there's plenty of room for boring software.

The hard part is navigating the bureaucracy and building consensus toward a change. This management-craft is where the clever thinking and emergent solutions are found and deployed.

jjk166
0 replies
22h28m

A backhoe is orders of magnitude more complex than a shovel. A team struggling to dig a pit with shovels would probably benefit immensely from a backhoe. The idea that workers shouldn't get better tools until they can succeed without them is insane.

graemep
0 replies
1d3h

Updating a spreadsheet with data pulled from another system sounds more like "boring software" than "experimental technology".

MichaelRo
3 replies
1d2h

> Why is it that innovations require heroics to occur in our organization?

Why do we immediately assume that innovation = progress? Sure, the things that SURVIVE are useful, but that's just the tip of the iceberg. The vast majority of ideas are just like mutations in evolution more likely to be at best useless and probably damaging in various ways.

You see, social constructs are not as dumb as they appear to be to the armchair intellectual. "Why, we should embrace innovation and immediately adopt any idiocy that Mary from accounting is suggesting as our global company policy". I assure you that by natural law, if "random idea from random guy" were profitable on average, we'd have a system that would encourage such ideas. The sad fact is that they aren't and will never be.

Friction (named in the article as "Dysfunctional Organization") is an unfortunate but necessary process which ensures "survival of the fittest", even among "innovation". It's as simple as that.

ahstilde
1 replies
1d1h

Innovation is required for progress. The alternative to innovation is stagnation. The end result of stagnation is death.

dmvdoug
0 replies
16h33m

Pithy but, alas, meaningless.

First, define “progress.”

Second, innovation is a new idea, product, strategy, etc. Stagnation is lack of movement or lack of growth. There are alternatives to stagnation that are not innovation. Consider satisficing.

Third, the end result of stagnation may be other than death. It may be a steady state. Surviving, but not growing, is not death.

esafak
0 replies
22h45m

Evolution requires mutation before selection can take place.

wonderwonder
2 replies
1d1h

A long time ago I burned out and took a few years of leave from tech and instead took a document processing / generation job at a very large non tech travel / leisure focused company. Company used an archaic document generation system that had to have templates built by hand. then there was a significant amount of manual work moving data from microsoft excel to word. All Manual. My group consisted of myself, 4 other document processors, 2 leads and a director.

This team generated all of the sales documents for the company; thousands of templates.

Within weeks I wrote some VBA macros that automated 75% of the process and they promoted me to manager a couple months later.

What I found fascinating though was the group of existing document specialists were not suddenly capable of doing 2 - 3x the work. They just appeared to move slower. It was like pulling blood from a stone. They had no interest in learning new work which they were now free to do, they just wanted to clock in, turn their mind off for 8 hours, click the button and clock out.

40% of the staff at large companies are likely not needed and can be automated away.

Kind of made me feel sad.

drewcoo
1 replies
1d1h

What I found fascinating though was the group of existing document specialists were not suddenly capable of doing 2 - 3x the work. They just appeared to move slower. It was like pulling blood from a stone. They had no interest in learning new work which they were now free to do, they just wanted to clock in, turn their mind off for 8 hours, click the button and clock out.

The ability to barely function and slowly grind was what made them good at the job. Either they were hired for those "aptitudes" or they developed them on the job. what did you expect?

djaouen
0 replies
7h7m

I assure you, this slow grind of replying to non-relevant threads of yours will end after I post the following YouTube video: https://www.youtube.com/watch?v=MUfgAbFY4CA

solatic
2 replies
1d2h

Author clearly has no understanding of why startups move quickly. Startups have the same Legal, Procurement, Security, etc. needs and responsibilities as any large company or government agency - its just that these responsibilities are looked after by generalist founder-executives, not whole departments filled with specialists. Any "innovation department", if it ever wants to actually ship anything, still needs to get BigBureaucracy on board, which is where the vast majority of time gets eaten up to ship anything. Startups don't need buy-in and alignment from whole departments full of specialists, just the executive-founders with unrelated titles and experience who are still nonetheless nominally responsible for those areas.

what we just witnessed was leadership rewarding and perpetuating a dysfunctional and broken system.

Leadership's first responsibility is to keep the system happy and running smoothly. The first responsibility is not to shareholders, not to attempting to seize potentially higher profits, but to the organization itself. The organization may be "dysfunctional" and "broken" but this is completely irrelevant - the Fortune 500 is still generating massive profits and the public institution is still nominally discharging its duties. This is why the vast majority of Fortune 500 CEOs are "caretaker" CEOs and why deep cultural change of public institutions is so difficult.

Culture is fundamentally a question of who you hire, who you promote, and who you fire. Those decisions do not happen overnight in healthy organizations. That's why it's slow.

smugglerFlynn
0 replies
1d2h

Some orgs of Fortune 500 caliber have innovation units that help to process ideas and changes. To be efficient, these units must have buy in on senior leadership level and high level of exec sponsorship, ideally they would also have C-level representation in a form of dedicated “innovation officer” or similar role.

It is definitely not a new area, and author is on point with one key idea: organisations where heroes are praised as miracles generally don’t give a damn about continuous improvement, innovation units just never happen there. Praising individuals who swam against the flow for a year is pure virtue signalling in these orgs. In 2024 “running things smoothly” is almost a synonym for “continuous improvement”; building latter requires intent, not miracles.

pdonis
0 replies
23h8m

> Author clearly has no understanding of why startups move quickly.

He also appears to have no understanding of why government agencies and large corporations don't have to. Multiple times he says that such organizations need an innovation strategy or they will be out-competed. But government agencies and large corporations (which are mainly large because of the advantages of being large in an environment heavily regulated by government) don't have to worry about being out-competed, because they have insulated themselves from competition.

novagameco
2 replies
1d2h

I worked for a large company and started seeing opportunities for automation immediately. I proposed some solutions to my boss, and he told me that he agreed that these tasks could be automated, but that we have 10,000 other tasks that could be automated, and each one takes a few months to get the resources provisioned and also set aside developer (me) time to get it done, which could be spent on other projects.

What was interesting to me was the self-fulfilling prophecy of dysfunction: because there was so much manual process and red tape, the cost of fixing a particular problem is larger than than the benefits (i.e: time spent on the task exceeds time saved by automating it).

But because the tasks do not get automated, the amount of time required to fix things increases bit by bit due to the processes in place. The cost of fixing a task increases marginally every day, and so the cost/benefit ratio increases every day, becoming further justification NOT to fix things. At a certain point you have to look at the bigger picture and recognize that there is a much larger problem in your company than a few excel spreadsheets that could be better automated.

theideaofcoffee
0 replies
23h37m

I know that feeling all too well. And it's such a hard truth that devoting just a little time, letting some projects slip just a bit, to fixing those systemic problems could make lots of others go away, it's just next to impossible to get people to want to change. I've found because people don't want to, they want to keep doing what they're doing and are scared of anything new because that may mean either they'd have to retrain, or more pathologically, that their position is threatened.

It all comes down to the people. The right people can make all the difference in something like that, the wrong people make it miserable for the rest.

JohnMakin
0 replies
1d2h

So much feel this and have seen it many times. A complete unwillingness to spend a few hours to save hundreds of hours later, because too much is urgent. It's a little bit like a thrashing OS, too many competing resources so the result is everything slows to a crawl or breaks.

I've clawed my way out of situations like this but it does require some heroics in the beginning and probably longer hours than your role requires. I am the kind of person that will ask to slow a project down or delay it if it means we get a chance to do things correctly and in a maintainable way, but certain types in management will not really understand enough to completely buy in.

btbuildem
2 replies
1d2h

Not sure that you can fix a calcified bureaucracy with "doctrine". I sort of get the angle, I think -- if the org operates within a rigid rule framework, you need to speak their language to get anywhere -- so, doctrine is best, because everyone is used to being told what to do? I feel like that approach is counter to the spirit of innovation, that's akin to being forced to have fun; nothing truly innovative will come from it.

I think it's more of a lost cause, really. If you want to work on cool new stuff, don't work at a large org. I play the "innovation hero" role often enough, and the diversity of pushback we encounter is impressive. It ranges from thinly veiled hostility, the sdev equivalent of NIMBY, lies through omission, through nonviolent noncompliance, all the way to blithely unaware absurdity.

One amazing moment stands out to me - we were jumping through the usual hurdles as described in the article, trying to get a prototype to prod, and in one meeting the head of IT indignantly exclaimed "I am not here to solve problems!". To paraphrase a scene from the Big Short: he wasn't confessing, he was bragging. The top brass exist to stifle any and all deviation from the norm.

Another commenter in this thread makes a very good point: most of these innovation initiatives die stillborn, because the existing power structures exist, their MO is to maintain, and the C-suites would rather buy a successful startup than take any political risks internally.

entropicdrifter
0 replies
1d1h

"Innovation doctrine" is a contradictory phrase on its face, really. Going, "Oh we need a static, unchanging set of rules for innovation" doesn't exactly sound like a good way to attract and retain innovative thinkers lmao

adrianmonk
0 replies
1d1h

so, doctrine is best, because everyone is used to being told what to do?

I think they mean doctrine as in military doctrine (https://en.wikipedia.org/wiki/Military_doctrine), which is a somewhat different idea than, say, religious doctrine. Religious doctrine can easily get rigid and legalistic, and compliance with it can become an end unto itself. Military doctrine is something a military would use in order to make the organization effective at a certain kind of task or endeavor.

treflop
1 replies
1d2h

I honestly don't know where innovation exists that doesn't require "fighting the system" to some degree.

There are a lot of bad ideas out there and the "system" is there to weed out the bad ideas. However, you can have too much system where it will defeat even the good ideas too, so there's a sweet spot.

IMO it's comes down to how much leftover budget (time or money) the organization has. You don't need processes or procedures for innovation... you need people with enough leftover money and time who can "screw around" a little.

Also a place where everyone is not grumpy.

justin_oaks
0 replies
1d1h

There are a lot of bad ideas out there and the "system" is there to weed out the bad ideas. However, you can have too much system where it will defeat even the good ideas too, so there's a sweet spot.

Innovation means figuring out which are good and which are bad. It's hard to tell which is which from the beginning. Failure is inevitable. It's just whether there's a net benefit to the company instead of a net cost.

The problem is that the people and processes resist ANY failure, which necessarily throws out any innovation along with it.

you need people with enough leftover money and time who can "screw around" a little

And this is about right. There needs to be enough slack to not strangle ANY idea that can possibly fail. The author just went so far as to say that the "leftover" needed to be part of the budget.

Also a place where everyone is not grumpy.

Quite right. We need a place where some people are optimistic about new ideas instead of rejecting everything.

profsummergig
1 replies
1d1h

Their organizations hemorrhage the very people they need to help them compete against aggressive adversaries or competitors who have them in their sights.

Govt is a monopoly. So it can't be bothered.

(The org. in charge of preventing monopolies is a monopoly.)

throwawayqqq11
0 replies
1d1h

You are comparing companies and states. Sure, there might be similar machanics in both middle management at play but upper management is very different, at least in a democracy.

drewcoo
1 replies
1d1h

Heroics are responses to crises.

Why are there crises?

djaouen
0 replies
7h39m

I am only responding to this post because I cannot in my flagged thread: your bogus self-defense mechanism is not permissible. You will be loved, whether or not you intend for it! :D

debacle
1 replies
1d3h

One addendum to this that I have observed: In many organizations, no one is "in charge." If change needs to happen, there are 100 checkboxes, 100 reasons not to change, 100 people who are concerned about their career, resume, budget, department, relevance, etc.

For every "innovation hero" that wins an award, there are 5 who are "managed" into leaving, marginalized, or disenfranchised. The high nail gets the hammer. I have been into and out of the startup space for the last 20 years, and most of the times that I was an "innovation hero" it was because there was one person in a corner office who was using me as a proxy for a change they wanted to see happen.

The American system of organizational management is...basically glue.

justin_oaks
0 replies
1d1h

100 reasons not to change

I'm reminded of the article "Layers of Management == Layers of Veto" [0]. In the article the author explains that each layer of management is likely to veto each idea coming from below. Each idea that didn't come from above is "insubordination" and likely to be vetoed.

[0] https://slott56.github.io/2010_02_12-layers_of_management_la...

EncomLab
1 replies
1d2h

Relate to the four stages of employment: 1) This is the new person "X" - they are amazing and are going to solve everything! 2) "X" is pretty good, but maybe not as good as we thought. 3) "X" turned out to just be another average performer. 4) "X" is obviously terrible or they would have left for someplace that would treat them better than we do.

quesera
0 replies
1d2h

I've never seen this written down, but I've definitely seen it in action.

Experienced leaders try to avoid deluding themselves at step 1, and work to prevent things transitioning from step 2 to step 3.

But sometimes it's impossible (or I'm not experienced enough!). Some employees really do start off strong and then fade out, even if the incentive structures are "good" / "above market" / etc.

Honestly, I've seen the roots of this in myself too, in classes, jobs, relationships: initial enthusiasm wanes and at some point it's time to make a different decision. Personally I prefer to move on instead of stagnate, but some people are perfectly happy to ride the suboptimal until a decision is made for them!

zmmmmm
0 replies
14h50m

The saddest thing is that this "innovation" was mere automation of:

reentering data from one spreadsheet to another and annotating it with additional data from another system

Basically pretty much routine informatics - about the lowest bar you could possibly have.

Which is to say the whole article is really about simple baseline competence to implement process. If a place can't do that then they certainly won't be able to implement things that embody actual novel ideas. But framing it as "innovation" understates the severity of the problem.

web3-is-a-scam
0 replies
17h14m

If you’re an innovation hero and you don’t own part of the company, you are a sucker

theideaofcoffee
0 replies
23h39m

The idea of calcified culture and process isn't limited to large bureaucracies and government institutions. It's just as prevalent in orgs barely tipping over three figures' worth of people. In many ways, seeing it from both the large and the small, the smaller ones are often the most difficult to change because there are one or two people touching everything. Most often they have been there for a long time and such having the 'trust' of management so the bare minimum of 'new' ideas, and new being the state-of-the-art of the industry ten years ago, because they might have to adapt.

Playing the innovation hero in that kind of org is dangerous, and I have the battle scars, the burn-out and resentment to prove it. Others have pointed out that the point of the power structures is to self-perpetuate and any threat to that is swiftly dealt with.

It's just so sad to see, all because people fear change. I don't know how to change that other than replacing those lifers and swapping out management.

Now I know what to look for and am willing to bounce at the first sign of it.

t0bia_s
0 replies
1d

- All large organizations – both government and corporate—need an innovation doctrine or else risk being outpaced by competitors.

They not risk anything, because competition to government is forbidden by law. They have monopoly and will to use force if their system is threatened by alternative one. Innovation is not priority as it is in free market.

satisfice
0 replies
11h57m

Heroism is not a culture problem. Heroism is always a good thing. But process weenies like to knock it because it makes them seem grown up and sober.

The point the author wants to make can be made without denigrating the glorious matter of a person who is willing to overcome adversity to and ambiguity and chaos to do something wonderful.

When process mavens attack heroism, they are essentially saying that high performance and creative solutions should be an algorithm, a mechanism, something inhuman rather than fully human. They want the business to be an equation, and that should offend us, not comfort us.

By all means let’s do something to make innovation reasonably acceptable and not unreasonably expensive— but no organization should lightly enter into highly speculative work. Heroism is heroic partly because it’s also dangerous. There are good reasons to discourage it and good reasons to engage in it. Heroism is fraught— but it’s essential to any kind of technical work, since we are surrounded by obscure, “wicked” problems.

A good organization will reward heroism that doesn’t have a happy ending, as long as it was responsibly and competently pursued.

rqtwteye
0 replies
1d

My company’s mantra is “change and innovation are great as long as nothing changes”.

rednerrus
0 replies
1d

If heroes are getting things done, it seems like it's possible. Wouldn't it make sense to study how the heroes are operating and try to make that part of the culture?

praptak
0 replies
1d3h

I could name some companies where you don't need to be an "innovation hero" to feel like the "innovation hero" from the anecdote and "Please ask next quarter" seems to be the motto :)

oweiler
0 replies
5h11m

What's missing from the article: The innovation heroes will not only leave, they will most likely also burn out.

obelos
0 replies
21h52m

Counterpoint: most innovation sucks. Sure, innovation is necessary for improvement and adaptation to changing circumstances. Sure, sometimes innovative change is stifled to the point that organizations crumble under the stasis. But the larger and more stable the organization, the higher the bar should be between “I have a great idea that will make things better!” and its implementation. Most ideas of change do not take into account the complex, multiplicitous, overlapping, crossed-purpose systems that conspire to bring an organization to life. They account for only locally perceived changes, not weird, unpredictable knock-on effects that can arise from introducing an optimization. Adopting every ostensible innovation that comes along is inviting only chaos. Change should be hard. Not impossible, but hard.

nlawalker
0 replies
1d3h

This is exactly why the common advice is to automate your own tasks on your own time, with your own resources, don’t tell anyone, and enjoy working less.

lawlessone
0 replies
1d3h

There was some anime/manga I recall from when I was younger that had the phrase "heroes require bad things and/villians to exist."

This reminds of that.

lanstin
0 replies
1d1h

The thing is there's a lot of dysfunctional organizations, and being an innovation hero is an easier and more fun career path than fixing dysfunctional organizations, which seems (as someone that only half-heartedly tries to keep insane decisions from the top from ruining things) to not necessarily be possible; there are often hidden agendas and people more interested in their cash out than the viability of the organization long term.

kepair
0 replies
1d

In my company, awards and prizes always go to people that go "above and beyond" the processes the company sets in the first place.

hayley-patton
0 replies
1d3h

"Unhappy the land that has no heroes!" "No. Unhappy the land that needs heroes."

groestl
0 replies
1d2h

Large organizations don't improve by innovating. As the article mentions, they focus on sustaining the existing business model, anything that diverges from process potentially destroys the revenue source. Instead, they improve by acquisitions. Of innovative companies.

ghaff
0 replies
1d3h

The basic idea here seems pretty sound. I remember years ago critiques about some Microsoft "Heroes" campaign along a similar line. While organizations often have superstars or whatever you want to call them, if you require them to at least minimally function you're probably doing something wrong.

That shouldn't mean that you don't celebrate those superstars. They're not a bad thing certainly--which is probably where I differ with the post a bit. But understand that we shouldn't be depending on them all the time.

didgetmaster
0 replies
1d2h

The problem with this particular anecdote is that 'innovation' and 'government agency' are used in the same sentence.

Most people do not equate any government agency with an innovative environment. Bureaucracy is the watchword where the status quo is not only encouraged but fiercely protected.

Any new hire or politician who threatens to reform the process is treated as the enemy.

contingencies
0 replies
20h21m

I thought this was going to be about those people with colored glasses and ultra off-putting psychedelic sneakers (often matching) that turn up overcaffeinated with gelled hair as stage-dwelling experts, or multi-day workshopping nothing in particular with ramp-in/ramp-out platitudes and zero content. What do you call those? Innovation consultants?

charles_f
0 replies
1d2h

I fully agree with the sentiment, as I immediately get discouraged of doing anything as soon as it requires the approval from someone in another organization, which in large companies will often take weeks, if it happens at all.

There's an argument to be made for bureaucracy: once you reach a certain scale (of complexity and/or size), the law of large numbers makes that you're prone to more risks than when you're a startup. It's "fine" for a startup to not be aware of the latest development of privacy policies in Finland, and if you provision a couple resources in the cloud everything's fine because when looking at the bill, you can simply ask around "who owns that thing?". At enterprise size, you're much more prone to be audited by governments, to be attacked by hackers, to be the target of lawsuits, to have runaway costs that get hard to diagnose ; and the complexity of your systems (wet or hard) doesn't increase linearly. I don't think you can avoid having some processes to control that.

The problem is when the enterprise gets into a vicious cycle in which the only fix to any process issue is to slap on more process rather than rethink the existing ones. Such as illustrated in that post, where the solution to a lack of innovation is the creation of innovation heroes to incentivize the behaviour. I've been in three enterprise organizations, two of them have policies where you need to ask for permission, the other grants permission by default but reviews what's been done. That second policy is much better, as the level was quite high, people tend to respect the rule, and get caught if they don't. That allows for faster work.

But then there's also the matter of recruitment. As a startup you can handpick people, create a team that's functional and with an edge. In an enterprise, recruiting is done at an industrial scale, where your inbound fresh flesh exposes you to the simple truth that, by definition, half of humanity is below the mean.

anders30
0 replies
1d

Last year, an Innovation Hero championed a completely new build system based on their homegrown version of asdf, essentially.

I was the one who had to stay late multiple Fridays making sure our Formal Builds would still work. I've been working to roll back the least well executed portions of this innovation for the past year because it doubled the amount of time required to release our software in the last stages of the pipeline (which was already five days, another rant, I work at VeryBigCo on a mixed HW/SW product).

I believe in this story the entrepreneur referenced worked to clear all similar hurdles, but it's hard to feel bad for folks who view themselves as Innovation Heroes when in many cases they're applying solutions in search of a problem. It's also very not fun to be "that person" aka "The Adult in the Room".

Yes, today the "Innovation" is part of our processes, but in stripped down form and it cost us two missed formal builds with the corresponding loss of credibility in our customer's eyes... in addition to my own sour grapes and missed dinners with my kid. It also cost me a personal friendship with the hero (my fault, but it's really hard for me to get past the fact this individual put their ego over my time with my family).

Agree it's a sign of a dysfunctional organization but it goes both ways. As an Innovation Hero, you should be aghast at how inefficient your org is, but as a part of that org, you should stop and ask, "How did we get this way and am I sure my solution truly solves all aspects of this situation?"

agentultra
0 replies
1d1h

“Engineering is doing well with $1 what another person could figure out with $2,” I heard from somewhere.

This often involves, in larger groups, communicating with folks and taking the time to understand the requirements.

Although I despise “5 whys,” almost as much as scrum; it does often boil down to dysfunctional organizations.

It saddens me when a highly productive development team gets pulled down by non-productive bureaucratic middle-management and politics. There often is a class of worker that doesn’t want to do meaningful work and sees the value generated by other people as an opportunity to take a ride to the top.

There should be a few check points in an organization where one needs to ask for permission but going too far down that road puts a ceiling on how productive any one group can be.

Good article; definitely need to remove barriers.

adverbly
0 replies
1d2h

To be fair, creating a process for innovation is hard and expensive - especially if you want it to apply across the board at a large organization.

I have been on multiple "incubator" development teams in the past and although we did move faster, my times at smaller startups still felt significantly more productive.

You want something that is the right balance of bespoke and standardized/transferable. Its not easy, but there is so much room for improvement at these bigger organizations that there is huge value to getting it right.

adolph
0 replies
1d3h

Lacking a method to implement continuous improvement is the dysfunction.

“When [W. Edwards Deming] came to spread the gospel of continuous improvement in 1950, he was preaching to the choir. Toyota already believed in it. [Deming] simply gave them a process to better understand how to progress from failure. The idea of ever-improving coupled with what they learned from Deming—especially the Theory of Knowledge and shorter feedback loops via the PDSA loop, as well as the Theory of Variation and the accompanying statistical process control—let them succeed in their failure.”

From “Deming’s Journey” by John Willis. Solid recommended new read.

https://www.amazon.com/Demings-Journey-Profound-Knowledge-In...

Scubabear68
0 replies
1d2h

I am currently working with a very large company that has this problem. They are highly risk adverse and are happy to pay $100 up front to avoid a $1 accidental loss.

Of course, all the processes don’t really protect them, they just get slowed down by a factor of 100 or more.

The successful people take on as many simultaneous projects as they can, because any one project will move at a snails pace.

I have told them tales of working on some internal document processing and classification systems in a fintech a few years ago where we would release to prod multiple times a day to test out different algorithms and approaches.

They flat out think I am lying.

Mathnerd314
0 replies
1d3h

what we just witnessed was leadership rewarding and perpetuating a dysfunctional and broken system.

It's not clear that it is dysfunctional. If innovation is not a particularly high priority, but risk reduction is, then the system is working as designed - all of the checks are necessary. Compare to the "innovative" Boeing-type company which streamlines production by removing all safety checks.

Log_out_
0 replies
5h58m

The problem i see is that process designers do not have to justify the costs they inflict on economic +generating activity. Its purely designed by informal power struggles of "of course my department has to be looped in".. And thats the dilemma at the heart of all bureaucracy.. At the core its powerstruggles eating time and effectivity.

And to fight it, one would need sort of a controlling office that reports to and limits the damage done by powerstruggles. Never seen that in a company.

JohnMakin
0 replies
1d2h

This is true, but I am unsure of the fix prescribed here.

For me, I had experience with a project like this - a large, extremely bureaucratic company tasked my team with a nearly impossible task, I think in an effort to lay off the team - which eventually happened. I fought with tooth and nails and was already doing fairly well in my role, so they gave me a shot and assigned the task, which I won't belabor the details of, but it was basically to implement a fundamental service that every application in the company would use to authenticate to the backend system (plus a lot of other reliability/availability guarantees).

The problem was, with their architecture being a central hub cluster of servers that provided core services to hundreds of edge servers around the world, it would have been fine to just throw this service into the central server and call it a day. However, the eggheads in executive leadership felt this was not acceptable, so the requirement was to make a new "central" cluster to connect to not only all the edge clusters in the company, but all the backend office/admin stuff as well.

Problem was - the people who wired up the networking and everything else with the central server had been gone for 10+ years. Barely anyone even knew how it worked, and unfortunately as I took this on, the network team got laid off, replaced by people who also had no clue how it worked. As we connected more and more services to this new hub, a bunch of skeletons emerged - the funniest being a server cluster in a region and account that no one had access to, or knew what it did, other than when it crashed other servers had issues. No one had accessed it for years. That was fun hacking into.

There were tons of hurdles like this at every step, basically being that this touched so many teams in so many different areas of the infrastructure, the organizational hurdles trying to even get the information you need to do the job required a ridiculous amount of heroics. I hated it. Basically, you need to create urgency any way you can - whether this is by breaking things, horse trading, political maneuvers, begging, intimidation - your livelihood is on the line. Other teams sensed what a difficult ask this was so early going was extremely difficult getting cooperation from the 30+ teams this touched.

Anyway at the end of the day I was able to do it by finally escalating urgency to the executive level and they made it a priority one quarter and it quickly got finished. If they hadn't, it would have been very difficult. They had the issue of having too many layers of management below them and had no idea, all they heard is "why is this project you said would take 2 months taking almost 2 years" and a bunch of manager speak trying to explain why.

I would never, ever want to work on any project like that again. 2 years to complete, should have taken 2 months if working as an IC, 2 weeks with a full competent team and good management. There were some good things that came out of it though - processes got improved, collaboration improved, and we were able to use it as a chance to refactor the IAC in a way you could deploy these hub servers again in a much easier way that didn't require the ridiculous amount of detective work to figure out the first time.

Oh yea, forgot to mention the actual application took about 2 days to configure. All the rest of it was what took 2 years.

BenFranklin100
0 replies
1d2h

Lack of innovation is of course a problem in industry, but it’s a particular problem in the public sector where many employees are attracted by the “can’t get fired” rather than the “let’s make things better” nature of the job.