Innovation heroes are a sign of a dysfunctional organization

My goto example of a dysfunctional and bureaucratic organization is an example I saw in a finance company I worked for.

I was a on team that we were doing a major project. We basically ran Kanban but had to run "sprints" so we chose 4 week sprints to get it out of the way and we put everything on the board we had to get done that month to stay on track. Our pipeline was setup in such a way that you were required to have a jira ticket to push a commit. We were really crushing our timeline, doing 2x the work we were expected, a really great team honestly. But we opened up bugs and additional features in the middle of our sprints to track our work as we did it.

What this amounted to was maybe us saying we have to do 40 tickets this month, and wed be closing like 80. Everyone should have been thrilled by this development! Well we had a sprint review where some "Senior Project Manager" That wasn't really affiliated with our project but was some manager higher up was mad that us opening up new tickets mid sprint and closing them was ruining the org level burn down charts and expected delivery. They wouldn't give up on this, and said it was our fault for not estimating better (which sure, but we were beating expectations!). So we did the reasonable thing and improved our estimates! No of course not, we doubled our number of tickets and filled in "placeholder" on half of them, and used them as needed then closed everything out at the end of the sprint, where we were congratulated by everyone for our phenomonal estimation.

This a million times! I’ve been in the position of the “innovation hero” but also was unfortunate enough to work on “innovation pipeline” implementations within a large company. These never work because:

1. (99% of) employees don’t care. They have their own job to do and working on other things is an eyebrow raise from their manager (see 3). And what’s the e benefit? Mostly it’s a pat on the back or some “points” in the company award system that you can use to buy shitty merch at the end of the year.

2. Executive leadership doesn’t care because for the most part they don’t trust their own tech team to innovate. Why take the risk when you can buy a startup which comes with a bona fide certificate of innovation.

3. But the real problem (as commonly identified) is middle mgmt, who not only don’t care but are generally hostile to nonstandard work. The reasons for this are complex, partly it’s the aging manager suffering from Peter Principle, partly it’s the fear of negative pushback from senior leadership.

PS: Steve is amazed, but 10 months for the sort of setup he describes which includes HW buy and setup, in a Big Corp is very fast.

I agree to the overall tone, but there are also counter points.

One of them is the Google example. To get promoted beyond a certain level, you must have brought some new product over the finish line. Result? They have so many new things happening all the time, all of them suck, and then just move on to the next. Eg how many chat products do they need to invent before they settle on one and let it mature?

I favor the general point here, but the leading anecdote doesn’t really fit with the lessons learned. A government organization generally does not need an innovation doctrine to avoid being outcompeted because they are a monopoly provider. They maintain that monopoly through force.

If you want to get a government agency to perform better, fix the incentives.

Reminds me of the Drucker quote about “culture eats strategy for breakfast.”

The problem is that large organizations naturally drift toward inertia and ossification. If a forward-looking leader wants the culture to change, they are faced with a conundrum:

- use a strategy like this (e.g. some top-down “innovation center” approved at the C level) which reinforces the rigidity and process-oriented thinking that needs to change, or

- create an insurgent skunkworks group that hopes to prove a different approach via undeniable results. This usually ends with back-alley knives getting unsheathed.

God yes please give us an "innovation doctrine."

This is certainly an interesting article to read about but I'm not sure his suggestions or analysis are that substantive. Complex institutions develop rules to simplify decision making and streamline information flows. They choke otherwise. "Develop an innovation doctrine" isn't really effective advice.

"Innovation Doctrine" sounds like something you put next to your Mission Statement. How about fostering an internal discussion board where employees can pitch ideas and get hooked up with others who know how to implement them? If you need guard rails around the anarchy, then you can tie action items to a ticket tracking system that's readable by the whole company. I can file a bug in JIRA against any product my company makes, why not the company itself?

Why Innovation Heroes are a Sign of a Dysfunctional Organization

Because often you can solve 99% of companies problems with boring software.

I am reminded of this blog post from earlier this week:

Most organizations cannot ship the most basic applications imaginable with any consistency, and you're out here saying that the best way to remain competitive is to roll out experimental technology that is an order of magnitude more sophisticated than anything else your I.T department runs,

https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...

> Why is it that innovations require heroics to occur in our organization?

Why do we immediately assume that innovation = progress? Sure, the things that SURVIVE are useful, but that's just the tip of the iceberg. The vast majority of ideas are just like mutations in evolution more likely to be at best useless and probably damaging in various ways.

You see, social constructs are not as dumb as they appear to be to the armchair intellectual. "Why, we should embrace innovation and immediately adopt any idiocy that Mary from accounting is suggesting as our global company policy". I assure you that by natural law, if "random idea from random guy" were profitable on average, we'd have a system that would encourage such ideas. The sad fact is that they aren't and will never be.

Friction (named in the article as "Dysfunctional Organization") is an unfortunate but necessary process which ensures "survival of the fittest", even among "innovation". It's as simple as that.

A long time ago I burned out and took a few years of leave from tech and instead took a document processing / generation job at a very large non tech travel / leisure focused company. Company used an archaic document generation system that had to have templates built by hand. then there was a significant amount of manual work moving data from microsoft excel to word. All Manual. My group consisted of myself, 4 other document processors, 2 leads and a director.

This team generated all of the sales documents for the company; thousands of templates.

Within weeks I wrote some VBA macros that automated 75% of the process and they promoted me to manager a couple months later.

What I found fascinating though was the group of existing document specialists were not suddenly capable of doing 2 - 3x the work. They just appeared to move slower. It was like pulling blood from a stone. They had no interest in learning new work which they were now free to do, they just wanted to clock in, turn their mind off for 8 hours, click the button and clock out.

40% of the staff at large companies are likely not needed and can be automated away.

Kind of made me feel sad.

Author clearly has no understanding of why startups move quickly. Startups have the same Legal, Procurement, Security, etc. needs and responsibilities as any large company or government agency - its just that these responsibilities are looked after by generalist founder-executives, not whole departments filled with specialists. Any "innovation department", if it ever wants to actually ship anything, still needs to get BigBureaucracy on board, which is where the vast majority of time gets eaten up to ship anything. Startups don't need buy-in and alignment from whole departments full of specialists, just the executive-founders with unrelated titles and experience who are still nonetheless nominally responsible for those areas.

what we just witnessed was leadership rewarding and perpetuating a dysfunctional and broken system.

Leadership's first responsibility is to keep the system happy and running smoothly. The first responsibility is not to shareholders, not to attempting to seize potentially higher profits, but to the organization itself. The organization may be "dysfunctional" and "broken" but this is completely irrelevant - the Fortune 500 is still generating massive profits and the public institution is still nominally discharging its duties. This is why the vast majority of Fortune 500 CEOs are "caretaker" CEOs and why deep cultural change of public institutions is so difficult.

Culture is fundamentally a question of who you hire, who you promote, and who you fire. Those decisions do not happen overnight in healthy organizations. That's why it's slow.

I worked for a large company and started seeing opportunities for automation immediately. I proposed some solutions to my boss, and he told me that he agreed that these tasks could be automated, but that we have 10,000 other tasks that could be automated, and each one takes a few months to get the resources provisioned and also set aside developer (me) time to get it done, which could be spent on other projects.

What was interesting to me was the self-fulfilling prophecy of dysfunction: because there was so much manual process and red tape, the cost of fixing a particular problem is larger than than the benefits (i.e: time spent on the task exceeds time saved by automating it).

But because the tasks do not get automated, the amount of time required to fix things increases bit by bit due to the processes in place. The cost of fixing a task increases marginally every day, and so the cost/benefit ratio increases every day, becoming further justification NOT to fix things. At a certain point you have to look at the bigger picture and recognize that there is a much larger problem in your company than a few excel spreadsheets that could be better automated.

Not sure that you can fix a calcified bureaucracy with "doctrine". I sort of get the angle, I think -- if the org operates within a rigid rule framework, you need to speak their language to get anywhere -- so, doctrine is best, because everyone is used to being told what to do? I feel like that approach is counter to the spirit of innovation, that's akin to being forced to have fun; nothing truly innovative will come from it.

I think it's more of a lost cause, really. If you want to work on cool new stuff, don't work at a large org. I play the "innovation hero" role often enough, and the diversity of pushback we encounter is impressive. It ranges from thinly veiled hostility, the sdev equivalent of NIMBY, lies through omission, through nonviolent noncompliance, all the way to blithely unaware absurdity.

One amazing moment stands out to me - we were jumping through the usual hurdles as described in the article, trying to get a prototype to prod, and in one meeting the head of IT indignantly exclaimed "I am not here to solve problems!". To paraphrase a scene from the Big Short: he wasn't confessing, he was bragging. The top brass exist to stifle any and all deviation from the norm.

Another commenter in this thread makes a very good point: most of these innovation initiatives die stillborn, because the existing power structures exist, their MO is to maintain, and the C-suites would rather buy a successful startup than take any political risks internally.

I honestly don't know where innovation exists that doesn't require "fighting the system" to some degree.

There are a lot of bad ideas out there and the "system" is there to weed out the bad ideas. However, you can have too much system where it will defeat even the good ideas too, so there's a sweet spot.

IMO it's comes down to how much leftover budget (time or money) the organization has. You don't need processes or procedures for innovation... you need people with enough leftover money and time who can "screw around" a little.

Also a place where everyone is not grumpy.

Their organizations hemorrhage the very people they need to help them compete against aggressive adversaries or competitors who have them in their sights.

Govt is a monopoly. So it can't be bothered.

(The org. in charge of preventing monopolies is a monopoly.)

Heroics are responses to crises.

Why are there crises?

One addendum to this that I have observed: In many organizations, no one is "in charge." If change needs to happen, there are 100 checkboxes, 100 reasons not to change, 100 people who are concerned about their career, resume, budget, department, relevance, etc.

For every "innovation hero" that wins an award, there are 5 who are "managed" into leaving, marginalized, or disenfranchised. The high nail gets the hammer. I have been into and out of the startup space for the last 20 years, and most of the times that I was an "innovation hero" it was because there was one person in a corner office who was using me as a proxy for a change they wanted to see happen.

The American system of organizational management is...basically glue.

Relate to the four stages of employment: 1) This is the new person "X" - they are amazing and are going to solve everything! 2) "X" is pretty good, but maybe not as good as we thought. 3) "X" turned out to just be another average performer. 4) "X" is obviously terrible or they would have left for someplace that would treat them better than we do.

The saddest thing is that this "innovation" was mere automation of:

reentering data from one spreadsheet to another and annotating it with additional data from another system

Basically pretty much routine informatics - about the lowest bar you could possibly have.

Which is to say the whole article is really about simple baseline competence to implement process. If a place can't do that then they certainly won't be able to implement things that embody actual novel ideas. But framing it as "innovation" understates the severity of the problem.

If you’re an innovation hero and you don’t own part of the company, you are a sucker

The idea of calcified culture and process isn't limited to large bureaucracies and government institutions. It's just as prevalent in orgs barely tipping over three figures' worth of people. In many ways, seeing it from both the large and the small, the smaller ones are often the most difficult to change because there are one or two people touching everything. Most often they have been there for a long time and such having the 'trust' of management so the bare minimum of 'new' ideas, and new being the state-of-the-art of the industry ten years ago, because they might have to adapt.

Playing the innovation hero in that kind of org is dangerous, and I have the battle scars, the burn-out and resentment to prove it. Others have pointed out that the point of the power structures is to self-perpetuate and any threat to that is swiftly dealt with.

It's just so sad to see, all because people fear change. I don't know how to change that other than replacing those lifers and swapping out management.

Now I know what to look for and am willing to bounce at the first sign of it.

- All large organizations – both government and corporate—need an innovation doctrine or else risk being outpaced by competitors.

They not risk anything, because competition to government is forbidden by law. They have monopoly and will to use force if their system is threatened by alternative one. Innovation is not priority as it is in free market.

Heroism is not a culture problem. Heroism is always a good thing. But process weenies like to knock it because it makes them seem grown up and sober.

The point the author wants to make can be made without denigrating the glorious matter of a person who is willing to overcome adversity to and ambiguity and chaos to do something wonderful.

When process mavens attack heroism, they are essentially saying that high performance and creative solutions should be an algorithm, a mechanism, something inhuman rather than fully human. They want the business to be an equation, and that should offend us, not comfort us.

By all means let’s do something to make innovation reasonably acceptable and not unreasonably expensive— but no organization should lightly enter into highly speculative work. Heroism is heroic partly because it’s also dangerous. There are good reasons to discourage it and good reasons to engage in it. Heroism is fraught— but it’s essential to any kind of technical work, since we are surrounded by obscure, “wicked” problems.

A good organization will reward heroism that doesn’t have a happy ending, as long as it was responsibly and competently pursued.

My company’s mantra is “change and innovation are great as long as nothing changes”.

If heroes are getting things done, it seems like it's possible. Wouldn't it make sense to study how the heroes are operating and try to make that part of the culture?

I could name some companies where you don't need to be an "innovation hero" to feel like the "innovation hero" from the anecdote and "Please ask next quarter" seems to be the motto :)

What's missing from the article: The innovation heroes will not only leave, they will most likely also burn out.

Counterpoint: most innovation sucks. Sure, innovation is necessary for improvement and adaptation to changing circumstances. Sure, sometimes innovative change is stifled to the point that organizations crumble under the stasis. But the larger and more stable the organization, the higher the bar should be between “I have a great idea that will make things better!” and its implementation. Most ideas of change do not take into account the complex, multiplicitous, overlapping, crossed-purpose systems that conspire to bring an organization to life. They account for only locally perceived changes, not weird, unpredictable knock-on effects that can arise from introducing an optimization. Adopting every ostensible innovation that comes along is inviting only chaos. Change should be hard. Not impossible, but hard.

This is exactly why the common advice is to automate your own tasks on your own time, with your own resources, don’t tell anyone, and enjoy working less.

There was some anime/manga I recall from when I was younger that had the phrase "heroes require bad things and/villians to exist."

This reminds of that.

The thing is there's a lot of dysfunctional organizations, and being an innovation hero is an easier and more fun career path than fixing dysfunctional organizations, which seems (as someone that only half-heartedly tries to keep insane decisions from the top from ruining things) to not necessarily be possible; there are often hidden agendas and people more interested in their cash out than the viability of the organization long term.

In my company, awards and prizes always go to people that go "above and beyond" the processes the company sets in the first place.

"Unhappy the land that has no heroes!" "No. Unhappy the land that needs heroes."

Large organizations don't improve by innovating. As the article mentions, they focus on sustaining the existing business model, anything that diverges from process potentially destroys the revenue source. Instead, they improve by acquisitions. Of innovative companies.

The basic idea here seems pretty sound. I remember years ago critiques about some Microsoft "Heroes" campaign along a similar line. While organizations often have superstars or whatever you want to call them, if you require them to at least minimally function you're probably doing something wrong.

That shouldn't mean that you don't celebrate those superstars. They're not a bad thing certainly--which is probably where I differ with the post a bit. But understand that we shouldn't be depending on them all the time.

The problem with this particular anecdote is that 'innovation' and 'government agency' are used in the same sentence.

Most people do not equate any government agency with an innovative environment. Bureaucracy is the watchword where the status quo is not only encouraged but fiercely protected.

Any new hire or politician who threatens to reform the process is treated as the enemy.

I thought this was going to be about those people with colored glasses and ultra off-putting psychedelic sneakers (often matching) that turn up overcaffeinated with gelled hair as stage-dwelling experts, or multi-day workshopping nothing in particular with ramp-in/ramp-out platitudes and zero content. What do you call those? Innovation consultants?

I fully agree with the sentiment, as I immediately get discouraged of doing anything as soon as it requires the approval from someone in another organization, which in large companies will often take weeks, if it happens at all.

There's an argument to be made for bureaucracy: once you reach a certain scale (of complexity and/or size), the law of large numbers makes that you're prone to more risks than when you're a startup. It's "fine" for a startup to not be aware of the latest development of privacy policies in Finland, and if you provision a couple resources in the cloud everything's fine because when looking at the bill, you can simply ask around "who owns that thing?". At enterprise size, you're much more prone to be audited by governments, to be attacked by hackers, to be the target of lawsuits, to have runaway costs that get hard to diagnose ; and the complexity of your systems (wet or hard) doesn't increase linearly. I don't think you can avoid having some processes to control that.

The problem is when the enterprise gets into a vicious cycle in which the only fix to any process issue is to slap on more process rather than rethink the existing ones. Such as illustrated in that post, where the solution to a lack of innovation is the creation of innovation heroes to incentivize the behaviour. I've been in three enterprise organizations, two of them have policies where you need to ask for permission, the other grants permission by default but reviews what's been done. That second policy is much better, as the level was quite high, people tend to respect the rule, and get caught if they don't. That allows for faster work.

But then there's also the matter of recruitment. As a startup you can handpick people, create a team that's functional and with an edge. In an enterprise, recruiting is done at an industrial scale, where your inbound fresh flesh exposes you to the simple truth that, by definition, half of humanity is below the mean.

Last year, an Innovation Hero championed a completely new build system based on their homegrown version of asdf, essentially.

I was the one who had to stay late multiple Fridays making sure our Formal Builds would still work. I've been working to roll back the least well executed portions of this innovation for the past year because it doubled the amount of time required to release our software in the last stages of the pipeline (which was already five days, another rant, I work at VeryBigCo on a mixed HW/SW product).

I believe in this story the entrepreneur referenced worked to clear all similar hurdles, but it's hard to feel bad for folks who view themselves as Innovation Heroes when in many cases they're applying solutions in search of a problem. It's also very not fun to be "that person" aka "The Adult in the Room".

Yes, today the "Innovation" is part of our processes, but in stripped down form and it cost us two missed formal builds with the corresponding loss of credibility in our customer's eyes... in addition to my own sour grapes and missed dinners with my kid. It also cost me a personal friendship with the hero (my fault, but it's really hard for me to get past the fact this individual put their ego over my time with my family).

Agree it's a sign of a dysfunctional organization but it goes both ways. As an Innovation Hero, you should be aghast at how inefficient your org is, but as a part of that org, you should stop and ask, "How did we get this way and am I sure my solution truly solves all aspects of this situation?"

“Engineering is doing well with $1 what another person could figure out with $2,” I heard from somewhere.

This often involves, in larger groups, communicating with folks and taking the time to understand the requirements.

Although I despise “5 whys,” almost as much as scrum; it does often boil down to dysfunctional organizations.

It saddens me when a highly productive development team gets pulled down by non-productive bureaucratic middle-management and politics. There often is a class of worker that doesn’t want to do meaningful work and sees the value generated by other people as an opportunity to take a ride to the top.

There should be a few check points in an organization where one needs to ask for permission but going too far down that road puts a ceiling on how productive any one group can be.

Good article; definitely need to remove barriers.

To be fair, creating a process for innovation is hard and expensive - especially if you want it to apply across the board at a large organization.

I have been on multiple "incubator" development teams in the past and although we did move faster, my times at smaller startups still felt significantly more productive.

You want something that is the right balance of bespoke and standardized/transferable. Its not easy, but there is so much room for improvement at these bigger organizations that there is huge value to getting it right.

Lacking a method to implement continuous improvement is the dysfunction.

“When [W. Edwards Deming] came to spread the gospel of continuous improvement in 1950, he was preaching to the choir. Toyota already believed in it. [Deming] simply gave them a process to better understand how to progress from failure. The idea of ever-improving coupled with what they learned from Deming—especially the Theory of Knowledge and shorter feedback loops via the PDSA loop, as well as the Theory of Variation and the accompanying statistical process control—let them succeed in their failure.”

From “Deming’s Journey” by John Willis. Solid recommended new read.

https://www.amazon.com/Demings-Journey-Profound-Knowledge-In...

I am currently working with a very large company that has this problem. They are highly risk adverse and are happy to pay $100 up front to avoid a $1 accidental loss.

Of course, all the processes don’t really protect them, they just get slowed down by a factor of 100 or more.

The successful people take on as many simultaneous projects as they can, because any one project will move at a snails pace.

I have told them tales of working on some internal document processing and classification systems in a fintech a few years ago where we would release to prod multiple times a day to test out different algorithms and approaches.

They flat out think I am lying.

what we just witnessed was leadership rewarding and perpetuating a dysfunctional and broken system.

It's not clear that it is dysfunctional. If innovation is not a particularly high priority, but risk reduction is, then the system is working as designed - all of the checks are necessary. Compare to the "innovative" Boeing-type company which streamlines production by removing all safety checks.

The problem i see is that process designers do not have to justify the costs they inflict on economic +generating activity. Its purely designed by informal power struggles of "of course my department has to be looped in".. And thats the dilemma at the heart of all bureaucracy.. At the core its powerstruggles eating time and effectivity.

And to fight it, one would need sort of a controlling office that reports to and limits the damage done by powerstruggles. Never seen that in a company.

This is true, but I am unsure of the fix prescribed here.

For me, I had experience with a project like this - a large, extremely bureaucratic company tasked my team with a nearly impossible task, I think in an effort to lay off the team - which eventually happened. I fought with tooth and nails and was already doing fairly well in my role, so they gave me a shot and assigned the task, which I won't belabor the details of, but it was basically to implement a fundamental service that every application in the company would use to authenticate to the backend system (plus a lot of other reliability/availability guarantees).

The problem was, with their architecture being a central hub cluster of servers that provided core services to hundreds of edge servers around the world, it would have been fine to just throw this service into the central server and call it a day. However, the eggheads in executive leadership felt this was not acceptable, so the requirement was to make a new "central" cluster to connect to not only all the edge clusters in the company, but all the backend office/admin stuff as well.

Problem was - the people who wired up the networking and everything else with the central server had been gone for 10+ years. Barely anyone even knew how it worked, and unfortunately as I took this on, the network team got laid off, replaced by people who also had no clue how it worked. As we connected more and more services to this new hub, a bunch of skeletons emerged - the funniest being a server cluster in a region and account that no one had access to, or knew what it did, other than when it crashed other servers had issues. No one had accessed it for years. That was fun hacking into.

There were tons of hurdles like this at every step, basically being that this touched so many teams in so many different areas of the infrastructure, the organizational hurdles trying to even get the information you need to do the job required a ridiculous amount of heroics. I hated it. Basically, you need to create urgency any way you can - whether this is by breaking things, horse trading, political maneuvers, begging, intimidation - your livelihood is on the line. Other teams sensed what a difficult ask this was so early going was extremely difficult getting cooperation from the 30+ teams this touched.

Anyway at the end of the day I was able to do it by finally escalating urgency to the executive level and they made it a priority one quarter and it quickly got finished. If they hadn't, it would have been very difficult. They had the issue of having too many layers of management below them and had no idea, all they heard is "why is this project you said would take 2 months taking almost 2 years" and a bunch of manager speak trying to explain why.

I would never, ever want to work on any project like that again. 2 years to complete, should have taken 2 months if working as an IC, 2 weeks with a full competent team and good management. There were some good things that came out of it though - processes got improved, collaboration improved, and we were able to use it as a chance to refactor the IAC in a way you could deploy these hub servers again in a much easier way that didn't require the ridiculous amount of detective work to figure out the first time.

Oh yea, forgot to mention the actual application took about 2 days to configure. All the rest of it was what took 2 years.

Lack of innovation is of course a problem in industry, but it’s a particular problem in the public sector where many employees are attracted by the “can’t get fired” rather than the “let’s make things better” nature of the job.