Suits ignored IT's warnings, so the tech team went for the neck

Being able to clearly articulate risk, and the business consequences of technical consequences, is a key skill for IT folk. It’s possible that management at this bank were profoundly thick in the head, but it’s also possible the tech team didn’t explain it very well.

There's this myth that IT people can't communicate. Maybe it's because I'm in IT that I think it's quite the opposite: we communicate very well and precisely.

The real issue is that middle management has a ton of politics that muddle up everything. I can say to my team that I messed something up and that we have to fix it; but I've seen people higher up who will bend over backwards just to not have to roll back a decision (and sometimes not even a very important decision).

So maybe someone had already sold to the boss that the equipment was good for another 10 years.

I’ve seen too much bad communication between software teams and nontechnical management firsthand to agree it’s a myth–I do think technical folks often communicate well and precisely amongst themselves, but often there’s just a fundamental lack of understanding as to what management cares about and needs to know about.

The answer to “why are we planning to spend so much time on this non customer-facing project” isn’t to just dive in and start talking about Kubernetes node pools and pod affinity rules, it’s to talk about how we’ll be able to cut cloud spend by 20% right off the bat while improving site speed, which is important because churn is up 25% YoY and a majority of recently churned clients have complained to support about slowness before cancelling.

So frequently I’ve seen former, then the engineers go off in a huff that management won’t support their project, and management goes off wondering why engineering is screwing around while customers are churning.

I’ve never heard an example of kubernetes making a deployment faster.

Was that a made up example? If not, is there a successful case study somewhere you can cite?

Not GP, but one of my previous jobs used to have a deploy method of "ping one of the ops team on Slack to release a certain commit SHA to prod/staging/etc", which might easily take an hour or more if they were busy and/or you asked around lunchtime. Even if they were available immediately it would still take 20 minutes.

After we moved to k8s, the release model went to "every dev can use kubectl to deploy" and it took a minute or ten to release something to prod. Needless to say, we all loved it much more than the previous setup. Even more so after someone whipped up a script to automate all the kubectl invocations.

One of my former colleagues wrote an article about it here:https://wetransfer.com/engineering/migrating-frontend-to-kub...

"every dev can use kubectl to deploy" may be less horrific than the previous situation but it's still pretty bad practice tbh. So is "a script to automate all the kubectl invocations".

There's existing tooling for that : Skaffold, ArgoCD ...

Of course, but those were not yet available in 2017. Skaffold v0.1.0 was released in february 2018, v1.0 (first version number which we would've seriously considered) not until november 2019. ArgoCD v1.0 was released in may 2019.

When we were first releasing this Helm charts were considered pretty unproven and AWS didn't have a managed k8s offering yet. It was very early days.

It was a contrived example given k8s is a ripe source of jargon and engineering rabbitholes/boondoggles, though I guess I could envision a real situation where some bad choices about which workloads run on which nodes wind up impacting application performance, which then results in over-provisioning of resources as a bandaid fix.

Nothing at a company is "non-user-facing", otherwise you wouldn't do it. If you're communicating the importance of some project or plan to people outside of your domain, it's important to keep the focus on how this actually benefits the company.

However that's not the problem described in TFA. It was clear what the user-facing implications of not upgrading the line were, and management didn't care. Sometimes people aren't rational.

What I was getting at was the idea of time spent on things that aren’t directly and immediately able to be, say, put into a sales demo, but instead impact the customer a but more tangentially, which is also when communicating clearly about business impact is particularly important. I would also argue that companies do tons of genuinely non customer facing stuff with varying levels of justification, for better or worse.

Regardless, who knows whether this was the case in the story being discussed, but it’s not unthinkable that this genre of communication breakdown was involved to some extent.

My point is that just about any justification for any business activity ends up being customer-related at some level.

Example:

Why do we want to spend money on Jetbrains licenses for developers? Because we want them to be happy, comfortable, and productive. And why do we want them to be happy, comfortable, and productive? Because they build stuff that customers want.

Why are techs the only one's with agency? Communication. Manage upwards. Dance, monkey, dance.

Shouldn't management communicate? Shouldn't managementmanage?

In all my career, I can think of maybe a handful of non-techs whose communication was measurably better than grunts, that could manage better than chimps throwing darts.

I will concede that most tech's, such as myself, fail miserably on other social niceties crucial for royal politics (and the money chase). Such as deception, stealing valour, skulduggery, sucking up, ambiquitiy, enthusiasm, and fashion.

but often there’s just a fundamental lack of understanding as to what management cares about and needs to know about.

At some point, you've got to think about whose role it is to manage who.

If your management needs to be spoon-fed the decisions they must sign off, and reacts poorly to signal that is not perfectly worded or inadequate because the relevant information was not transmited down, what use are they?

it's not that uncommon to work at a company where the engineers have no idea that churn is up 25%, much less that customers are complaining about slowness.

companies get so fixated on having their engineers do "engineering work" that they don't communicate what's actually going on at the business level.

People in upper management only speak in dollars. If you can't speak in dollars you literally can't speak their language.

They shouldn't have asked management for money, they should have presented management with two alternatives: pay $x now to upgrade network infrastructure, or pay $y later in rush fees, IT overtime, and customer loss due to degraded UX. Then you let them make the decision.

It's absolutely possible the business was in a temporary cash crunch or needed the money to get a more important revenue-generating initiative past the finish line, and eating the higher costs later might make more sense to the business.

Assuming the story is accurate, it was not money that ultimately swayed them, but directly experiencing the problem.

That sounds like the job of management to do..

A response to that would be that translating different work scenarios into $ is management.

In my opinion IT people are often great at communicating clearly and with what in other parts of business would be called vision. They think ahead and see connections between processes and systems that non-technical management (even the process owners) miss. They can see the dot on the wall and deliver a program brining you there.

However, and that’s the big gripe, each and every one IT proposal I see costs money. It not just costs money, everything costs money, but it raises the long term average cost per unit. Even when we are dismantling ancient and totally non functional systems, the TCO of the modern solution is higher than the previous. Look I’m here on HN, can program a bit, understand quite large parts of the tech stack. But not why even highly capable IT-staff and management whom I have confidence in can’t seem to explain their contribution to the basic underpinning of business: average cost should go down over time and with scale.

(Advice about how to talk about this with IT people is more than welcome! A lot of organizational distrust and poison sadly follows from this.)

One of the more current examples is that IT-staff forgets that they are 2-3x more expensive per FTE than general workers. So when I see a plan and calculate how much middle management should save in reduced workload thanks to plan X both sides don’t like the outcome. The end of the day picture that companies exist by grace of customers willing to pay for your product (and cost loading!) is lost at that moment.

IT people are terrible communicators if they have to speak to someone outside their domain. I’ve seen plenty of miscommunications because people don’t try to grasp whether the other person gets what they’re saying. You can have network, storage, os, and developers talking and using the same words and not conveying anything useful to each other because the words mean something specific to each party, but not the same thing. I don’t know if they’re to deep in their own bubble or if it’s just arrogance and they do not want anyone knowing what they’re doing.

So somehow everyone believes that IT people don't communicate well, but the IT people know that they actually communicate well. Erm, okay. I feel like there could be another explanation, but it escapes me.

Unfortunately, playing politics is part of communicating.

I find it’s more that people suck at listening, not communicating. IT people will happily explain if you ask and listen, but frankly most people dont like to feel dumb for asking questions.

When was the last time a manager asked you what you needed to do your job better?

Management is absolutely necessary. A good manager is worth their weight in gold.

But sometimes, management is:https://www.youtube.com/watch?v=r8miwsWtzRw

Exactly, use the politics to your advantage! Depending on who you're talking to, you have to have a different raison d'etre.

o Talking to HR: "We need to do X to reduce/increase headcount needs".

o Talking to legal: "We need to do X to comply with Y regulation or decrease our exposure"

o Talking to other teams: "We need to do X because our current system can't scale to take your load".

o Talking to other teams that don't want to work with you: "We need to do X because your system can't scale and we're going to drop our dependency on you".

o Talking to the business: "We need to do X to save $$$ this year"

But if you simply say "We need to do X because of [highly technical reason]" then no one outside of your team is going to care or help you.

This leads to the obvious conclusion that effective engineering in large organisations is about being able to talk to the political caste and identify who needs to be brought on-side.

If someone told the boss the equipment is good for 10 years, maybe you can work with them to come up with new requirements the old equipment can’t meet, and let them present those requirements to the boss, so they have a reason to support instead of opposing you.

The alternative to this sort of political shenanigans is to be ineffective, or to work in a small enough company that you don’t need to.

I'd agree that tech folks in an org can communicate very well and precisely -- among themselves. But there's a gaping chasm in communication between the technical and non-technical people.

It comes from a difference in motivation, in what the different "tribes" believe to be important. Bridging this gap is more about translation and seeing the others' perspective than anything else. This is the reason project managers exist, and why their existence is mostly miserable.

We need to acknowledge that every layer of this cake thinks they're the most important, but in a de-facto dictatorship fuelled by customer spending, the lines of power are closely aligned with the flow of money.

Tech decisions are approximately irrelevant to the people closer to the money stream, and they're primarily evaluated in terms of potential profits and actual costs. In other words, tech only becomes important when it facilitates more profits or incurs unexpected expenses.

I’ve found this to be true much more often than one might think.

The suits are cheap or dumb or it’s their fault they can’t speak tech. But often there’s just a lack of communication skills.

Maybe it is time we acknowledge that tech skills are a core requirement in today's world.

Maybe it is hard to communicate with people who lack a fundamental understanding of the critical tools they use.

I call this software literacy - we would not hire an illiterate CEO and we are in the middle of a transition from software eating the world to the world is software.

If you don't understand how to write software it's like not understanding how to run a newsroom for a newspaper - you simply won't make the right decision often enough no matter how clever you are

You don't need to understand how a fuel pump works to manage a fleet of vehicles. But you need to trust that when your mechanics are telling you that a particular model has a lot of fuel pump problems, they know what they are talking about. If you, the fleet manager, are approached with a plan to replace all the fuel pumps in a particular model with an in-house design, you still don't need to know the particulars of the in-house design. You do need to have some sense of whether you really have the in-house ability to competently design, manufacture, and sustain such a device.

It may seem like a ridiculous idea for a floral delivery service. But if you know yourpeopleenough to trust them or not trust them, you don't have to be an expert in everything to make the right decisions.

But making that right decision is largely based on the answers to “how long will this take?” and “how much will it save/earn me?”.

Which brings us a full circle to the nerds hating the suits because they want to quantify the unquantifiable.

OR, you could just go with what the experts say and be tagged a pushover.

it's not unquantifiable, it just has huge uncertainty. good managers supposedly can pick good strategies to manage exactly that.

This is what I thought off about the original story. Yes, the team got things done and was able to provision needed hardware by managing the pain signal. However, upper management was still isolated and didn’t trust or respect the IT team.

It's not really a cheap or dumb thing, it's a money thing.

Business execs want everything broken down in "how much will this make me" and "how much will this cost me". The problem IT faces is that sort of impact isn't easy (or is impossible) to articulate. We can easily speak to how much the fix will cost, but not to how much it will save or bring in.

This is why orgs are terrible at security. How much will fixing the log4j issue save? Who knows, maybe nothing, maybe the entire business.

My fellow autists in the IT industry have a hard time coming to terms with the fact that the answer to the question "how much will this cost me" in other departments is mostly made up bullshit, so you have to learn to say made up bullshit too.

... exactly, but professional bullshitting is it's own - very profitable - industry, underwriting

It's slightly more subtle than that.

The budgetary control management system that companies like this use isn't primarily designed to limit spending to the minimum necessary. Its primary purpose is to prevent any political power being built away from the management office, by constraining access to the critical resource.

I mean this literally and historically. Budgets being used for management control dates from when the CEO of General Motors couldn't control the sprawling divisions they'd bought, so he (well, his CFO) brought in budgetary controls invented by McKinsey (the man, not the company) so they couldn't move without his say-so. And of course because it worked for GM, it must be good for everyone.

Actually the prevailing wisdom is that it is those in the IT departments that lack proper social and communication skills.

And posts like this where you act so smug and callous do nothing to change this.

This post is an example of IT making it clear as day what the problem will be before it happens, it's amazing communication when the very obvious "trust what the actual experts are telling you" advice isn't being followed.

I've often wondered about this myth that the tech-nerds can't articulate stuff properly. In my experience with my coworkers it's not really the case, we go out of our way to explain everything to various levels of detail depending on who's on the receiving end.

What happens more often than not is that the C-suites and their underling managers just don't give a shit. They've got their "grand plan" that works in their minds, and no amount of annoying devs that tell them that their fantasies can in fact, not, be made into reality can sway them otherwise. Doesn't matter how well you explain it, they want it done, so it'll get done, regardless of any other problems.

Hell, they perfectly understand what we're talking about most of the time, it's not like we're hitting them with obscure jargon known only to the highest caste of the nerds, it's really just that they don't care. I do get where they're coming from as well, as we can be extremely annoying oftentimes (at least, I know I can be extremely annoying), but the feeling is often mutual when I'm given a ticket I've argued vehemently against, more often than not with good reasoning.

I just wish more companies weren't being run by brainless, heartless MBA types and there was more of us engineers in charge, but c'est la vie.

I think it’s going to always keep being like this. The people that would be good to have in charge do not actuallywantto be in charge because being a manager sucks for them.

it's because tech people - like any employee who doesn't have to bear professional liability - can be pushed around easily (because who cares if the suits are occasionally blaming IT, who cares if the company fails, the pay is good, there are other jobs, etc.)

also from the perspective of the aforementioned suits, they can hire IT guys easily, who cares if one of them gets very uppity and quits if we blame them for everything/anything.

"Good leaders have power thrust upon them, bad ones seek it."

the cynical offshoot here, is those managers, executives will blame the "communication problem" of IT When faced with their dereliction.

People also find it hard to understand or recognise exponential growth.

That’s part of the communication challenge. If the audience doesn’t understand exponential growth (kind of surprising for a bank exec) then you still have to explain the problem in a way they will understand.

“The current growth trajectory means we’ll be out of capacity in a month, and it takes 3 weeks to install more capacity, so you’ve got a week to find the budget. If we don’t do this, customers will experience a slower service which may lead to complaints and reputational damage.”

It’s also possible that the execs understood perfectly and were happy to accept the risk of the service being slower for a short period of time. Maybe the Internet-using customer base was tiny, or the service in question wasn’t critical, or the execs were handling general financial pressures leading them to want to sweat every asset until it was impossible to ignore, or maybe they just understood that “the internet being slow” was normal for the 90s and it would be unlikely to hit their reputation in the long term.

Or the execs were sweating assets as you say, but also understood that they could lay the blame entirely on IT when things started breaking.

Also the behavior of queues is surprising. Average wait time does not tell you much about the distribution of wait time, and is an extremely nonlinear function of the number of queues.

https://en.wikipedia.org/wiki/Queueing_theory

Perhaps using a log-scale y-axis would help!

As portrayed, it sounds like management agreed the growth would happen, understood it would become a problem, and were even willing to pay for the upgrade. They just insisted on delaying paying for it.

That doesn't sound like a failure to communicate.

That's why firms today have CTO with seat on the table.

Corporate IT grew from office clerks. Obviously you don't need strategic planing to see that fax machines and PC's work, so corporate IT started as being nobody important.

yup, i was waiting for at least a cursory mention of the actual growth rate and details to the cost and how long it actually took to install the line.

I would argue they communicated the risks by showing them the user experience

I once worked with this awful third-party software that caused tons of trouble for our clients. We had an in-house solution on the works, but some people wanted to renew the contract and keep using the same buggy software. Sometimes I would spend the whole day just fixing stuff on this system.

I made sure all the tickets opened by our clients went to the people who wanted to keep the old solution. It worked, we moved to our system and we rarely have issues with it.

I've found that the easiest way to effect change is by making the people who make decisions feel the pain of the problem. A lot of line levels do a lot of heroics in an attempt to keep pain from rolling up hill

Instead of the organization just doing the work to stop the pain, they throw bodies like so much human analgesic at the problem - I hope you'll let me torture this metaphor some more - and become addicted to this state of affairs.

Yeah I totally agree. I’ve met a lot of people who feel like a hero when they work crazy hours to “save the day”. But if the problem they fixed was caused by bad planning or bad decisions made by upper management, their heroics prevent the decision makers from learning how to run the company properly. Pain is an incredibly important signal. Our bodies need it to survive and companies are no different.

Go above and beyond, sure. But never ever be an invisible martyr. If it’s management’s mistake, management needs to feel some heat or they won’t learn.

A colleague once said that if you sacrifice yourself to make up for the firm's inefficiencies, they will never get fixed.

Which is basically what you've written. This concept should be discussed more often.

Tangent to this idea is the concept of 'back pressure' related to data flow, and the need to have a way to return feedback in a system.

I think reading this and other comments slid something in place for me about “Living Systems”.

What does the individual care more about tho, a chance at affecting change, or potentially getting fired?

Really shows the limits of an individual when their livelihood depends on the source of the problem

The trick to spreading pain upwards without getting fired is to communicate the situation upwards, repeatedly, ahead of time. “Hey I think that artificially imposed 6 month deadline is unrealistic”. “We’ll see”: Closer to the date: “That deadline is coming up fast. As I mentioned a couple months ago, I think we’re making good progress but I don’t think we’ll hit the deadline with the feature set we’re building. I think we need to cut features, push back the deadline or all start working crazy, unsustainable hours. And that’s not a precedent I’d like to start. What do you think boss?”. Etc.

If they don’t change anything, don’t suddenly pull all-nighters to then meet the deadline. A bad timeline was never your mistake to fix. And assuming you’re a productive member of the team, you’re not going to be fired for helping management plan like this.

I agree but my real experience with similar situations (twice so far) was at badly managed firms. I was ignored and branded a "trouble-maker"

The principle of spreading the pain upwards extends to making ownership feel the pain of bad management. But that’s a slow mechanism, and in the case of public companies, shareholders are so thoroughly isolated and insulated that it hardly works at all.

It works in the evolutionary sense - the worst compianies just die.

Counterpoint: SAP remains a going concern.

Hmm. That brings up an interesting take on “make work visible”.

making the people who make decisions feel the pain of the problem.

At a macro level this is exactly why we're seeing such a strong disconnect between political/corporate leaders and "regular" people. The ones up top who make the decisions don't feel the pain of bad policy because they're so sheltered from day-to-day reality of most people

It's a really deep problem. Representative democracy, for all its faults, beats autocracy thanks to the improved incentives of leaders. Businesses as a whole, if they have to compete with others, face good incentives. But within a firm the incentives can be really screwy. One obvious response would be to abolish the firm and make everyone an independent contractor, buying and selling stuff from each other. That would improve accountability but at the cost of an enormous friction replacing a process that used to be as easy as asking someone to do something. Ronald Coase dedicated much of his career to the question of how to balance those forces, how far the boundaries of firms should extend.

Thanks for the reference to Coase, I'll be sure to look him up!

sometime ago this article was discussed here: "let it fail" :

https://www.maxcountryman.com/articles/let-it-fail

I often wonder how some companies survive despite bad or even catastrophic decisions by the management - it's only because the soldiers in the trenches kill themselves to get things done and to unf*k all the C-level incompetence.

And then it reinforces the management's conviction that they are absolute geniuses.

This is the art of navigating a bureaucracy: make the pain happen at the same point that the decision is made. It's not easy, and from the outside it's almost impossible, without being very dogged.

I worked at one place where we had to deal with a lot of off-hours on call issues. Our manager focused on quick resolution and never let it escalate past her. Most of the time she wanted us to simply reboot service or server. Never had time to find root cause.

It happened a lot affecting our personal lives significantly, waking up in the middle of night just to reboot/restart some service.

Over time, the whole team started ignoring on-call alerts, and worked real slow. Pretending that our internet was down or laptop is not booting up. Longer it took to resolve alerts, our automated systems would start sending alerts to higher up in the chain. Also it started to impact our SLA and other metrics.

Finally, they decided to allocate resources to fix and stabilize the system.

I once worked with new awful in-house software that (thankfully) depended on 3rd party software. We had the legacy in-house system parallel running, which was even more awful. The legacy's awful came from it being unable to do anything it didn't already do without extensive re-wiring, and a team who believed if they don't share their knowledge of it they had a job for life. The replacement's awful came from it being inefficiently programmed with large DB latency due to the team's belief at throwing more CPU behind queries instead of re-writing queries.

The teams sat next to each other but didn't speak.

The legacy team called in a bomb threat to a package left under the IT director's desk. It was taken very seriously. While their cause was already lost, that didn't help them personally.

The new system team eventually called Oracle, who came and re-wrote their queries.

When you say called in a bomb threat, do you mean they reported a package as suspicious?

The call-in was an anonymous, apparently, call to the company's switchboard stating a bomb package had been left in the vicinity of the desk. We were evacuated to a separate part of the building for a few hours while it was dealt with.

I think it means they call the police (from a payphone if they are smart), and tell them there’s a bomb under the IT directors desk that’ll explode in x minutes/hours.

Reading some comments, like 'this is wrong because your job is to respect hierarchy decisions', reminds me how corporate still are military minded...

military minded

There are different military philisophies, not all are top->down strict hierarchies. E.g. in the Prussian army of old, missions were communicated through objectives (I want to achieve X, your part is to do Y, while these other guys will do Z), and execution is up to to the local officers who actually see the things as they are locally and have all the knowledge about what they need to do, how it fits in the wider plans, and are best placed to decidehowto do it. Furthermore, if an office or NCO disagreed with their direct commander's orders, they could go above their head to their commander's commander to protest. Add in the general staff system where staff officers had to service in command as well as part of their training, and could counteract commander's orders, it made for a very robust and flexible system capable of adapting to evolving situations and where officers didn't hesitate to argue with their superiors, go above their heads, and even disobey orders if they really thought it made sense.

What happened to the relationship if one went above an officer’s head? Would they be on their direct superior’s “hurt this person as much as possible from now on” list?

Not necessarily, because the direct superior knows thattheirdirect superior is aware of the situation, and the subordinate can always escalate if they're being unfairly targeted. But there were definitely grudges formed that way.

E.g. in the Prussian army of old, missions were communicated through objectives

This is called "mission-type tactics" or "mission command" (DE: Auftragstaktik):https://en.wikipedia.org/wiki/Mission-type_tactics

Also, special forces (especially the SAS) work this way. As ex-(regular) forces i understand the value of hierarchy, but getting the buy-in comes with unlimited trust in the leader (it happens) and/ or asking for opinions, and allowing decisions to be challenged, even over one's head. 'Get the result', is the request. How it is achieved is often secondary.

bacause management has roots in military. eg book extreme ownership

If only it were that benign. Management has its roots inslavery.

A significant part of why early company-owners in the US preferred slaves is not because their labour was free - they were expensive assets, particularly skilled ones - but because free workers were less compliant to hierarchical control. They could walk off the job, they expected their opinions to be listened to, and you couldn't whip them into compliance. There's basically a straight line from plantations through Gantt then Taylor and his Scientific Management to McKinsey's Budgetary Control.

You're both right. There's always some mixing of the two.

Culturally northern corporations tend towards the West Point Peter Drucker management by objectives schools of thought.

Whereas the culturally southern plantation class is as you describe. Walmart is always my go to example.

This also roughly tracks geographically with strength of Labor. To no one's suprise, the neoliberal (confederate) generations long assault on Labor means the southern style has been waxing. Hopefully, with the youngs' (Gen Y & Z) new found enthuasism for Labor -- in the form of life, liberty, and pursuit of justice -- the slave owners grip on our society will begin to wane.

Even today in many countries schooling system is for only making model workers for factories

https://en.m.wikipedia.org/wiki/Factory_model_school

If you actually read the book Extreme Ownership, you'd realize how wrong your comment is. It advocates decentralization and ownership from the bottom up.

which is ironic, because a large part of current/western military leadership is "coming up with the plan together"- because it builds buy-in from the lower ranks.

Like LARPers everywhere, corporate dudes emulating the army tend to project what they feel, which isn’t very informed.

I think it were the ancient Romans that created the distributed command for armies, where every decision is taken at the lowest hierarchical level possible.

It became somewhat out of fashion at the Modern age, but AFAIK no army operates purely on "your job is to respect hierarchy decisions".

Maybe I'm too spiteful, but with leadership like that I would've just looked for a new job and let the entire ship sink.

Yeah but you’d be blamed for the problems in your absence and the warnings you issued beforehand would be magically forgotten.

... and it wouldn't matter a nano-bit, life is too short to care what others say about you in some office you will never see again 10km away

Kinda, but it could harm your future prospects though if you’re unaware you’re getting badmouthed in your absence and word gets round. One degree of separation is all it would take before people who’ve never met or worked with you have been persuaded you’re a problem.

... still my advice is that life is too short to care about such things. In my experience people (guys too) that indulge a bit too much in rumors are massively insecure folks with various other issues, and it shows.

Have confidence in your skills and the rest are details, especially if we talk about IT where job mobility is generally way above average, youreallydon't have to kiss assess or be worried what others think/say about you to be successful.

Pretty accurate take. There are two truisms is corporate leadership world

1. They are never at fault 2. It was always the fault of the prior leader after a re-org

It's perfect.

face it - no matter where you leave from, or why you leave, your successors will almost certainly blame you for anything goes wrong in the first few months after you leave - you can't worry about it, because there is nothing you can do about it.

Maybe it's just that I haven't been fortunate enough to encounter leadership that's not like that, but this story sounds right to me.

Try to think of it from leadership's perspective. For the sake of simplicity, let's say there are 100 similar initiatives proposed each year, each costing $1 million. That's $100 million per year in pure costs to the business (ignoring amortization, tax shenanigans, etc). Even for a bank, that's real money, to be strategically spent and not wasted.

With that setup, making leadership feel the pain and understand at a visceral level why this particular million dollars is well spent, is all at once a good strategy, rational for leadership, rational for IT, and a healthy-ish dynamic for the business.

Aside/tangent: In a very real sense, executives are managers of the finite time, employees, expenses, etc. of the business. Now, you could, and I would, argue that they're nearly always not especially good managers, and that the purely rational choice from "the company's" perspective is to pay them more like regular employees and not so lavishly. However, "the company" doesn't make decisions, people do, with all their political games, incentives and self-interests. Since executives control the flow of information/decisions/resources/money in the company, they siphon off an out-sized share for themselves, acting as a (perhaps inevitable?) parasite on the host company. Fixing this problem is left as an exercise to the reader.

Since executives control the flow of information/decisions/resources/money in the company, they siphon off an out-sized share for themselves, acting as a (perhaps inevitable?) parasite on the host company.

Reminds me of the sociopaths in The Gervais Principle. Highly recommended.

Or to the writer. Bjarte Bogsnes has written a couple of books on this; Implementing Beyond Budgeting is worth reading.

The core of the idea is that the decisions that the overall body makes are so much worse if you try to combine managerial control with budgetary control that you want to separate them as much as possible. Set goals; provide budgets; don't link the two. Yes, as an executive you have responsibility for the overall spend. So employ people you can trust to do that well, and don't sweat the details.

I don’t know. It depends on how it was explained to them. If they just heard that by date X the link is predicted to be 50% saturated then IT did a really poor job. We know that 50% saturated mean with this technology that customers are going to experience delays, or connection errors. So why don’t say that? We expect that by date X the customers will experience connection problems.

Or to put it other way you know that the 100% limit is a theoretical limit only achivable in ideal situations, so talk about the utilisable limit. People need to be fed the right information to be able to make informed decisions. And since it is IT who is paid to understand the tech things it is their job to communicate the peculiarities in a way the suits can understand.

Now maybe they have done their best job at this, but the article makes it sounds like they just sent a memo without this info.

If you are a marginally competent leader you’ll ask the obvious question “why is it a problem if our link is only 50% saturated”. If you are slightly more competent you’ll have read some literature about business processes and intuit that the same queing theory might apply.

Once usage ticked past 50 percent, the IT team again made a pitch for an ISDN order. And were again rebuffed – with instructions not to ask again until utilization was nudging 100 percent.

This reminds me a lot about Covid perception by authorities: it's as if people in charge are all unable to make any kind of anticipation based on existing trend and have to see the disaster in front of them before reacting.

What's worse, fast forward N years into the future until the next pandemic, and all lessons will be dutifully forgotten, triggering exactly the same sluggish and misjudged response.

Don’t worry we won’t have to wait that long. However that won’t matter, people will protest any measure before politicians are able to react.

people will protest

Having a vocal minority protesting against something usually don't slow down politicians when they have clear goals[1], but they become a convenient excuse not to act when they don't want to.

[1] I've marched in vain for many causes, most of which had much, much more intense protests than what Covid restrictions triggered.

Let's say you marched for "in favor of choice", but the politicians chose to ignore you because they're backed by just-as-large numbers of people who feel strongly the other way.

With COVID, people who felt strongly about staying at home, were probably the minority, especially as time went on.

If a new pandemic starts in 10 years, people will still be burnt out from the last one and resist closure in greater numbers, immediately.

With COVID, people who felt strongly about staying at home, were probably the minority, especially as time went on.

Yeah definitely as time went on. At the start I think a majority probably supported staying at home but after 3-6 months the numbers flipped.

Had government acted as quickly as their population wanted them to st the beginning (that is: before it was too late), stay-at-home could have lasted much less time: when a things double every 4 days and take 2 weeks to half, waiting just 16 days before lockdown makes you needtwo additional monthsof lockdown to get back to the same point, which is exactly my point about people in charge not acting before it's too late.

Looking back, nothing could have prevented the pandemic, if not China itself in January. Once the virus was traveling, you can’t prevent spread. There will always be countries and people who will not follow the rules.

The lockdown did save lives, but could not do anything else.

Given that contagion to African and South American countries came from Western Europe, at a stage when China already had gained control over the Virus inside its borders, it's not clear that “nothing could have prevented the pandemic”.

It's obviously something that governments who failed to act want you to believe, but it's not definitive truth. Maybe the uncontrolled situation in Iran would have been enough for the pandemic to happen, but travel from Iran is also much less important (in volume and economic consequences) than travel from the US, France or UK, so there's no certitude here.

And again, late lockdown did save lives, but with huge economic cost whereas early lockdowns saved many more lives with only a fraction of the economic cost, people who delayed the lockdown bear the entire responsibility of the additional deaths and economic disaster.

They see lots of potential disasters all the time and have a hard time knowing which ones will materialise

When the potential disaster is following an exponential trend and have been doing so for quite some time, not acting because the figures don't look high is plain stupidity.

128 isone thirdof the way to 2 millions. And 2000 is more than half the way.

My first reaction was: ha, funny and a nice way to leverage your power as an IT person. Kind of revenge of the nerds towards the ‘suits’.

But then I realized that IT abused their power and lied to the suits which is outright wrong. At the end of the day you’re in IT and the suits are in charge. Don’t like that? Find another job or grab a position that gets you this power.

The problem is: you don’t always oversee the consequences of your roque actions. In this case the stakes were low, but where do you draw the line? If I were a suit and found out that you deliberately throttled the connection to force us in a decision I would be furious. That is simply not your call.

That sort of bootlicking has a history of leading to ‘problems’.

Your job working for a company is to respect the decisions of your manager.

Choosing not to undermine them is not bootlicking. It's called being a professional.

Your responsibility as a professional is towards the customer and company. Not towards incompetent individuals. Don't call yourself a professional when you are reday to do unprofesssional work at the first sign of resistance.

Your customer and company doesn't approve your paycheck. Your line manager does.

Going behind their back because you don't agree with their decisions or actions is not professional.

Your customer and company doesn't approve your paycheck. Your line manager does.

Here's a quite common problem. Execs tell you to listen to the customer, that's where the money comes from. Your direct line manager cares exclusively about himself at the customers' cost. What do you do? Which level of the hierarchy would you obey?

Going behind their back because you don't agree with their decisions or actions is not professional.

Should you go behind the back of your manager and report his corrupt behavior to their manager?

Please don't take this as snark, I've been in this situation multiple times and it's actually a difficult moral decision depending on the stakes.

When will people learn that there's a fundamental difference between obedience and respect? Respect has to be earned. My job as a professional is to execute the decisions of my manager, not to respect them.

Then, in the end you'd be blamed when the entire network died.

They saw the car sliding towards the cliff and did something about it. Nothing that would cause immense damage, just a minor annoyance.

To skirt rules to get shit done, or watch everything burn and say I told you so. What's your choice?

Then, in the end you'd be blamed when the entire network died.

I mean, you clearly have a paper trail showing you raised the issue at the highest levelstwice. Not really sure how you could be blamed.

To skirt rules to get shit done, or watch everything burn and say I told you so. What's your choice?

A couple of months of inconvenienced customers isn't exactly "watching everything burn", nor is anybody childishly saying "I told you so".

You're just respecting the choices management made. It's 100% their responsibility, not yours. It's not your job to save the world, or save management from their mistakes.

This isn't a situation of life or death, where you're in the military being told by your commanding officer to commit war crimes, where you have an ethical obligation to disobey. This is just going to inconvenience some internet banking customers. Your job is to raise the issue and let management deal with it and abide by their decision. Not to go around management.

> Then, in the end you'd be blamed when the entire network died. > > I mean, you clearly have a paper trail showing you raised the issue at the > highest levels twice. Not really sure how you could be blamed.

This is sarcasm, right?

The "highest levels" can't possibly be wrong, so either it's space aliens, witches, oryou.

Not at all -- I've seen it happen many times. Higher management tries to blame somebody, that person shows a paper trail, and higher management shuts up. They can't do anything, there's nothing they can put in the employee's HR file.

People literally generate paper trailsso thatthey can cover their ass. Many times after a particularly controversial meeting, I've immediately written out a summary of what was agreed upon and why, e-mailed it to everyone in attendance, and included the note to reply if anything is incorrect. Then when somebody's upset 4 months later and tries to blame you, you show them the e-mail and the lack of any disagreement.

Obviously at the end of the day, legally a company can fire you for mostly any reason at any time. But generally speaking in corporations, there are processes in place around penalizing employees that require documentable facts.

Of course I'm talking about places that have at least a couple hundred employees, with an HR department as such. (Smaller companies are much more at the whim of their founders, for good or bad.) But this bank certainly sounds large enough.

People may claim this is an unlikely story, or entirely made up. But I don’t think it is. In fact, if you want to get anything done in a big organisation, you have to make management feel your pain. This is not being cynical, it’s simply how the world works.

This makes a lot of sense - but how? By definition, management does a different job from individual contributors. How do you make a manager feel the pain of a messy code base, for example?

Making changes take longer. This will take 6 months today or 2 months in clean code base

How much time it will take to get to clean code base ?

The pain is indirect. They’ll just see tons of failed production deployments, rollbacks, client calls, defect rate, delivery rate etc. Somehow you have to make it clear to them that yes, all of that, is caused by the message they send out every single day, reminding people to focus on delivering all promised features today.

Like some IT version of ‘the whippings will continue until morale improves’.

all of that, is caused by the message they send out every single day, reminding people to focus on delivering all promised features today.

More realistically it was caused by those messages, but with a 4 year lag time, back in the days when the CTO decided to start the project from day 1 with a team of "senior" consultants that were all "full stack" and "excited to learn AWS".

- Very long time to implement new features and even more time to change existing ones.

- bugs that customers complain about.

When asked for why you can explain how the messy codebase is causing problems and how there no bypassing it.

I've seen this in a big medical equipment company where the CEO made a whole speech about how he now understands how legacy codebase is making our work very difficult, it caught me by surprise to see a C level bring up the topic.

I have management introducing unnecessary operational work into what would otherwise be a fully automated process. The code for it being fully automated is already written, it works, and worked in production for a year with 0 issues. They told me to remove it as a means of them controlling things. Anytime there is an issue with the artificial controls they asked for, I have everyone go to said management to get approval. This not only lets them feel some pain, but it causes huge delays in the process, which makes their entire product look bad. I'm also looking to throw the operational work back on the manager and his people. My goal is to annoy them into changing course while I don't do any extra work. If they want to make stupid rules that create work, they can be the ones to do that work and deal with the fallout. I'm not going to act as an enforcement arm for rules I don't agree with or understand.

how the world works

How communication is sometimes bound to work.

Those who expect agents to perform all mental steps on their own to understand situations are in queue for harsh surprises.

I like the stories on the register, but keep in mind that those stories are always told from only one of the sides.

And now they have double ISDN while the first one is used at 55%, so 26% capacity usage in total.

Or maybe the CEO knew something that the IT people didn't.

Like for example the business was struggling and they weren't expecting the same growth i.e. their capacity planning calculations were incorrect and there was no need for an additional link.

Then the CEO should have explained that, end of story.

(It's possible that they did, and the retelling is so biased as to exclude that fact. But it seems unlikely.)

If another ISDN is the straw, the back is broken.

Once usage ticked past 50 percent, the IT team again made a pitch for an ISDN order. And were again rebuffed – with instructions not to ask again until utilization was nudging 100 percent.

If you know that installation of the new line will takeNmonths, and traffic is creeping upX% each month, simple arithmetic will tell you how far ahead you should order more capacity.

The instructions of waiting until 100% shows a lack of understanding of logistical capacity planning (perhaps through a bad explanation/communication of the problem).

Wilfully sabotaging executive's network connection in order to get more money for another link because your boss couldn't justify it properly is inexcusable.

These people deserve to be fired and the behaviour is disgraceful. I can't imagine working with anyone like this.

I’d agree with you if the problem wasn’t real, but we all know that utilization at 50%willcause trouble.

I say these people were skilled in savingthe company’sass. They could have just as well waited for the “I told you so” moment, which is what I learned to do.

The problem with waiting for the "I told you so" moment, is all the people you warned conveniently forget about all the warnings they were given. Then if you bring it up and say, "I told you so," or provide proof, you get labeled as petty or focusing on the past instead of fixing the problem in front of you.

I mostly keep the "I told you sos" in my head for these reasons.

Yeah, at that point I either kindly "add context" with an issue from last year or quietly bask in schadenfreude.

Sabotaging? Clearly not.

They applied proactive network management to fairly allocate an expensive resource according to the anticipated needs of the company.

Hilarious and all too believable. What on earth is wrong with organisations??

Bean counters, among other departments, are distant from the technology powering their business. They only see tech as a cost sink and don't understand or care about the implications of their decisions.

Actually you can scale that out to more than just the finance departments. Another example is in large orgs, tech teams don't meet customers and don't understand real world usage and subsequently don't care or put thoughts into the implications of their actions.

Inefficiencies of scale? Would that be the term for it

Give a team a budget - if you achieve your goals and come in under budget you get half the under-spend as a bonus.

Don't let the bean counters force budgets - incentivize them.

What on earth is wrong with organisations??

They are made of people.

What on earth is wrong with organisations??

They get too big.

It's too much to hope for, but I wish the right people could do this for the nonsense around e2e encryption these past few years.

Imagine if politicians' https connections could be identified and automatically downgraded to http.

What would happen then? How would that lead to the change you hope for?

I think they would stop pushing so hard to break encryption if the people sitting next to them in cafes started live-tweeting the politicians' internet traffic.

But that misses the point. The politicians are not arguing for a world where nobody can encrypt. They want strong encryption which can only be opened by two groups: the intended recipient, and the authorities.

We know that such a system will inevitably lead to other groups also gaining the ability. Rich criminals, security services all around the globe (friendlies and the enemies too). Either by bribing, or impersonating authority figures or by stealing the backdoor keys. What is at dispute with the politicians is the probability of this happening.

How would a service voluntarily leaking the private information of politicians convince them that we cannot build a backdoor into encryption without that backdoor proliferating widely?

Honestly, I don't believe this story for a second -- and believe me, I've witnessed my fair share of corporate incompetence, so it's not that.

But it's that if there's anything executives aregoodat, it's looking at graphs, extrapolated trend lines, and setting deadlines before disaster strikes. This is basically one of thecore, day-to-day competenciesof the executive suite, because there areso manythings that can go wrong regarding finances, infrastructure, legal, etc.

There are plenty of things executives can bebadat managing, e.g. when it comes to tech debt or user experience or company morale -- especially anything subjective or hard-to-measure like these.

But capacity planning foranymeasurable resource, be it an internet pipe or otherwise? This is MBA 101 stuff. So this particular story just reads like a fantasy of "tech people smart, business people dumb". If an executive team couldn't handle something this simple, there's no way they could have been running a functioning, established bank in the first place. It just doesn't add up.

I agree but I think the more likely disconnect here is that the tech team couldn’t be bothered to properly present their case. They might’ve just passed on a note with a vague “we will run out of resources soon” or something.

Given when this was (early 90s going by the story?) I could easily believe it... nowadays there would be an executive level senior manager ultimately responsible for "run" who would actually have a clue on forward planning for capacity but back then running capacity to its limit, being told several times "no" along the way and then being blamed when it all fell over were very much the norm, especially in banks, in my experience.

I've worked for several large corporations now directly supporting c levels, and their primary skillset is self confidence - a belief that they deserve their position.

I think the real problem being illustrated here is that the IT side of the shop needs autonomy to do what it knows to be correct somewhat independently of the business side of the shop.

Ultimately, trust is what governs how much oversight you should be expected to suffer in an organization. In my shop, I don't have to ask for permission to do anything unless it touches our customers in some way. If I see a machine that is struggling or a SaaS plan that is reaching quota, I simply upgrade. Every now and then someone will ask me about a new bill or increase, but I don't have to subject myself to multiple depositions to move the ball forward.

The suits should consider the downsides of creating an environment where one has to plead for permission to conduct the most trivial tech work. How much innovation is going straight into a burning trash can over convoluted change policies that were established in some ivory tower a decade+ ago?

Can we somehow reframe the organization in a way where the business is modeled as acustomerof the IT team? That seems to me like a much simpler way to think of things, because at the end of the day if the business fails, the IT shop is has no raison d'etre.

If the business is the customer, upgrading a machine increases the customer price hence it requires communication as it changes the benefits/cost value proposition.

The trick is to wrap your technology with a service-style abstraction so you don't have to communicate all of these painful little details. Then, all of those details get rolled up into something like a quarterly report.

The terms of this relationship (contract) should be approximately along lines of "you provide us aservicewith a certain level ofqualityand in return we grant you an annual budget of $X".

At a certain point, having detailed analysis of every last expense will cost you way more money than it will save.

Not upgrading a machine affects quality of service. If they don’t trust that that’ll happen until they can feel the problem themselves, what do they have an IT team for?

Ah yes. In my first job the CFO repeatedly rejected IT manager's requests for a proper backup system of the AS/400 that held all of the business (ERP, CRM, accounting) because backing up the work of 300 people on 8" floppies simply didn't work.

One day, a major disk crash occurred, and the whole company went frozen for a couple of weeks (couldn't fulfil orders; couldn't manage support requests; couldn't make any proper commercial proposal; couldn't find a customer's number or address from the database; etc). IIRC some disks were even sent to Kroll Ontrack. After that disaster, of course, a proper backup unit was bought...

"We need a test environment!" "We don't have the budget for it."

Months later, a student developer runs a delete query in production with the where clause commented out. Oops, an entire table got wiped. Well at least there are backups... except a background job noticed and quickly created 4 years of invoices and emailed every past customer to pay again.

A test environment arrived soon after.

Many organisations take a similar approach to the problem of cybersecurity.

Early in my career, I'd cobbled together an internal server which effectively did monitoring (RRDTool) against the internal BugZilla(!) server. The end result was speedily cached charts and graphs of bug counts by team, by project, by release, etc.

Over time, the system had grown to be basically _THE_ source of truth for the release management process for a Top-500 website of the era. Go/No-go decisions were based on looking at the trend-lines for bug graphs. Projects were merged to "trunk" only if their P1 counts were low enough.

When I decided to leave I wanted to make sure things were left in a sustainable place, and asked around for who to transition the server to (it was a random beige box in the QA Lab, circa 2005). "It's Debian running MySQL, it has an auto-update script which will bug you if you need to update packages, here's how to restart the server, here's the root password, here's how to update/deploy it, docs are in this wiki."

I'd given 4 weeks notice, and week 2.5 rolls around and even though I'd kept asking around about who to transition this pet project to, nobody was swinging by my desk (so to speak) to pick up the responsibility.

So I made a little script: `@hourly: if (( $RANDOM % 2 )) ; then ifdown eth0 else ifup eth0; fi`

Suddenly people started showing up at my desk... "What's going on, can you fix it, what's happening?" Sorry, nope, can't fix it, the network on it is down... see, I can't ssh into it, by the way, this is what could very easily happen once I've left. Just wait an hour or two and it should come back online. If you send someone over who you want to be taking care of this machine I can walk over to the keyboard with them and show them how to fix it.

Once they committed to sending over someone to transition to, I turned down the BOFH to a 25% chance of 15 minute outages...and all was right with the world. Later on that week, a really good guy, one of the main end-users of the software (sorry Mike!) came over and explained the relative immaturity of my actions, but I couldn't help but point out their effectiveness.

In retrospect, it was an internal "API brownout" but that didn't have a name yet. The concept of "controlled unreliability" (eg: chaos-monkey) being a tool to increase business resiliency is something to take to heart, if even only unofficially.

So did you eventually transition it to someone? Wouldn’t be the first time that I’ve seen someone commit to something and not do it without notice.

Or learn to live with random 15m outages because it’s apparently less frustrating than doing something.

Yeah, minimally I could pass off the root password and after that I turned off the unreliability script.

In retrospect, probably frightening how quickly the business could become accustomed to random 15m outages as a cost of doing business.

What I don’t get is “50% utilization” as if you have a faucet on half way. Surely with a single pipe and a spikey load you would get occasional performance degradation anyway? I would imagine that “shit experience” time percentage driving up sharply (depending on the distribution) so that even 60% would be intolerable.

For comparison imagine a bar decides to have 1 server based on average demand. Then Friday night comes.

Yes, latency will depend on the variability of the load. That's Kingman's Formula for you.

I get this is probably a reflection of corporate culture in the 90s (a bit cartoonish, I think) but I think this attitude does little to make Tech taken more seriously and “at the table”.

Instead we self-reinforce the image of childish kids doing mischief and pranks when the things don’t go our way.

The CTO was at fault in this story, for failing to convey the gravity and for being used as a doormat if things went bad.

Bold of you to assume there was a CTO back in the days of ISDN... even in a bank.

“Regomise” kinda threw me. Here’s what it’s supposed to mean:

https://english.stackexchange.com/a/546162

The Register itself uses the term Regomiser as a user name generator for its registration. If this contributor is a "reader Regomized as 'Felix,'" it would mean a reader registered and was reassigned that random name.

Is it regomiser or regomizer? Or has the term remomize been regomised as regomiser?

He didn't go for the neck. The corporate view was that the executives detected the problem, allocated resources to it, and were applauded when it was "solved". Heck, some may even have got a promotion.

Going for the neck would mean documenting everything and letting the bomb tick, and when it exploded make it clear to even higher-ups what was happening.

You'd probably go as well but that's not a good place to work anyway.

I just love awkward English.

Thinking, or evenknowingyou're right isn't sufficient grounds for concocting fraudulent ways of proving it. This approach runs the risk of blowing up in your face if discovered.

It might be seen as underhanded but in fact it was the diligent and proper thing to do.

Creating calamity in a controlled way is a tried and true way of being sure one is prepared for the time it strikes uninvited, see DiRT exercises.

Ah yes, "managing upward"...

A tragically believable story.

Australia? Banking? I assume the early 2000s? Absolutely plausible.

But they never learn. And this experience didn't teach em. What they get is destroying a business due to random/bad decisions, which is basically what most business majors seem to operate on. Then they get their golden handshake and leave to become a C level at another place where one of their buddies from their fancy business school is already parasiting.

Most of them seem to barely get by by operating on the surface level strategy of "do whatever x creates y money more". When they succeed, it's their good job, when they fail, it's the bad job of everyone under them.

They only look at surface level money, they'll take surface level "we can make 35% more if we do x" rather than the deeper and more considered "we can make 25% more if we do x but limited by y for z reasons because x has a negative impact on customers but mitigated by y, costing 10% but guaranteeing the 25 via happy customers and stable systems".

read the dailywtf for more and better stories like this

Looks like BOFH is writing under a pseudonym.

https://www.theregister.com/offbeat/bofh/

I had a Cisco 1604 ISDN router setup at work to autodial to save money rather than being connected all of the time. Unfortunately, I discovered IBM AIX when the web browser package was installed contained periodic telemetry dial-home to Big Blue every hour or so that prevented the lab network from idling happily until I added a firewall rule to the router to block that sneaky shit. Even in the late 90's, Microsoft, Sun, and Novell weren't as brazen about telemetry as IBM.

Sometimes it's hard to communicate technical problems to non-technical people. They article calls that as going for the neck, but I think of it as effective communication.

Boomer executives especially at banks are like children. They only understand what they see with their own eyes. They can’t think in abstract terms.

Plot twist: This was 1999, the brave techies convinced the dastardly "suits" to double down on the internet revolution, the dotcom bubble burst and the bank went bust.

This works for getting attention from IT too. I’ve had issues that IT said they couldn’t fix, and I just put tickets on the system for every instance the issue came up. Then, oddly fixes appeared.