I've been contemplating the issue in the headline, as pertains to my value. When I help somebody in 40 mins with something they've been stuck on for 3 months, my value is clear to everyone. When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear. Don't know how to deal with this paradox.
In about that same timeframe, Y2K is also really good example. Nothing of note really happened. But if people had just ignored it a lot of things probably would have.
This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs.
There were a few jokes about it in the newspapers but by and large the public just ignored it.
I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
I'm sure I'm not the only one that can't wait for the 2038 problem to get closer. bank accounts gonna be jumping like a 3-peat Micheal Jordan when those consultation fee checks start rolling in.. provided the Boston Dynamics dogs haven't taken over yet
Java EE will be the COBOL of 2037...
Don't insult COBOL like that
The mitigations were done so well today the public remembers Y2K as a “scam”.
I suspect most people don't even remember a 25-year ago (what intervention made a) non-event as anything.
Since you "work on climate" I am sure it got your attention that there is a massive effort happening all across the board. R&D has made tremendous advances, whole industries are investing to get away from fossil fuels, and governments have ongoing programs to accelerate and support lange hydro, solar, and wind projects.
Some countries have not heard the call yet it seems (ones that like their role as perpetual victims and blame everyone else but themselves), but even just five years down the road, the world will already look very different for all the effort that's being made.
> I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
Unfortunately those fixes are slated for Humanity 2.0, Homo extinctus.
Y2K would negatively impact companies' bottom lines if not dealt with, directly, in a short and known timeline, in predictable ways.
Climate change will impact companies in different ways at different times. Oil companies, for instance, are incentivized to ramp up production as much as possible while also diversifying.
The way to get all companies to take appropriate actions is to not allow them to externalize costs. That means a significant carbon tax.
On climate change, it already did work once with the hole in the ozone layer.
People ask why they don't hear about it anymore -- oh, it's because we listened and banned CFCs and fixed the hole.
What, what do you mean nothing happened? I'm in my underground bunker, do you think it would be safe to come out and look around?
You guys are all in your Y2K underground bunkers still, right?
I didn't worry. Even took a trip to a different country. Though I can totally understand why some people (know you're joking) might have played it more conservatively.
Yeah, it's the year 24, don't worry about it. A human is probably still able to type this.
My advice looking back over the past 25 years . . . stay in the bunker.
Another example is ozone depletion.
yeah, I've had that discussion
"we were all worried about the ozone and nothing happened, this is just a repeat and people will stop talking about it any time now"
and my response is "'Nothing' happened because the governments acted fast and turned it around so now it's mostly just boring stuff like catching some factoryy that decides to go cheap and use banned chemicals to save a buck"
the readiness paradox:
If you are prepared, nothing interesting happens, life moves on, people opened a notebook and pressed a few buttons.
If you are not prepared, the texas grid freezes, people die, people lose their savings, and "nobody could have imagined it would be this bad".
Recommend "Left of Bang"
In strategic sysadmin and cybersecurity this is staple thinking, and a useful part of what I teach now.
Problem for vigilant thinkers is the rewards, as explained in TFA, are perverse.
People are content with technology to be magical. They don't see the millions of people quietly repairing it and planning to keep it all going smoothly.
To "bosses" cybersecurity only bring them problems, and cost them money apparently doing nothing. The same for any defence force, until you need it.
Did some nice interviews for international sysadmin day last summer [1].
[0] https://www.goodreads.com/book/show/22663095-left-of-bang
Yeah. I've heard folk who should know better (looking at you, John C. Dvorak) call Y2K a nothingburger and suggest the effort to prevent problems was a total boondoggle.
Disclaimer: I was part of that effort. The funny thing was that I was brought back to a previous client to fix something that my previous efforts had literally caused. Took me 20 minutes to fix it once I saw what the problem was. And then there was a "while you're here, can you take a look at this ..." That lasted about two years until they closed the department and moved it to NY, NY.
At least I got credit in terms of billable hours.
While I often enjoyed reading Dvorak columns in PCMag, he was always more someone who liked to provoke people than offer thoughtful commentary.
Not "probably". I was involved at ground level fixing code for BP in '98. Can say without a doubt that bad things would have happened. And not because my salary depended on it - there was no shortage of opportunities to work on other things, obviously. It was legitimately an issue that would have crippled the energy industry (major players, and all their many, many dependents). I infer from this experience that the impact would have been the same on many other industries (e.g. finance, resource development) directly and indirectly.
But it is a great example. I still run into people who recall Y2K as an example of much ado about nothing. No, no!! It wasn't an issue for you because many people worked hard to prevent the problems.
Those problems were not terribly complex just pervasive and critical, requiring a high LOE. It wasn't like a huge moon-landing engineering problem to be held up as some accomplishment of humanity, more akin fixing a lot of dumb Challenger o-ring problems before a blow-up.
Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
This is a management problem, because no-one wants to be accountable for a repeat incident even if it was rational to be working on something else more important.
There's a ton of reactionary legislation on the books that exists pretty much entirely because politicians wanted to be seen doing something. It's mostly crap.
Yeah, the legendary politician's fallacy:
1) We must do something.
2) This is something.
3) Therefore, we must do this.
I thought that was the voter's fallacy.
The voter's fallacy is "this happened while this guy was in power; we should replace him" repeated ad inf.
This is probably a bit too cynical; it's not just "politicians who want to be seen doing something"; the public often wants something done as well. And "we didn't do anything after that incident from ten years ago and now it happened again" is not a good look. Complex stories about trade-offs and the cure being worse than the disease often don't "play well" in the media, especially not with the opposition demanding that "something could have been done!" And corporations tend to be pathologically risk-averse.
Politicians, the media, corporations, and the public all have a part in this.
the public often wants something done as well.
Sure, that's why the politicians want to be seen doing it. But the legislative focus is often on getting something passed now while it's a hot issue rather than on eventually getting a good bill passed (let alone deciding that we actually already have the correct amount of legislation on the topic).
A lot of gun legislation is like this. Whatever your view of guns is, there is no rationale for a "safe hand gun roster" like California's where the Glock 17 Gen 3 is on it, but the Gen 5 -- which introduces a basic safety improvement for left-handed people -- is not.
Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.
I've heard this called "institutional scarring" in a blog post somewhere. The idea is a small wound can be replaced with tough inflexible tissue. The jist of the blog post was that just because something happened, doesn't mean you have to change things to ensure it never happens again because that can be an over-reaction that really burdens your future. Accept that loss and that it just might happen again, but that may be better than onerously preventing it with certainty.
I remember this blog post, it was also a chapter in their book Rework https://signalvnoise.com/archives2/dont_scar_on_the_first_cu...
For posterity: the term is actually "organizational scar tissue" not "institutional scarring".
Surprised you found it despite my butchery!
Same thing constantly happens in governments, too. "Oh no, something that's been done for 200 years now caused issues once! We have to restrict/regulate/bureaucratize/outlaw it immadiately!"
I called the dev form of this “tech trauma.”
e.g., You were tasked with working on a ball of mud and it was miserable, so the next system you get the chance to build has to be the most scalable, modular, and cutting-edge thing ever, just to be safe.
The thing is: it's very easy to play up the likelihood and severity of problems that are entirely imaginary. This can be just a bad habit, but also a deliberate tactic. In either case, a lot of effort, time, money, etc. is wasted.
Waiting for something to actually happen before allocating resources to preventing it is (to some degree) a rational policy.
yeah that to the Ai doomers who are making six figures off of it
The argument there is that waiting for "it" to happen would be way too late. Especially if "it" is a billion people are dead and the rest of the people don't have much time.
This is basically how all bureaucracy comes about. In startups everything is so new that problems haven't had time to happen. In big tech, because they have a tremendous knowledge base of previous incidents and the resulting safeguards, every single step seems mired in bureaucracy.
This is why death is a feature and not a bug.
I have worked in finance tech all my career, so I'm not sure how other orgs work, but this is extremely pertinent. Large investment banks react exactly like this.
I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.
Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.
I tend to prefer working with people whose stuff "just works". I take that back. I strongly prefer working with them.
And people who comment their code. Even if it is just a little. I swear, 90% of my time in coding is just fiddling around with functions to see what they do so that I can actually pull them together in the right way. If there was just a little documentation this would greatly reduce my time. It should also be in every team's and company's best interest because people hours are expensive. It should also be in the best interest of an open source project because more people will be able to build on your stuff. It should also be in your own interest because taking the 30 seconds to a minute to write those lines interrupts you and allows you to rethink and verify your method. So not doing it is only in the interest of moving fast and writing spaghetti code. But you're not actually moving fast. You move fast at that moment, but not in the race.
Documentation exists in one of these states:
1. incomplete
2. wrong
3. missing
My goal is to write code that is so clear it doesn't need documentation.
Some functions are doing jobs well-defined enough that this is sufficient, but there's plenty that aren't and do deserve additional explanation.
I don't want to read the code of your function to figure out what its doing. I want to read the function signature, and if that's not clear enough a comment explaining its purpose and parameters. Code explains how it does something, but often does not clearly explain what that something is.
I don't expect to attain it, which is why I remarked on it as a goal. I've discovered that a lot can be done to eliminate the need for some forms of documentation. Also a lot can be done with the language to make it more expressive, and thus eliminating the need to document something.
For example, having the parameter to a function declared `const` means the function cannot alter it, and this is checked by the compiler. It won't be necessary to mention that in the function documentation.
P.S. This doesn't work in C and C++, despite them having a `const` type qualifier. This is because the `const` is not transitive, and can be legitimately cast away. Therefore, it's useless as a guarantee. D's `const` is transitive, and although you can cast it away in @system code, you're on your own with that.
For example, having the parameter to a function declared `const` means the function cannot alter it, and this is checked by the compiler. It won't be necessary to mention that in the function documentation.
Actually you should mention it in the documentation. There's 2 types of documentation, so it's easier to just do the one: documentation for users, documentation for developers. Unless you work absolutely alone on all your projects and you don't open source them, someone else is going to be reading your code.
Why should the documentation repeat that a pointer to const cannot modify what it is pointing to?
Why should the documentation
> someone else is going to be reading your code.
Someone else is going to {read,edit,write,maintain} your code. Which means __anything__ the code does should be explained. Before you were suggesting documentation is only for the user. Documentation is also important for the developer. Whoever takes over your code later or works on it with you.
I was objecting to comments like these:
i += 2; // add 2 to i
const(int)* p; // p will not change what it points to
Of course, a precondition is that the person reading the code knows the language reasonably well, and I'm not writing a language tutorial.
In your opinion/experience, does (or: would) a docs-as-code approach help ?
Docs-as-code will never work, as human languages are way too mushy and imprecise. That's why we have programming languages.
I document as I code all the time. It is immensely helpful. As I write the signature I add a few lines saying what the function will do and often it results in me slightly modifying the function because I realize a better way.
I think you miss the point. Docs-as-code says keep the docs where the code is, and manage the docs like you manage code, using tools like git. Mostly(?) to reduce friction for software developers to actually contribute useful documentation.
My goal is to write code that is so clear it doesn't need documentation.
I hear this so often but I think it is an excuse. Those 3 points are also true about your code.
There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors
Those 3 points are also true about your code.
The difference is there are things the compiler can verify to be correct.
This conversation is a bit difficult when you're ignoring a fair amount of what I and others have said. I think you're wanting to be right so interpreting the words to support that.
The compiler does not tell you that code is correct. It has no such capability and even LLMs are far away from doing that. The compiler can only tell you that the code is compilable. That means the syntax is correct, but it does not mean the logic is. The logic is much more abstract in terms of correctness.
Please at least document the 'why' of the code. Anyone can spend an eternity on a codebase and figure out 'what' it does, but its very difficult to figure out the 'why' without someone explicitly telling you.
As a software engineer who pays more attention on security than average, I often feel my work didn't get recognized. For example, if I successfully prevented a supply chain attack, nobody would say thanks to me for a thing that didn't happen, even when they see a competitor product gets attacked. Similarly, most C/C++ programmers do not really care about integer overflow. But I know we are no longer in the world that computer viruses are everywhere(though ransomware are still common). The, who made it better? The people who are not satisfied with stuff that "just works".
I think you misunderstand what I mean by "just works". It means it doesn't have bugs in it. That includes security holes.
This is wonderful description of the kind of hellhole corporate Scrum is turning into.
Agile was literally about doing things quickly and quick cycles of capability improvement. But Scrum is a worse version of the planning processes it meant to replace!
If anything, the way scrum lays out the work into immediate problems exacerbates the cycle. In the long run it just turns into a ticketing system where fires get pushed up and technical debt gets pushed down.
It even spits out a super easy-to-track, meaningless set of efficiency numbers for consultants/executives to min-max!
I found the idea Scrum being efficient when done right was true at one company I worked at. Everyone was committed to the process, the Scrum team included debt as a priority for 20% of the effort and everyone had a fairly accurate velocity that we could bake in an additional 20% of your work as your own interests, so stakeholder priorities filled that remaining 60%. Then we would pivot in some Sprints if any epic/team goals required a push to get done or other emergencies/bugs that required a change in priority.
I don't think it can't be done right, but it's been cargo-culted pretty hard at Fortune 500 companies. But the best scrums I have ever done have just been literally post-it notes on a whiteboard - I think Jira ads a level of complexity that is unsustainable for most orgs.
In my 17 years in the industry I saw working agile twice. Both were small places, both used post-it notes. Both were consultancies, each team dedicated to an independent project.
Everything else, world-class, public, private, consultants or not... was a joke.
I don't think cross-team collaboration works in agile. Either one plans ahead, or everything becomes an unpredictable mess.
Similar, but one place, and same with post-it notes. It went down the shitter when Jira was force fed and all the problems related to keeping them in sync arose.
"b-b-but other teams need visibility of your sprint!" no they don't. I don't think I've looked at other board other than by accident.
Scrum
I always thought this just meant to scrumble every week to get things done with a weekly standup?
I believe it's actually a rugby term, and the idea is to wrestle things across the line at a regular cadence as a team.
The unit of work is even called a sprint because the idea is specifically to commit to very intense units of work.
Some managers tell their teams that they need to be more agile, I guess when referee yells "scrumble!" everyone sprints with a lot of agility.
I'm an engineer type, most of this makes very little sense to me.
Scrum is a worse version of the planning processes it meant to replace!
You say this as if it was a coincidence.
I'm allowed to say such things. Some of my best friends are scrum masters.
I still can't believe a scrum master is an actual FT job - last company I worked for had one FT scrum master for each project - so they ran the meetings and as far as I can tell, did nothing else - what the heck do they pretend to do for the rest of the week?
I went to all the same meetings, and was still supposed to actually develop software in between them all.
I've found it works better when you "build up" to Scrum by incrementally adding processes when you identify issues and relaxing the process if things are working well.
Adding a bunch of processes because you want to add processes doesn't provide value.
Agile seems to have ended up as a way for PMs to report upward to senior managers who also need to report upward.
You can see why. These people have to decide what will be worked on out of all the potential things that can be worked on. Someone has to make that decision. Will this feature make us money? What about that bit of work that doesn't add a feature but reduces resource costs? What about tech debt that I'm told is building up and slowing the ability to deliver features?
I'm not a senior manager, but ultimately someone up the chain is responsible for the company surviving and making money and paying our salaries. They are just like you and I, trying to make decisions based on what little information that can glean. So part of that is "what will this cost and how much will it be worth" vs "what will that other thing cost and how much will it be worth".
To that end, they need some way to estimate this. They latched on to agile as it was being promoted by tech as a way to do this. Whose fault is that?
And so with that came all the frequent estimations, are we on track, rituals, etc. Some people don't believe this should naturally follow. I agree. But somehow all those rituals have become part of the cult.
We abondanded scrum. We abandoned refinements and estimation of stories and story points etc. Now we meet with a PM once a month formally and as a team perform a t-shirt size estimate on where we are. Otherwise we update her as frequently as she asks (which isn't often) or we want. This gives power to us but because of that we're conscientious and make sure to inform her timeously when things are looking sketchy or whatever. Yes, we still have to give "estimates", because ultimately, senior management want to make decisions, but it is otherwise quite lightweight.
It is so liberating.
I would call it "the hellhole that corporate performance assessment is turning into". Scrum is just a tool
Nobody ever gets credit for fixing problems that never happened (2001) [pdf]
I'm reminded of this every time I see some YouTuber or other social media click bait claiming that the Y2K bug was no big deal.
The reason it was no big deal is because thousands of graybeards, like myself, stayed up many long nights for months ahead of time making sure things would work.
I still remember the tension during the countdown to midnight UTC. Then the tension rising again during the countdown to Eastern time. Then once more at local time.
Only when it was 2000 in Pacific did we unclench our cheeks.
I still don't quite buy that. Surely it's because computers mostly use epoch time for dates, not dd/mm/yy?
Guess we'll find out in 2038.
Now they do at least
They did then as well. I remember before 2000 people saying it wasn't going to be a big issue due to this.
"Most" programs used epoch time, like video games. The minority that didn't were the big, stable, old programs that ran on big computers. Things like payroll and flight schedules.
Many old programs written last century did not use epoch time.
https://en.wikipedia.org/wiki/Year_2000_problem#On_1_January...
It's worth noting that most of the programs affected would not be the ones people used on their private pc.
It was also to do with how dates were stored in very old databases that were conceived in the 70s when storage was at a premium - a lot of them just used two digits for year.
Interesting that everyone is responding to the first half of your comment. I think 2038 is going to be more interesting than Y2K...
That's not what I remember programs looking like. You would have counters that went to 99 and no more because the UI wouldn't accommodate another digit, and programmers didn't know what they were doing, and figured it would be clearer if the software reflected its UI and couldn't handle over 99 at all, invalid UI means unrepresentable state.
Lots of stuff didn't care about the day, too, like you'd get 0997 and that meant September 1st 1997 because the first (or last) of the month was inferred, and you'd get goofy logic around year++ where you add 100 years whenever you want to increment the month, and the whole thing is written as a modulo 1200 but whenever the date is about to be 0000 you look at what's stored in year instead and add one to it instead of 100, because that way you have less variables to allocate and every "little bit" counts.
Everyone knows now, but lots of people writing programs didn't have any prior art, they were just good at messing with computers and sort of fell into programming by accident.
Depending on which Y2K bug you're referring to, this very much happened and is an easily believable scenario. I don't think this is a comparable situation.
Went through this exact scenario at my job recently. As the tech lead/architect of an org I reviewed a bunch of recently released projects and identified areas that needed critical improvements because they had major reliability/performance issues. One team had multiple releases at the top of the list, and their PM and eng manager (and really everyone else up the chain) simply ignored all concerns because they had to prioritize feature updates instead.
Fast forward a few months – I was on vacation and shit blew up. There was a sev 1 escalation, several customers were pissed, the CEO/CTO were involved. The team in question – the same one that wrote the shoddy code and ignored all warnings bells – worked round the clock and got the service back up. Now they are heroes for their effort and the same manager I mentioned has a great reputation at the company for being so active and communicative and showcasing his leadership during the outage.
Well, you could dust off the old email and 'mistakenly' send it again. Though it will seem petty, a small fraction might rethink the past few months.
It might be better to keep it low-profile. Go to a superior with the story of "told ya so".
Actually any organisation worthy of the label "engineer" should do a thorough, no-blame, and collaborative post-mortem of such events so they don't happen again, and that would include warnings given.
Feature on time + risk, feature late + no risk (and anything in between), it's in the end an engineering and business decision and either choice might be the right one depending on the circumstances.
Sounds also like a good way for the two tribes (eng & mgmt) to formulate a way to communicate tradeoffs better.
I’ve started doing something like this more frequently, but it is in the interest of job security / cover-my-butt rather than pettiness.
I build something that depends on many systems that others have created and that uses data from probably about every dataset we have. What I build also has very high customer visibility. So whenever something breaks upstream, the first place customers notice the problem is in the system I maintain. Psychologically, people begin to develop a mental association around your system being a “problem” if it keeps getting brought up as the starting point of SEV discussions and customer tickets.
As a result, a lot of what I do now is defensive. I investigate and reverse engineer upstream codebases to identify likely failure modes, and I spend hours analyzing datasets of questionable origin for data quality issues and inconsistencies. I document and date stamp all of the problems that I find, file bug reports and assign them to the relevant teams, and write proposals describing potential solutions to what I see as large architectural design flaws that will come back to haunt us at some point.
All of this work is promptly ignored with “not enough bandwidth right now” or “not a priority compared to feature development”. Which is fine. I document all of those responses too.
Then eventually, something breaks in a big way that is again first noticed by customers within the system I am responsible for. In the past, I would immediately drop what I was doing and scramble into investigation mode for a few days to prove I wasn’t the root cause of the X hundred thousand dollar issue, er... I mean the blameless SEV review & postmortem... but more recently my preemption seems to be paying off, and lately I just reply to the panic with a bunch of links to old Slack threads (where we already discussed the issue), the documents and proposals I created (that no one read), or the bug reports I filed (that were never followed up on).
Perhaps it does come across as a bit petty, but I try to be as polite as I can, and from my perspective it’s an improvement over the previous situation. The only downside is that all of this preventative work takes away time from the primary work that I was hired to do.
Not petty at all... pretty smart imho. Does the metaphorical front-end catch back-end errors with a warning and link to the issue? That would light fires under asses. :-D
This can be framed constructively, do an analysis of what was fixed during the outages from the originally identified and now lobby for funding to fix the rest.
Performing some heroics once in a while shows someone can be relied on. However, if heroics are the norm, it’s either bad work or bad management, and it should be looked at with some scrutiny.
I also find it more impressive when someone fixes someone else’s problem. I’m not one to shower praise on someone for fixing their own mistake, nor do I expect any praise if I fix my own mistake. I would be apologizing to everyone for screwing up in the first place.
In our org, this was brought up. People who had problems, then fixed them, were the ones being seen and recognized by higher level management, and awarded for fixing the problems, when they, often, shouldn't have happened in the first place.
Now we have something akin to a "person that prevents problems" award. It's a nice gesture, but, even though I received one, I think it's mostly nonsense. You don't get the face time/experience presenting with the higher level managers. It's almost impossible to prove that you prevented catastrophes, by working extra hours, or doing the right thing. Nobody likes a "Hey look, I told you so!" sort of perspective, no matter how it's framed.
To me, this is the most gratification-limiting aspect of being a software engineer: time is required to test quality of foresight/design. Rewarding retroactively, for great past decisions, doesn't happen. There are systems I designed, when making a pittance for compensation, that's still being used a decade later, across multiple companies, almost entirely unmodified. That, now objectively, good work wasn't seen or recognized by anyone paying me at the time, and can't be seen by anyone now. I burned good time for a "good job me!" as a reward. As I get older, that mental masturbation seems less desirable, since it's usually at the expense of the surprisingly limited life credits that we spend on those tasks. But along with most other nerds, I can hardly stop myself because it's an innate obsession.
Award everybody with a bonus that can be deducted with every bug report
Oh, maybe adversarial salaries. You each, meticulously, go through the others code, trying to find as many bugs as possible, in your spare time. Each bug you find transfers a percentage of their salary into yours. End result: everyone's code is perfect, and everyone hates each other. :D
Potential dual Ph.D in game theory / management.
Reminds me that one time I found 17 databases in our org without backups configured. All I got was a thanks from some devops guy and no manager even heard of the problem...
This happens in organizations that don't measure and respect precursors or leading indicators of success/failure -- organizations that do not practice organizational observability, to coin a phrase. Which is to say, most of them, because this is not something one thinks of until one has suffered its absence. Your managers have to think like SREs, and be technically sophisticated enough to model and address this problem.
Does anyone know any great management books on this subject?
The High Velocity Edge had a lot about this subject.
The most memorable example was at a hospital, nurses would occasionally pick up the wrong vial. They always noticed before injecting, but there were lots of close calls that didn’t get tracked anywhere. Once they started tracking that as a leading indicator, it became pretty obvious that the vials needed a better design to prevent even the possibility of grabbing the wrong one. Also, if the hospital never realized this problem and someone got hurt, they would’ve ended up with a much worse solution like “double check really close”
This happens in organizations that don't measure and respect precursors or leading indicators of success/failure
This happens in organizations that become over reliant on metrics and indicators of success/failure instead of recognizing that these are fuzzy concepts and indicators are just that... indications, not answers.
This is a great insight, the idea that management has to look at leading indicators for the org itself. Makes me think that beyond line management, a manger’s job is to behave as an investor and steward of the org, instead of getting involved in the details.
Not a management book per se, but Working Backwards popularised an Amazon approach of thinking about the organisation in terms of mechanisms. You might find that useful.
I’ve always wanted to dig into Scarlet Ink, Dave Anderson’s blog, but never have, and I suspect it’ll be quite useful for you too.
This us an incentives problem, plain and simple. Its extremely difficult to quantify things that don't happen, and because we've been building our economies and societies around a heavy focus on quantitative metrics we end up only incentivizing people to solve problems. Even create problems on purpose if you need to, you'll get credit for fixing them.
Human Resources departments often run into this exact scenario. Many of the tasks that fall into HR are hard or impossible to quantify, and if done well they're also solving problems before they manifest.
I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.
I'm sure there are some that do this.
I once worked with a test manager that was effectively judged by how many bugs were found and how many tests were written. He was actually a nice guy just trying to play the game given to him, but if you didn't know this you would think he was doing exactly as you described.
There will always be a few bad eggs in the mix, but I'd check the incentives forced on them before judging too harshly.
The ones I’m dealing with now aren’t very transparent about these things. I’ve tried fishing a little to see what’s behind it, but need to tread lightly to avoid getting yelled at for an hour.
Suppose on Sept.10, 2001, you were an airline executive who successfully forced installation of jimmy-proof, secure steel doors between the cockpit and the cabin?
The only "credit" you'd get is "This guy cost the airlines god-know-how-much money, and for what? An imaginary threat!"
The book "The Black Swan" by Nassim Nicholas Taleb contains this very example. It is a great read on why the improbable events have the biggest impact.
which is where I got it (forget the attribution, though, but the name "Taleb" seems to trigger some people).
Wow, really puts things into perspective.
Yup
My manager just told me I got a 3/5 on my eval yet again because in the past six months
- I did everything that was asked of me and
- I did a lot of things that weren't asked of me and
- I didn't catch anything on fire and
- I put out someone else's fire and
- I prevented several fires from ever being started in the first place,
but that's not good enough. The person who started the fire that I put out is getting promoted because they showed initiative in starting the fire. I didn't show enough initiative putting it out for them or preventing fires on multiple other projects. Apparently creating two of my own projects that never caught on fire also didn't demonstrate enough initiative. From now on I'm applying for one new job per day on company time.
Don't take it personally, managers at bigger companies have to evaluate their teams on a bell curve. The whole team can't score a 5 and someone has to be worse than the group.
It sounds like the fire starter should get the worse score. Of course, we’re only hearing one side of the story.
Your manager probably just doesn’t like you, find a new one if you can, there’s no point swimming against the current - you’ll just get tired and get nowhere
This explains anti-vaxxers beautifully.
If everyone is vaccinated and the disease disappears (and kills few people) anti-vaxxers think: “why do we vaccinate everyone? The disease didn't kill anyone... and it disappeared."
So if you have a headache and you take an aspirin and the headache disappears you're going to conclude it's from the aspirin, right?
In headache case, the link between illness and cure is instantaneous.
But when it comes to vaccination, due to the time needed for everything to take effect (many people need to be vaccinated, it takes time) anti-vaccines are unable to link the vaccine to a cure for the disease.
For humans, time changes everything, even the ability to make a connection between illness and cure.
The opposite. Department heads get fired for allowing blow-ups in their area, that should have been avoided. "Getting credit" means keeping the job managing the department via preventing problems.
But then you need upper management that appreciates the "risk potential" of every area, and what is required to keep that risk at bay.
I’ve seen it mean the department head is told to cut staff. If things run smoothly with 100 people, let’s see how runs with 90, then 80, and so on. The preventative measures start taking a backseat as the staffing issues are felt. The cuts keep coming until something falls over and the department head is pushed out after failing to maintain the service.
This is true in a sense but also it isn't. Our institutions spend a lot of resources on preventing things from happening, are successful, and the people who work in these institutions get a paycheque every month.
One could frame the entire US military budget as preventative spend.
One could frame the entire US military budget as preventative spend.
A decent percentage of the population thinks the country should reduce military spending, because they are undervaluing the preventative spend.
Or because the prevention ends up killing hundreds of thousands of innocent civilians. Oh, and it turns out the reason you supposedly went to war in the first place is completely fake.
Corollary - create fake problems then solve them.
So become a management consultant? ; )
Or mid mgmt at tech co
Reminds me of this cartoon (which I have posted on my office): https://naksecurity.medium.com/the-detriments-of-hero-cultur...
This is why in many corporate cultures it pays not to proactively stop a problem that you know how to fix if the problem is not in your immediate problem area. Let it be noticed, let it become someone else's emergency, then fix it. Much better path to a reward that way. Of course, you should also be planning to leave said organization, in the long run it won't do well.
I think that over time people realize who has a fire drill every month, and who quietly just gets things done.
I wouldn't be so sure. As an employee and team member, you'll be better perceived if you do your share of performative firefighting every one in a while. I wish it wasn't true, but it usually is. (Though at remote teams, it's less and less true, with the remote teams I was on, nobody really cared).
"wow, last week we had such nasty bug, impossible to track down, also caused production reliability issues. Tom stayed up all night, and finally pushed the fix at 3 in the morning."
Now, if you then say that it wouldn't have happened if 1. Tom didn't overcomplicate the system, 2. Tom learned our tools properly, 3. Tom actually understood the requirements and at least tested his changes at least once manually, 4. Wrote good code so that it's easy to troubleshoot 5. Wouldn't have forced a rushed PR review together with the product owner.
You'll just sound like a bitter, know it all.
My memory is a bit hazy but it's a good story.
At one place, there was some important order processing taking place. As is fairly typical, couldn't rely on getting all the required info. Or critically, getting all the required info correctly. Some extra data, slightly missing pieces but enough to work, etc. etc. Some could be pretty gross. We built validation to massage some inputs, modify processing, what have you to address as many of those as we could think of. The team put in some metrics to identify which validations were "triggered" for each order. Neat. If we'd add more, we'd add a date to it.
It was great. We reported those stats so anyone could see anytime, but we'd also send out some comms about it every now and then. It also helped tremendously when a coworker or customer or whoever would say "Oh no! What happens if XYZ?" and we'd say no worries, we already addressed it, and we prevented #### orders from getting stuck for XYZ.
It showed that the team was thoughtful, that we were invested in making this work, that work needed to continue to keep things running smoothly, and we had data to back that up.
Really helped switch the conversation in the org because folks could see it. If someone pointed out that we hadn't thought of something, it honestly was more about what do we do vs. why didn't we think of this. (Yes, there is a comment here about poor org thinking or blame culture, but some of that exists everywhere.) More proactive. Recognition of preventive quality. People gave us accolades and it bubbled up to some good and real recognition at higher levels too.
I'm mangling the words here a bit but I hope you get the idea.
There is a simple but undeniable element of genius to the idea of, instrumenting the number of validations triggered. I would hazard a guess that 90+% of teams would focus on an "order success rate" type of metric and call it a day, instead of a "# of times bad data was handled" metric, which is exactly how you escape this trap of good work going unnoticed.
Thanks for sharing! Shamelessly stealing and applying this concept at work next week!
Yeah. Another way I would put it is to also look at inputs that strayed off the happy path. There may be something to learn from them as well, maybe something that adjusts the happy path.
Ah the old ops justification problem
Everything is working => "What are we even paying you for?"
Everything breaks => "What are we even paying you for?"
Only take. No give. - Dog
Or: The Preparedness paradox [1] / "There is no glory in prevention"
and the related discussion: https://news.ycombinator.com/item?id=32466346
There's one thing I don't get about Y2K:
It was supposed to be banks with lots of COBOL code being impacted, right?
Banks issue 30-year mortgages.
So, why weren't banks being impacted in the early 1970s with mortgages due to be retired in the early 1900s?
A better way to think about it is that you can trust the code you test. Y2K was a problem because it affected _everything_ because anywhere a date was used had to be checked. For mortgages, yes, banks did fix that code earlier but they did it because someone noticed a problem in 1980 and fixed that specific problem, not because they transformed how they wrote code and checked all of the millions of lines of other code which nobody had reported problems with.
Roll ahead a couple decades, and now you have even more code with lots of interconnected assumptions and the guy who originally wrote it retired to the beach at Margaritaville with zero interest in going back to Cleveland to talk about date math. If you’re lucky, a pile of cash could change that. If you weren’t, they passed in a boating accident last month.
My first tech job was working for a COBOL vendor in the mid 90s and heard a lot from customers and other people in the industry. Much of it was cosmetic (rolling from 1999 to 1900 on a display, etc.) but a lot of it would have had real consequences: not accepting credit cards with dates in the new millennium, calculating interest incorrectly, sending people incorrect notices or failing to send correct ones, etc. I remember a someone at Mastercard saying they wouldn’t have been able to processed credit card transactions at all after midnight if they hadn’t done several years of careful work.
The two things which helped were that it got enough attention to get the business people to pay for remediation, and there were enough dates in the future that people got wake-up calls in different industries. Mortgages were the earliest but closer in you had things like credit card expiration dates which caused enough problems that people got serious about deploying updates.
These days, I’m wondering how 2038 will go. We have a LOT more devices floating around and there are still plenty of new devices shipping with 32-bit embedded Linux which may never get updated. Hopefully most of that cheap IoT stuff won’t last another decade but I’m kind of something like an automotive company getting publicly outed for skimping on technical debt management and some people getting locked out of their smart houses.
I thought this was going to be about Y2K, since it was written just after.
I worked for years in the late 90's on Y2K projects, helping to stop the UK's critical infrastructure from just stopping at midnight. Wales would not have water or gas without our efforts, for example.
But I've heard people since say things like "why did we spend all that money on Y2K, when it clearly wasn't a problem since nothing happened?" and even "Y2K was a hoax invented by the IT industry".
We won. We successfully stopped the Y2K bug, and it was hard work, and it wasn't obvious that we had caught it all by midnight. Yet rather than celebrating, some people saw it as evidence that we had been ripping them off. People are weird.
I know of several other issues like that. I myself made sure the highscore in a video game I was working on would work :-)
The thing that annoys me is that this is best case scenario when it comes to climate change. If we do in fact manage to avoid the apocalypse, all the "climate deniers" are going to feel vindicated.
Take a leaf from the OH&S book, as OH&S is all about stopping things from happening. Many work places have a huge "Days Since Last Lost Time Accident" sign hanging out the front. It increments every day and the aim is to get it as high as possible.
The same metric could be applied to "Days Since Last Stuff Up That Cost Us". It would be easy thing to apply to an office door or cubicle wall.
I beg to differ. If you have a metric on "time since last outage", people will be incited to hide problems.
There is extensive writing on this subject http://rachelbythebay.com/w/2021/06/01/count/
I always get cynical when I see awards for software releases. Someone always gets an award for staying up late to fix a last minute problem. Normally the person that should have written good code in the first place so they wouldn't have had to come in late. The award should go to the person (likely people) who never got called in at all in the first place because we didn't find a bug in their code.
You guys get awards?
FTFY: nobody ever gets credit if they don't publicize their successes. By default, fixing problems before they happen doesn't make headlines, so you have to find clever ways to earn publicity. One politician classic: let it become a "problem" enough to earn publicity, THEN fix it. Two examples: Y2K and Healthcare.gov
spot on. As a general rule of thumb I keep a form of activity/issues log and send a monthly summart to the finance guys and head of dept along with "expected issues". Letting things crash and burn is a thing I learned from one of my managers a while back. It's great for raises, bouses and rep. Takes a good deal of effort not to fix things I know will break though.
I don’t get credit for my tiger-repelling rock even though we haven’t seen any tigers.
Are these magic rocks for sale?
The corollary (if you’re cynical) is that you get Massive Credit for waiting to deploy a solution until the problem becomes a Major Crisis. That is, there’s a big incentive to to prepare the solution preemptively but not deploy.
I call these problems ticking time bombs. This way business has a catchy name to use for the thing they dont understand, its also very clear this needs to be looked at.
I work on a large distributed infrastructure. I always joked that my team's projects and people's careers are outage-driven: the only time we become important and people get opportunities for promotion is when where were big-enough outages that executives have to invest heavily on reliability or scalability. Other time, we are just minions who must listen to and serve feature or product teams. Nobody listen to us when we ask a product team to implement a reliability contract in their shining product.
Pre-outage improvements, reliability defense in depth, eliminated scalability bottlenecks before they are hit, are all ignored by leadership and the company: it is just human nature that even though they understand you have to prepare for possible issues, if it hasn't happen yet, you won't take it seriously. I've seen this in many internal performance reviews and promotion committees. People who haven't ever got bitten badly by an outage may call these premature optimizations.
Nice article!
just aint true if you file a bug report
Reminds me of a place I used to work. Every time they asked for feedback, my refrain was "nothing here gets prioritized without a PIR (post-incident response)". Towards the end of my time there, whenever a PIR-related ticket showed up, I would mark it as a duplicate of whatever actual ticket I had dying in the backlog that would have prevented it. It really hurt team morale to have no influence in preventing foreseeable issues in our domain area. Most of the rest of the team had even stopped suggesting improvements altogether because management refused to let us pull our own tickets in.
Having spent several years as a manager, as a once-again IC I now include lack of problems with my features in production in my written self-assessment. I'll often refer back to major features I shipped ~2 quarters ago with a statement like, "Project Foo continued to function as expected, scaling as designed under load while incurring zero production incidents." I'll often include the words "engineering and operations excellence" and "commitment to product quality" when doing so.
Its tricky to get credit when your skills are x10 or higher compared to your team. 2 companies ago, a team of 40 engineers took 4 years and $29M to do a generation product update. The next company I worked for took $1M 5 engineers and 2 years (product from scratch). My current company I did the whole thing myself in 14 months (@ < $0.5M), basically on autopilot. Unfortunately, the stakeholders are unaware that they got enormous value by being lucky of hiring the right person for the project. And I still fight management/budget constraits and hot headedness. Almost enough to drive me to next employer.
One time I worked on a team where we got rewarded for being "bug heros", because we resolved a lot of bugs. It was kind of funny because we also generated the bugs.
SAP S4/HANA- the "new" ERP software is hardcoded to calculate depreciation until 2045. As programmers you probably know it could be untill year 9999 or much further.
But the consultants will not be out of work. Also a way to push customers to upgrade.
Artificial problem.
On a side note, I there is in my opinion a place for a new ERP disruptor. However making it is hard. Still even those small fish often manage to get money.
Just responding to the title, but I think this is why it's important to have leadership with real "IC" experience. If leadership has an intuition for what's hard and what the failure modes are, they will be able to correctly divvy out acknowledgement more often than a "pure manager" type.
Great article, and much of this resonates - but what template has been used here? It looks like it was created using LATEX.
This is why you don't travel back in time and kill Hitler
Life is not fair move on
It even gets worse sometimes. You detect a problem that poses a risk to your organization, manage to convince management that it's dangerous, you get some time to fix it and you fix it. Then, because it's fixed, nothing will happen. And then management frowns upon it and wonders why all this effort was needed.
A big scale example of this was the Corona vaccination. After the large vaccination campaign in 2020, hospitalisation numbers stabilised to levels that were also seen sometimes in previous years during some flu epidemic. Which led to criticism from some people, who said that this was evidence that the whole Corona epidemic was a hoax, organised by a conspiracy of medicin manufactories and policitians who wanted to scare people to gain more power ..
Like the button logic that works but never had problems?
I call this “I, Pencil Socialism”.
We live in a world where everyone depends on the output Of everyone else - none of us could live without farmers and hauliers, doctors and street cleaners, all of whom Build on a foundation from the past
Assigning value, billionaires collecting rent, that’s the flawed model
It’s not capitalism that makes this world possible (the assumption of I Pencil).
It’s the sharing, the trading, the leaving money on the table because it’s too complex to work out how much Elon Musk’s 3rd Grade Math teacher should get for teaching him the basics of finance.
You don’t get credit - you get to live in the modern world
If there are those who do not get to share equally in that world - that’s a bug not a feature.
We need to fix the bug.
My background is systems operation. The exception I could see to this is when there are rewards/incentives/etc for uptime or non-change-window uptime. Usually this would be coupled to meeting the goal of 4 9's or 5 9's of uptime, meeting SLA requirements, etc.
Granted, in my time working for a major 3 letter tech company on account with a major finance company my highest exposure to VP/C*O types was fixing the fallout from a whole chain of corporate culture related mistakes. If I hadn't already had a job lined up elsewhere I probably could have gotten a good bump out of it. I doubt that would have happened if I'd identified and fixed the problems. Part of that busted corporate culture. We didn't have metrics around uptime/SLAs there though which considering the clients business was pretty ridiculous.
This is kind of a problem with preventive ethics in general. It seems to me that given two bad choices, the best option is actually neither - that is, to prevent the choice from being necessary. To use the common example of the trolley problem, the best answer isn’t to choose one option or another, it’s to prevent the trolley from being about to run someone over in the first place.
Unfortunately the visibility of these non-events means that their heroes go unnoticed. Noticing them would require a fundamental reorganization of how we perceive time and the future.
Nobody gets credit for fixing:
- Problems that never happened.
- Problems that people don't realize are happening (where people cannot relate visible symptoms to their root causes). You only get credit for addressing visible symptoms. Though, disturbingly, you will get credit even if the fix is only temporary... This creates an incentive to only address symptoms since not solving the root problem allows you to get credit over and over again for fixing the (recurring) symptoms... In fact, people will praise you even more as a 'Relentless hero who dedicated their entire life to the problem' LOL.
Also, nobody gets punished for creating new problems if the visible symptoms cannot be traced back to the root cause. That's why I oppose complexity in all matters - It obscures root causes and sends everyone off on a wild goose chase addressing symptoms... Complexity turns everything into a frustrating game of whac-a-mole.
The person causing a problem can even get the credit for solving symptoms of that problem.
In all seriousness, do companies which don't suffer from this dysfunction exist? If so, are they hiring?
That's why I deploy buggy products
There is an interesting twist to the issue if you are doing development support, having a customer ticket queue. If you are diligently fixing errors before they ship you can profit from that. The queue shrinks, the stress levels sink, management is happy with you.
However, if the number of tickets gets too low over some period of time, well, management gets ideas that you have free capacity. ;)
It seems like many of these TQM discussion miss one key factor of TPS.
Respect. Engineers and management respect line workers. They have a word for the expert line workers, ("craftsmen"), Takumi. https://www.allaboutlean.com/toyotas-takumi/ and Takumi have a valued place in the company.
Without that basic level of respect... i dunno.. seems like it wont work
This is something that really annoyed me at a company I worked at for a while. And it was far worse - it wasn't a case of not being rewarded, people who planned properly and got the job done were punished for their efforts. If a project got behind they would take people from a project that was on track or better to put them on to it. I pleaded for a super green status for the best projects on the basis that the team would finish sooner and then the entire team could work on something else nnd get it done well too. They could take people from the next tier but that would be an incentive to do well. Not a hope.
The same thing occurs for many optimizations. Unless something is unbearably slow and you speed it up 10x or 100x; most people won't notice.
You might spend some time making an often called function return in 1 millisecond instead of 2 or 3 and some might notice that things 'seem' snappier; but no one knows why.
This may be one of the biggest challenges facing human civilization. Climate Change, Financial Bubbles, etc.
Except TSA security theater. They see their budgets grow and seen as essential, for preventing problems that never happened.
But in this case, if those problems were to happen, they wouldn't be stopped by their measures.
I run an ad agency. I only touch ad accounts with pre-existing spend snd results for that reason. If I take the client from $200 CAC to $100 CAC I'm a hero. If I go from scratch to $100 they don't know my value and it's much more work.
Volvo probably is an exception in the car world.
My first ever job was a placement during university -- like a year in industry.
It was a regular-joe IT support job at an eLearning company (if they still exist).
The FD walked past me one day, having just left what was clearly a deeply uncomfortable meeting.
He stopped in his tracks and asked, quite abruptly, why I didn't ever seem to look busy. I replied - with the arrogance of a cocky 20 year old - that it was because, right now, everything is working as it should. Nothing is broken and people are working and productive, which is because of the work of the IT department.
The company later become insolvent and was dissolved. So it turns out it might have been him who should have been busier, and he was just taking it out on me.
"Great wisdom is not obvious, great merit is not advertised. When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks of bravery?" - Sun Tzu
Reminds of when you see people questioning the usefulness of getting vaccines for some diseases because they have almost entirely disappeared for the past 50 years (especially in 1st world countries for some), without realizing that this is precisely the reason why they don't reappear.
Welcome to Infosec.
Reminds me of the millennium problem. There was a post on the large forum asking if that issue was over-hyped because we didn't see the world crumble. The top response was that it was handled because we threw a lot of money at the problem. I tend to think that was right, but there's no way I know how to verify.
Companies say they care about security, but don’t pay as much
On their start date, I would to tell new grads assigned SRE / DevOps roles that their job was thankless and at best if everything worked fine no one would notice them. However, if anything went wrong, they'd not only be on the hook to fix but would do so under tremendous pressure. "Who wants this responsibility?" I'd ask.
Those who decided they were up for it usually did a great job, about half opted out at the end of the conversation and were put in roles more suited to their ability to handle stress.
For the auditory learners, here is a summarized audiobook of this PDF :
https://player.oration.app/c3feabfb-d437-4545-ae17-235d50d61...
I like to classify what to prioritise by how many standard deviations from the happy path a certain issue is.
For example, I’m working on a pay-as-you-go SaaS - so the probability of using the SaaS while your account balance is low is 1-SD away from the happy path.
Figuring out if the user has JavaScript enabled (SaaS is an SPA) would mean the user first signed up with JavaScript enabled, then disabled it. Id put it at 3-SD away from the happy path.
Keep doing this and work on the bugs nearest to the happy path.
Of course, the 1-SD, 3-SD estimates are just hunches.
Only previous submission that had a discussion (and it's worth reading through):
https://news.ycombinator.com/item?id=8940820 - Jan 24, 2015 (50 comments)
It’s the Jonah paradox.
Theoretically this is the military's entire job.
This is a failure of internal accounting. If maintenance is reduced, the value of capital equipment declines faster. Production reject rate increases. Accounting systems are not good at catching that.
Repetitive manufacturing operations which capture many metrics about what a steady production process is doing may capture such info. Non steady state systems have a real problem.
I'm both a developer and a security consultant, both jobs deal with this problem. Being a security person, you go from being a no-one until something happens, then every eye is on you. And with the dev side, I work mostly on backend, so my work goes unnoticed always, the frontend dev gets all the credit, no matter how many thousands lines of code I commit, if something looks prettier or just works it's because of the frontend dev.
There's two choices, you either don't give a flying fuck and deal with it, or you go as loud as you can to make yourself visible beforehand and try to get credit for your work, your choice.
Easy - half a$$ your code and when it breaks - swoop in, "fix things" (actually do it right) and play the role of hero! (I've seen so-called "Rock Stars" at places I worked do this over and over)
Don't do it right, double down on the flaws. You had a 3k SLOC single function (all in main) C program to do something that could be expressed cleanly and clearly in 200 SLOC. Some specific sequence of inputs leads to an error. Instead of tidying it up, removing the repetition that led to the mistake, you copy/paste everything again and add another 100 cases to your various switch/case statements (actually you use if/else because switch/case might make things clearer). The specific problem is solved, but in a year another buggy code path will be discovered and you'll have another chance to play hero. In 5 years it'll be 50k SLOC of C all in main, that could have been under 1k SLOC (still all in main). No one else will be able to fix it but you!
For extra points, keep the sane solution in a secret source file, then implement a compiler which generates the obfuscated spaghetti. :-P
I kind of did that once. For a proprietary language that didn't support functions and had very limited support for loops (mostly labels and gotos, it is sold for way too much per seat for a niche industry and the language and implementation are awful). I needed to implement something that was too tedious to write out by hand (~1500 lines of this nonsense) so I made a Python program that spit out the code I needed.
The company that makes it also has a generator they use when they're contracted to make more code in their awful language. It's bad, the code has to be manually cleaned up and sometimes is delivered broken. Then they get the money on the support contract to fix their problems. (It's not their only product/service, but it's key to many of their services, a way of hooking customers.)
Their generator did not using the looping constructs, and several of their "senior" engineers actively discouraged it because "You can't be sure a loop will do what you want it to do." (Wish that weren't an actual quote.)
From my limited experience with "proprietary bullshit languages that only exist to hook customers", there's ways to mess up how loops work in the language. So maybe those senior engineers actually know what they're talking about.
I've seen "languages" that treat variables differently based on whether they have uppercase letters and underscores. Before I saw this in action, I believed the seniors telling me to "never use uppercase letters ever" were insane...
Given the cost of the product, I considered all documented features fair game to use. If a bug was found because of a primitive looping construct, then they were on the hook to fix it (there weren't any, they worked fine). One of those same senior devs also didn't know basic logic rules like De Morgan's Laws and tried to "correct" my conditional logic and broke the program instead.
MUMPS is a hoot isn't it.
Not MUMPS, that one actually has a standard. This is a straight up proprietary, maintained-by-one-company language. The concept isn't bad (that is, the objective of having a way to automate a formerly manual, and lengthy, set of processes), but the implementation is. Market capture, that's the name of the game for that company. They slip this into all their projects so even if they lose a follow-on contract they still get license fees for their proprietary components.
and enjoy your participation to the ioccc :)
Ah, I see, the legacy RDBMS developer approach!
Would you feel good about being that kind of engineer, if the external validation was great enough?
There are people with integrity and principles; and then there are people who can pay their rent every month.
I was kinda happy when my company replaced their previous feel good nonsense ‘principles’ with ‘do the right thing, even when nobody is looking’.
Of course it doesn’t change anything about the people that work there, but the people at the top now clearly have a better idea of how it works than the last ones :P
We review each other's code before it goes in. If the team says they're happy to work with it, then I've fulfilled my obligation. I am literally not paid to write the best code in the world. In fact, the company gives bonuses for fixing things, so they are in fact paying me to sneak bugs in.
What you do though is, you leave a very obscure edge case unhandled, make note of it, then just don't tell anyone. Invariably someone will hit it exactly one day before you're scheduled to go live. Then just apply the fix (make sure to procrastinate long enough to simulate working on it) and voila! The longest I've seen an edge case go unfixed in this manner has been three years.
Though truth be told I'm not smart enough to add bugs on purpose, I just sometimes notice I've failed to address an edge case and ... leave it. If it turns out to be important - I can fix it when the fix's value is maximum. Half my day is spent fixing others' unhandled edge cases, so it's nice to have one tucked away for a rainy day that you know you can fix easily.
How bizarre. I guess I can count myself lucky that I don't seem to live in that particular world.
Is the problem the incentive itself, eg. if they didn't reward fixed bugs would you write better (even just better communicated) code?
You mean money? Abso-fucking-lately I would do it.
Moral points don’t pay my mortgage.
When the external validation is monetary then you gotta do what you gotta do.
It's OK, the new validation method is AI driven, we're good.
There's a famous green text on 4chan, where a user tells he got a sysadmin job which was so boring he started crashing stuff left and right and blocking an entire office even for entire days.
By the end of the day he would plug back something and come out the "servers room" saying he fixed that and get everybody's praise. Even got him two raises in the span of 18 months.
That's how crazy it is.
I know a variation of this sort of story, where a good sysadmin/DevOps team was halved and then the problems started. The company didn't have those issues exactly because they had a good surplus of eyes to handle everything.
They realized only later the mistake.
I do not think you can trust everything you find on 4chan. Yes, dysfunctional companies exists. No, someone writing something on 4chan does not imply it ever happened.
Neither does it imply that it never happened.
You cant use it as an argument, just like you cant use a fiction book as an argument.
Parent post didn't cite it in an argument.
They shared a story.
what about someone claiming it was written on 4chan, but it was actually made up for hn. smh can't trust anyone
Which is why I stated I've read about that on 4chan and not taken is as a truth.
That being I highly believe that stuff like this has and is happening right now.
I've worked in consultancy enough to know that many job's securities revolve around solving specific issues, and if the issues don't exist job security gets lower.
It is more easy: do your job right, but do not comment, fix or somehow improve work of your colleagues. Let them fail, aknowledge failure and only then come with fix, get all praise.
Never point to the possible issues at the code reviews, retirement refinements etc.
That is naive take, you really think if you work hard on improving code, pointing out flaws, there will never be any issues?
In reality that I live, there is never enough time, if you have 3-4 team mates pumping features out, it is already impossible to prevent every problem and review every piece of code.
I don't have to be cynical about it, it just happens that issues will crop up over time and I am there to fix them and there is never enough time to prevent them up-front, because if you will try then you will never deliver anything. Ship fast and break things is maybe too far - but still shipping something beats not shipping ideal state.
We have a very good test suite, I don't think there's been a bad release in years.
We also don’t have bad releases but we had a new customer that added 100 users that used specific workflow that was working for years for other customers.
Those 100 users took performance to a crawl but we never seen it earlier because it was never used that way in that capacity.
This, but seriously:
Let your junior colleagues fail on a schedule within the margin of error on your planning, but keep a close enough eye on them so you know how to bail it out if they can’t pull it together with a little extra time/guidance.
Okay — still give good code reviews, but if you let them face-plant a little on design, etc, then they get experience. And you don’t look dumb when it turns out they were right. But you do look like a hero when they’re struggling to get across the line, and you whisk in to fix it.
This is also just good practice in general. You learn through struggle, so giving juniors a safety net (when they don't know they have it), is a good way to get them up the curve.
In high school I actually had a friend teach me this in a humorous way. In football we did sprints (the kind where you physically run) at the end. He said -- half ass the sprints. Then on the last sprint, run as fast as you can, you'll have more energy left, and when they see you running fastest you'll get all the credit. He was doing it as a joke, but lo and behold he out runs everyone on the last sprint by a lot, and the coach specifically pulls him aside as an example of how everyone else should carry themselves.
He's one of my favorite pranksters. But needless to say, the football team was not very good.
In the case of football that might actually be a good lesson: save enough energy for the end of the game so that you can play best when the other team is tired. Of course you need to play the whole game, but if you play all but the last 5 minutes working just hard enough to keep a tie game, then you can pull ahead in the end.
This only works in time limited sports though. Most real world situations are not time limited like that and so the advice doesn't apply.
On sprint training. Me and a friend figured out that if you moved your arms fast whilst not running a full sprint it gave the appearance that you were trying very hard and coach didn't give you a hard time. Like most slackers instead of using the trick once in a while we over did it and the coach soon figured it out. The result was even more sprints. Needless to say our team too wasn't too good.
That really only works if the other team is terrible relative to your own. ‘Playing to tie’ a team that is an even match (or close to an even match) results in a competitive advantage for their side.
This does work ok for long distance competitive running, since the factors at play are time and metabolic rate and is missing the strategy present in team based ball sports like football, soccer, basketball, etc.
Nice example for the (common?) inability of many managers to assess performance realistically.
Another option is to work at a consultancy where you hop aboard a new customer project every 3-6 months. Approach every customer as an opportunity to do some resume-driven development and pick a bunch of untested new technologies to experiment with. Be sure to do at least a couple of presentations to tell everyone about the hottest new things you are doing to bring value to the customers. Leave the project once it slowly starts sinking and then just keep hopping from customer to customer. You will be far away once the sea water starts coming through the windows and the non-technical people directing these projects will never figure out what you did.
I have seen that this is one of the most efficient ways to advance your career especially in larger consultancy companies with hundreds or thousands of different customers.
For maximum results, trade-off with a buddy:
Spend Q1 each crashing your respective project; spend Q2 as a fixer undoing that damage using his knowledge of his own work.
You’re both cutting edge and a proven fixer!
Reminds me of The Good, The Bad and The Ugly.
Tuco is a wanted man, Blondie brings him in and gets the reward money. Then just as Tuco is about to hang, Blondie shoots the rope and they both escape to another state ... where Tuco is also a wanted man.
Government contracting is the pathological case for that: if they haven’t been allowed to hire civil servants, they have no choice but to use a handful of big companies. The shortage of in-house expertise makes it hard to know if the work is being planned or performed correctly so a common strategy is to hire one company to oversee the work of another company. Unfortunately, since there are only a few companies in the space and job hopping is the best path for a raise or to avoid layoffs when contracts turn over, it’s rare for anyone to be very critical of past or future coworkers.
I don't think you're wrong, but god what a waste of time. Can't we just fix the actual problem?
How?
One middle ground I've found is to write tools or processes for myself or a mentee, with the intention to avoid avoidable problems. And then use the next time someone breaks things as an opportunity to advertise the preventative measure.
Frequently I end up having to do one more PR to get the tool to correct the exact problem the team experienced, so in practical terms is much less of a 'witholding' and more of a 'burning in' situation. But the final product does end up getting written in hours instead of days so while I don't get the credit I feel entitled to for foresight, I get mis-attributed with being able to solve difficult problems quickly.
Which is kinda true. I hate being caught flat footed so I'm always squirreling away fragments of a Plan C.
This is exactly what happens at many software companies. People,sometimes under pressure to meet deadlines, apply totally crazy fixes to stop one problem and create new problems.
I’ve seen it both ways. In one case dev spends 3 days refactoring code beautifully to fix the bug; ships to production only to find out that it wasn’t the only bug in this code path. Now they spend another 2 days refactoring before repeating the process and so on. Meanwhile, the company is losing money/customers/favorite vanity metric/whatever.
Or. Dev hacks a fix up, sends to prod. Sees another bug and fixes immediately. No loss in revenue/customers/favorite vanity metric/whatever. Dev creates ticket to refactor/fix code properly and adds to next sprint. PM removes from next sprint to get capacity for further resume driven management projects. Issue crops up again due to hack. Original dev is now able to do ticket originally removed from sprint.
Definitely. Like so many other things it's the spirit and intent that matter.
Maybe I'm the only one who notices people whose work "just works". Those people are solid gold.
Well, if it "just works," then it wasn't that hard to do anyway. No raise for you.
Your comment got me irrationally mad, until I picked up sarcasm. Thanks.
don't even need to half a$$ it, just use timebombs.
https://en.wikipedia.org/wiki/Time_bomb_(software)
Usually only works if you're the only dev, unless you get creative with counters like the original devs that made some nice cash fixing it all for y2k
(officially: don't do this)
The AI will analyze your codebase next year and the company will sue you if you do stuff like that...
2038 is getting closer every year...
Having people thank me for being 'on top of' a problem that I already know about because I'm the one who pushed the button that broke it... It's gross. It feels gross.
Much happier when I can fix something quickly because I have some tools or logic in the codebase already that lets me do something quickly by being covered with sanity checks so I can zip along without driving us off a cliff.
There's some people who break something, realize it, just leave it as is, and cannot comprehend why you would think they should maybe help fix it... So I'd guess the praise is for not being like that.
That's literally how promo packs work in FAANG (minus the N) and other adjacent tech companies.
Nope.
This is a slippery slope, IMHO. It can easily get you to be the next "tactical tornado" of your team (cf. John Ousterhout, A Philosophy of Software Design).
For i in employment ensure there is someone who checks in with me every month when they read this line of code, else, stop code from running so they come to me.
I don't do this for one reason - I cannot hold others to a high bar if most of my code is half-assed.
I usually write my best code to show others it's possible to write good code and ship on time, with exceptions of course (and I usually document in code why its half assed)
I have seen so many people do this wherever I have worked. In fact there is another thing that they do i.e. when management pulls in the schedule they will meekly agree to it because they already know they are going to half-a$$ their code. Now they are heroes much before as they are "yes-men" and will be targeted for promotion. Swooping in to fix things just adds to their already loaded credentials.
The ones who are honest and actually disagree are banished and their lives are made difficult. They are called all sorts of names the most important being "not a team player".
Is it important that no one knows it was the Rock Star’s fault in the beginning?
I was just considering it, but never had a chance to see it in real life.
I'm dumbfounded about all this, you spend years learning and reading books about the craft but the only topic which will matter is how you can con the game ?.. weird.
Hah, that was in my archetypal description of the fake 10xer, that somehow they are able to swoop in and fix their spaghetti code, come what may, while no one else can make sense of it:
https://news.ycombinator.com/item?id=18462325
Life and its rewards aren't perfect. Work with honest, intelligent people; genuinely do your best; your days will be much better and the odds will be with you.
This is good advice that’s well intentioned, but (sorry), it can be interpreted as elitist, and in a way that’s detrimental to the reader.
I am no way suggesting that this is the intention or belief of the parent, but while I’ve got more miles on my odometer than I’d prefer, they’ve informed me that “reasonable” is better than “intelligent.”
My god how I’ve found that working with reasonable people is so much healthier, more productive and rewarding than working with the unreasonable* intelligent folks.
*I fully grant to my current and former colleagues, friends and associates that I have been irredeemably unreasonable any number of times. Consider this a small thanks :).
There's nothing wrong with elitism as long as it leads to initiation rather than gatekeeping.
Could not agree more. Most people can be trained well to do any job required of them. What cannot be trained, and certainly at the behest of the employer is interpersonal skills.
The one situation where I would prefer someone who is intelligent at the expense of being personable is if I intended to hire 2-3 absolute weapons to be the core of a startup.
I really wish this were true, but especially in programming I don’t think this is the case.
I’ve spent years as a programming teacher. Some of my students have been among the most wonderful, enthusiastic, hard working students you can find. And yet, despite both of us working hard for a year or more, some never develop any talent whatsoever for programming. Statements like “everyone can code” can easily turn into a rod for their backs. - “Therefore if I’m not succeeding like many of the other students, it must be because I’m not trying hard enough, or there’s something wrong with me”. I don’t think this is anyone’s fault. Perpetuating the lie that all our brains have an equal capacity to program is a terribly cruel injustice. Some students would be much better served by finding another career that they can excel at. The faster they figure this out, the better.
Programming isn’t for everyone. It’s hard. Not everyone has the same capacity for it. I believe accepting that is an act of kindness.
I should have made my prior assumptions clear. The "most people" I'm referring to are CS graduates or people with IT diplomas and the percentage among those who are trainable to a capacity of competence in most companies are roughly 70%.
When I was still a uni student I used to be a CS tutor in a help desk setting and I especially remember one guy who used to come in a lot. He had a really great attitude despite being humbled by the fundamentals. He used to repeat to me: "Sometimes you have to ask for help when you need it." After that semester I didn't see him again. I also remember a particularly annoying guy who used to come in and waste my time and the other tutor's by bringing in a problem and then solving it himself within seconds of sitting down just so he could talk about how he solved it and how much he knew. The world is not a fair place.
I think that those can be trained as well. The fact is that just acknowledging the issue requires a level of self-awareness that not everyone has, and training THAT as well requires being aware of it. External input and help from a specialist or a dear friend can get the ball rolling.
If you want a concrete example backing up your stance, consider the letter sent by Hans Reiser, discussed on HN in https://news.ycombinator.com/item?id=39042626.
Elitism is literally a form of gatekeeping…
Totally get where you're coming from, but it's also kinda splitting hairs.
Being reasonable is part of being intelligent. Surrounding yourself with intelligent people doesn't necessarily mean "surround yourself with the highest IQ individuals you can find." (not saying you're saying that explicitly, just that i think you're just using a definition of intelligence that's narrower than the parent) Working well with others, understanding when one has made mistakes and being able to admit to it, understanding both the known unknowns and the unknown unknowns of a problem...these are a better mark of intelligence than a mensa membership.
Not in the slightest, those two are quite separate.
Well the parent made sort of a case for why being reasonable is part of being intelligent, and your response was to claim there was no correlation without an argument.
I happen to believe that reasonableness is part of being intelligent, by the following criteria:
1. when you are reasonable you do not make unreasonable demands that will just be troublesome and cause workflow issues because in the end they are unachievable.
2. a reasonable person will be able to determine what other people are capable of in given situations, and be able to structure things in such a way that other people can perform to best meet expectations.
3. the root of reasonable is reason, a reasonable person can be reasoned with because they possess the quality of reason, in most of the history of philosophy if you do not possess the ability to reason you are an idiot.
Alright, my argument is that parent needed to redefine what those words mean. Intelligent means having high intelligences, high IQ. That does not imply being reasonable or having emotional control. It does not imply ability to shut up when you do not know what you are talking about either.
Moreover, being able to admit mistakes (specific thing mentioned by parent) is oftentimes detrimental for you. People who do not admit them are typically rewarded, people who easily admit them punished. So, what you are looking at is "ethics even if it does not benefits me".
Yes you’re right. I guess I was mostly trying to underscore the point you’re making here by being a smidge provocative. +1.
As the GP, yes good point about reasonableness. I didn't word it well: I meant people who act intelligently; intelligence by itself certainly doesn't yield that!
If "intelligent" is taken as 'naturally superior intelligence', then I can see what you mean (and I think that idea is a egomaniacal delusion). What I mean is people who choose to act intelligently; that's quite democratic.
I know OP already responded but people who are smart, kind, and are down to earth are special. You have to keep them close, and it’s pure bliss if you find a company filled with them.
It’s life goals to be smart, kind, and down to earth. It really comes from a LOT of experience though.
I think [reasonable] falls under the heading of [honest].
Both honesty/reasonableness and intelligence are required. A family memebr is tearing her hair out about having to work with some assistants who are honest, well-intentioned, and pleasant, but who are just mentally incapable of keeping things straight — they literally screw things up and make more disorganization than they fix (fortunately, better help is supposedly on the way).
Overall, in hiring and working with people, most of the time, a warm body is definitely NOT better than nobody.
People known for being purely highly reasonable people know that they don't know everything and aren't afraid to learn and be humble and are often a net positive.
People known for being purely highly intelligent people often think they know it all and are often a net drain.
Contrary to the example set by the loudest voices in tech, "intelligent" and "reasonable" are not mutually exclusive.
In the context of one's effort valuation it seem to be a very bad advice. Unfortunately.
Intelligent people will see that you solved issues before they even happened.
So in small teams where visibility is clear those things are obvious.
It's in larger orgs where you want to climb the ladder that you get bonus points for the problems fixed but not those you prevented.
Intelligent people are in larger orgs too. They learn very quickly that promotions aren't tied to effort and anticipating problems or resolving future problems.
The intelligent, optimistic and genuinely good, helpful people who don't learn this (or refuse to compromise their good nature) end up burning out, getting performance managed or with PTSD
Right, don't work in that org. Find the org with more of the intelligent, optimistic and genuinely good, helpful people.
Some people don't want to give up the job at High Status High Pay Corp. That's the trade-off they are choosing. (Not every HSHP Corp. has to be a bad place to work, however.)
You can have intelligent people in the largest orgs, if they are good at hiring. You can have inexperienced people in small teams or companies, and it happens often.
It also really depends on your organization and how they quantify and reward impact. There are ways to get credit for preventing issues. Establish the baseline level of quality, broadly create classes of issue, define a plan to solve that class of issue and measure your impact relative to your goals at the end.
At a big company you should be able to turn that into a number - 'this kind of issue was costing us $X/quarter, and thanks to my work, it is now costing us $(X-N)/quarter, in line with my estimates.'
It's performance season so I've been giving a lot of thought to how you quantify and attribute impact especially for folks in lower-visibility roles.
Not every company is going to see it this way but that's kind of a truism. Not every company sees any one kind of impact the same way, and you have to think about that relative to your career goals. I think aligning with your manager at the start, quantifying your impact and showing your results is going to get you the recognition you deserve at any company worth working at.
This kind jaded, reductive answer reeks of sour grapes. I've seen so many adopt a transactional mentality as a sort of defense mechanism against the indifference of the universe, and in the process they unintentionally blind themselves to numerous daily examples of people just trying to do the right thing. Sure the world has problems, but if we can't see both the good and the bad then we become embittered and small, hateful to ourselves and those around us.
Well thought and expressed. Thanks.
And most people reading HN can do that.
The key is learning to identify the situations where a transactional mentality is the rational thing to do to protect yourself.
That's a tremendous amount of assumption being piled onto that one line comment.
That hasn't been my experience. I don't get to choose who I work with, my days are stressful and unfulfilling, and my career has stagnated.
Not everyone has that option, as you say. My career was in a similar dead-end situation at one point; there was no immediate solution - nothing was going to make tomorrow or next month better than today.
Finally, I accepted the solution was long-term. I made a long-term plan - to do what I really loved - and focused my attention on that, and slowly built up the resources, skills, etc. for that plan. One wonderful side effect was that the dead-end part diminished in my mind; sure, I still had to do it, it still sucked and there were some awful days, but it would pass. I didn't matter so much; those people didn't matter; it was like one of those movies where the kid knows that someday they will leave the depressing, dead-end town they are stuck in today.
I hope that helps a little in your situation! Good luck!
Sorry to hear this! At the risk of repeating what others in your life may have told you, have you considered working for a new team/company? I find it of utmost importance to find a job/career that is intrinsically fulfilling and rewarding to you. Extrinsic rewards like money are great, but you can only slog for so long for only money before you burn out.
I've personally been fortunate to have been able to form a network of a bunch of different well-connected folks so I can always choose who I get to work with in the event that companies shut down or get acquired or such and such. I feel the value of such a network does not get touted often enough.
That's such good advice.
On the same note, it's so discouraging when someone who is not/neither takes over, those people leave (one way or another), and you have to start the search again.
Or when someone is intelligent enough you don't figure out they're not honest until it's too late. <shiver>
Anyone who's been around the block a time or two will accumulate those experiences. I hope I've not been jaded by mine.
Yes.
Very true, and it may not even be malicious on their end, just avoidance and their brain rationalizing/minimizing without them realizing it.
(Otherwise known as pathological lies). To do as the saying says, and "know thyself" is difficult. People are prone to be overly harsh or generous when evaluating themselves.
But most of the time you can not chose with whom to work
Then you just have to refer to the Karma bucket. "In my/their next life..."
haha :) you are funny :)
Great approach! For the past 2 years I keep experimenting with different ways to track day to day productivity. It has helped me tremendously in assessing how I should feel at the end of the day about my work day and the amount of time I put towards work. (I tend to overwork myself significantly). It is of course a lot harder to apply this approach to individual productivity tracking across my teams. But I do now believe, based on self experiments and within smaller teams, that there is generally a lack of visibility of knowledge worker productivity. Especially within larger companies. WFH and hybrid makes this more crucial to have for both management and self assessments on the IC level.
What are some approaches have you used to measure your productivity?
However, those who choose to be dishonest are more possible to be your leaders.
I'll let you know when I find this communist utopia.
Another perspective: if you are not in a position to change the rules of the game, understand the rules of the game and play that game, not another one that you would like to play or that you think is more fair to play. Alternatively, if it is not the game you want to play, find a place where that game, or a similar one, is played.
I'm going to add: in a place where they have the bandwidth to be honest and intelligent. Sometimes good people are in bad situations. It's important to recognize this.
reality feels like there is a mix of people with different motivations. I think the paradox exist because the odds are not clear
This is a great answer. The only thing I would add is that you can take mental note when you accomplish something that anticipates and preempts future needs and challenges. It is ok to make brief mention of these things from time to time when appropriate. If nothing more it helps coworkers and bosses realize that you are putting in some thought and effort for the needs of others and to make things go more smoothly behind the scenes.
Doing this effectively requires tact. Try to mostly bring up your silent efforts casually, and be judicious about how often and in what situation you mention it. If it gets interpreted as an "I told you so," an excuse, self-importance, etc, then it will probably do more harm than good in terms of your standing with others.
Might be helpful to some degree. But won't save you if your organization's culture simply doesn't value your work or more generally doesn't care about proactive, methodical improvements that have no flash or immediate payoff.
I've done that as an hourly contractor, I fixed their 6 month problem on my first day, I hoped they would then hire me for more hours on other stuff but they were like nope, that was all we needed. They did tell some other companies that I was good at this kind of stuff but nothing came of it. My first and last day in contracting for small companies.
The trick is to do these sort of engagements as a fixed price contract. You could have probably done an assessment in a hour or two. Then you charge on what you estimate is the value to the company.
I should have, but it would have been obvious that I pulled a fast one on them as the fix was really quite simple. Instead of building a Hadoop cluster I replace a sql sub query with a bitmask. It really only took an hour to figure it out but they let me charge the whole day.
Ok, it doesn't sound that bad to me? You learned something about consulting, got paid 1600$ for one hour work, got good recommendations and now have time for another interesting opportunity.
Don't waste your time and clients money on pretending to do work that you don't. Its ethically wrong (maybe even criminal) and it sounds like you would be bored to death wasting your talent.
Don't blame the company for not wasting its money either. It did pay for the whole day and gave you good recommendations.
You could do fixed price. Or split the difference (on both sides of the estimate), which provides good incentives for both parties.
It took 3 months to get this first contract, I was trying to break into a new space. Shortly afterwards the government gave many millions of grants to all my potential customers in the country to spend on consulting with a brand new company that was spun out of the government so the entire potential market dried up overnight. The customers didn’t care it was free money. This spinout then tried to hire me where they correctly pointed out that there is now no alternative to working with them and gave me a low ball offer. I left the country instead.
The problem is it takes typically takes days or even weeks of unpaid work to find the next paid job. If you get 6 months contracts that unpaid time is easy to deal with (and much of the unpaid work is really half an hour here easy to put in after hours while in the middle of the longer contract). Thus a 1 day job needs to pay a lot more perhour than a 6 month contract as the costs to get that contract needs to be paid from than 1 day - a fair price for a 1 day job is likely in the $20,000 range.
If they were ready to have you on for 6 months, You'd still probably be one of the most honest contractors if you stretched that out to a week, or even month. It's a shame honestly isn't always rewarded proportionately to suggesting an entire rework of their infrastructure.
I really did think they’d have more work for me.
How much was it costing them?
Also, can you describe the problem? It sounds interesting.
It was $200 an hour for a total of 8 hours. They had a team of 5 working for six months on it and they planed to spend another six months on it.
The UI had options to check some categories and exclude others, this was mapped to sql with a nested query IN (…) and not in (…). I noticed that there were less than 64 categories and always will be so I figured pack the category membership into a 64bit ints and use bitwise operations. The UI query generation would map to use bitmasks instead. They didn’t even let me implement it, just agreed that it would work and took it from there. I think they were quite embarrassed by it. It ended up being 10K faster.
Even with this I would argue that fixed cost would have been the way to go. They were losing $X every day for that six months that it wasn't working. They hired you to fix it, and even if you charge them way more than your day rate you're not solely charging them for the change - you're charging them for "knowing which screw to turn".
The hardest part of getting into contracting was the guilt of feeling "I'm charging too much for a simple fix!". If the fix was so simple and the client didn't find it in those six months then maybe your assumption that the fix was simple and obvious was incorrect. Regardless, your fix provided value, and you should be compensated for the value that you provided, not just the number of hours worked.
Or worse: People get stuck quite frequently and ask you for help pretty quickly. You get everyone unstuck, but your own work falls behind and when your boss's boss asks for metrics on developers you have few points delivered and few LOC changed. Your boss tries to explain, but your head is the one that rolls next when layoffs happen.
Great workers know when they can help others without falling behind themselves. Or have good communication skills to explain what they were doing.
Or are good at communicating and don't need to do anything because they can just lie.
Well at some point you expect some artifacts to show up, even if not just code, design docs, RFCs, code reviews, something. If there truly is nothing I don't think that's a viable way to contribute either. Sure one can "only help others" but all good engineers I've worked with can help others and still do their main tasks, and if they get to a point where they are a point of reference for everyone else and everyone else gets blocked without them this is a high priority thing to fix in the team, probably one of the highest priority things to fix.
I've seen management let bad engineers re-do projects from scratch 2 or 3 times. Because they blame bad requirements, poor choice of programming language, and whatever on all the issues that the project has.
They eventually acquire enough experience to be able to produce something that sort-of works around the 3rd attempt.
But thanks to how good they are at communicating, are considered by management to be good engineers.
While doing a working project at the 1st try, without using this week's new framework and so on isn't as valued.
I mean at some point we just get generic enough that we have to agree on the ultimate fact that at least somewhere in the management chain someone has to actually know enough engineering to recognize good work from just running around in circles. I agree what you say happens but if good work isn't consistently recognized and you think you've worked on your comms, there's not much else to do really.
Unfortunately, this can cut the other way as well where you can have someone incompetent override competence or not even involved people further down which is common during large layoffs
which is why one should not self-sacrifice. It garners no reward for one. Secondly, if the boss doesn't realize how much of an enabler you are, then it's time to start looking for a new job before the layoffs even starts as a thought in the boss' head.
As a senior engineer I've learned that is my job. I take on much less work than anyone else because I know I will be called on to solve those problems.
This is in fact known in management literature: assign your best people to the least important project. That way the second best can grow to become the best, while the best are always free to help out if your fourth most important project has issues - it isn't a big deal if your least important project doesn't get done on time and if they manage to finish it so much the better. (your best are also free should sales discover a short window where a quick feature can bring in a large sale - though this is obviously easy to abuse)
I think an underrated way to deal with this phenomenon is proper self-marketing. Talk endlessly what you have done to prevent catastrophe. Describe the avoided catastrophes vividly so that people get a clear picture.
Self-promotion is incredibly difficult for some people.
fair, but it's better to point out the solution:
"learn how to be better at self promotion"
Do you have another solution to this issue?
Find someone to champion your work. Also, if you aren't already, start praising others' work in the hopes that they will reciprocate.
The caveat to this is to make sure hope doesn't turn into expectation. Expectation can turn into sourness when the others don't reciprocate. Also, some people feel when you do something just because you expect something in exchange, and that tends to make them not reciprocate.
And the people it comes easy to tend to be labeled as narcissistic, resulting in animosity. It's a double-edge sword.
Sometimes you can't quantify actual avoided things. For example you can't prove that a regulation stove actually prevented what could have been a kitchen fire.
Better yet, be in a work culture where people happily give credit to and praise others.
One approach to showing value is as follows:
1.) Create a spreadsheet with all of the features of your group's/company's products(s) listed in rows.
2.) Create a column for every team member in the group and highlight the lead developers for each feature.
3.) Then ask each team member to add a checkmark in their column for every feature for which they would be willing to be on hook for 24x7 triage pager duty.
Over the long term - the most valuable contributor(s) on the team will be the one(s) with the most checkmarks next to the features they led development on (i.e. they write understandable code and document well) combined with the most checkmarked rows in their column (i.e. they proactively seek to understand other peoples codebases).
Couldn't there then be a risk that now some people want to avoid higher risk projects, and just build the simpler features, and get more checkmarks
What if the table in fact shows which people are best at dodging the hard work
Combined with showing who has the most friends in the office (giving checkmarks to features built by one's friends)
Agree that there is probably room for improvement to such a process and still a great amount of subjectivity required and a presumption of good faith participants who aren't actively gaming the numbers.
In general though even if some developers are gravitating to only projects that are simple/trivial, those wouldn't necessarily be differentiators because such projects would have checkmarks by other developers as well.
Also it can help to have the Product Management team rank the features in terms of strategic importance and criticality to the functioning of the company.
I'd say the biggest benefit of such a spreadsheet is to provide visibility to leadership about the bus factor of the team. Too often the critical projects are really only maintained by a few team members. There's no incentive for new team members to learn the "legacy" projects versus creating their own pet projec. Then the inevitable RIF or transition happens and the lack of long term support becomes an issue.
This is kinda a well known phenomena in medicine[0]. Same with lawyers. I'm just reminded of this scene from Silicon Valley[1]. It's messed up and why everyone needs to be very careful with metrics and remember that metrics are only guides, not targets.
[0] https://www.theguardian.com/society/2016/jan/29/doctors-avoi...
[1] https://www.youtube.com/watch?v=5sTbjO3eI_0
Kind of ignores the risk externalities have on projects no? If you tried your best to deliver something, but another teams priorities changed, or the business/market shifted, or the project got bogged down in planning/politics outside of that team members control, or if the team needs to integrate with a system (external or legacy) that’s out of their control and is notoriously flaky and torpedoes their velocity and morale.
Modern “agile” systems don’t work well with you. They want you to deliver X points of features in 2 weeks. Sounds like you’d be better off as a consultant? But then you need to do all that biz dev stuff! Or find a great CTO to work for (or be the CTO) who understands. You only need one job.
I always laughed about those X points targets.
In the end people ended up simply inflating the value of stories.
In my previous team I had to argue with a person that 2 story points to add translation keys were ridiculous.
To which the EM argued that 2 was okay because he also needed to write few accompanying tests.
I have nightmares thinking about this stuff. It was literally quicker to add the translation keys and that pointless test than to even discuss and vote the story and debate the points..
Company has a goal for 5% increased velocity, what they want: teams get more efficient at pushing stories through to completion, what they get: teams just estimate 5% higher.
Goodheart strikes again.
And large companies will probably never learn, because at some critical mass it becomes less important to ship than it does to tell people how cool the product you want to ship is. Devs don't get paid for shipping outside of a short surge of stocks (which they proceed to not be able to take advantadge of because potential insider trading). Heck, some industries just let you go after you ship.
The solution is to be a shameless self-marketer of all the work you do.
This does seem like the antithesis of useful, productive engineering work.
Even spending 5% of your time self marketing has massive dividends on your career vs spending 0% of your time. The virtues of your skills aren't self evident.
True. But when you're a smart kid and you get auto-noticed in school with no effort on your own part (aside from the academic effort!), maybe it's time to add a course like "Creating Your Own P.R... And Believing It Too" into the curriculum.
See also: The Locksmith's Paradox - https://medium.com/@pk.patrick.kelly/the-locksmith-paradox-6...
"When you do things right, people won't be sure you've done anything at all."
There is a quote about the job being free but you are just paying for the 20 years experience.
The opposite is the lemons market. It is why getting some wet building work done that is actually waterproof is a fucking dark art. Even the pros hiring pros get fucked because the “is waterproof” part is invisible.
It's possible to phrase the value proposition to align with what you want to deliver.
You can phrase it as "I help teams deliver on time by making sure they don't get stuck." Or "I increase a team's velocity by preventing mistakes that stop work."
It's how you sell yourself and then, how you tell the story of what happened. Being able to tell a compelling story is important.
The issue is how do you provide the evidence for that. In other terms, how do you differentiate your effect from that of Homer Simpson's Bear Patrol and Lisa's tiger-repellant rock [0]?
[0] https://www.youtube.com/watch?v=xSVqLHghLpw
It is really really hard to do that in a context where its not embedded in either work culture or appreciated by your lead.
It is easy enough to 'measure', a good lead would value how you benefit the team and enhance your co-workers productivity and seek to understand this part of your contribution. He or she could just ask your peers, it is not science.
If this does not apply to your situation at all, then leave because no amount of evidence will ever force your superiors to accept it, it will only antagonize them. You do need to make your case sometimes, but the level of proof should not be that high, you need some trust in order to function as an org even if it is exploited sometimes.
That's not true, your value is clear just subtle.
If your solutions tend to be more stable from the get-go, it will get noticed at most places. There are always going to be that odd company where that's not true, but in my experience, simply doing higher quality work gets appreciated.
Well it's not 'clear' when every promotional process I've encountered doesn't reward such forward-thinking preventative maintenance compared to someone who postures themselves as a hero going around putting out manufactured fires.
cynicism is easy.
Don't help anyone too much until a manager asks you to help. If someone is really pressing help, tell them that you are busy, but they can ask the manager to tell me to help. This way, you always get maximum credit. Every single manager that I have worked for claimed they were "different" and did not suffer from it. Usually, I give them many chances to see my value. After I am overlooked too many times, I revert to this selfish mode explained previously. In all such cases, my recognition from the manager improves. Sigh.
I don't know about that, you might want to receive some help yourself right? In some cases it might be better to work at a place that values cooperation more than competition. You sound like you would thrive better at such a work culture.
However, often leads just need something in order to know what you are doing, especially managers that don't really closely work together with their team. It can help to just mention it to your lead. Because it is easy to see what you done, but not how you have helped somebody. Just mention it in whatever recurrent meeting you have. And if helping out takes more time, I'd say it is only fair to have whoever is responsible for delivery involved in prioritization, because then it is at the detriment of whatever you are working on, which might be more important.
Often, it takes less than 15-30 minutes to help somebody get unstuck. I wouldn't enjoy working at a place where people would refuse to help me with something like that, or I would be so pressed to achieve things that I can't spare 30 minutes out of my day to help someone out.
The educational system i was put through was set up to teach that outcome scales more or less linearly with effort and time. The first lesson after graduation is that this is not so. Increased effort and time most likely primarily yields more effort and time being expected of you with lagging compensation.
Value and opportunity are chaotic processes in effort and time.
All we can do is to try to maintain the levels of workload such that we have clarity of mind to seize opportunities when they reveal themselves. Honest and balanced colleagues help there, but that is ultimately a missions for yourself only.
Helpful partners and friends who try to find opportunities most definitely can help (and in unbalanced relationships, can do it all), like the market research team at a startup.
Security/insurance paradox: when you do some jobs perfectly, it feels like you're doing nothing. Sadly, the only (ethical) way to counter such narrative is to spend a good amount of your time making reports showing how well everything is working and how it's not something you can just operate autonomously without disaster (to answer the inevitable "well the job is done, what are we paying you for?)
How do you get people to read such reports?
Your managers need to understand the work that you're doing. When I review someone's code and I see them fixing something potentially catastrophic but that never happened, I definitely remember that and congratulate them. When I see a team member ask a question that makes me or the team rethink implementation strategy. That is very valuable. When a code reviewer finds a potentially critical bug in my code, or just a nuanced bug, I remember and appreciate. Because I fully understand that those are potential p0 or p1 issues.
And great engineers are consistently good at this type of passive, keeping the lights on work, but it does not reflect on their quantifiable work, so orgs do not include this in performance reviews. It is up to your manager to recognize this and advocate for you.
Even when they do recognize it it’s unlikely to be weighed the same as the engineer who heroically stays up all night fixing a critical issue.
Especially with many organizations focused on data or metrics for performance reviews or promotions being able to say you fixed an outage that was costing $X million per hour comes across much better than a vague counterfactual notion that your high code quality prevented Y such outages in the first place.
I am dealing with similar. I can provide solutions 10x faster than anyone. Everyone looks up to me, and are dependent on me for solutions to everything.
However it is clear that noone is improving and that the process IS me.
What I've been trying to do is make sure that my scope and role is fully clarified, and any "extra" activity that I perform is documented and flagged. Anything that becomes a "common" activity implies a missing part of the process - be it a role that is missing, or a skillset that is lacking.
Before you start thinking that it is pretentious or self serving, it's perhaps the opposite - you owe it to the process and the team/business/organisation to make them see what you're putting in, else they fail to find the gap.
It's not paid off yet, but hopefully it will yield results within the next 6 months.
I do hope you’re recognized.
One piece of advice, I found myself this person once - it’s the road to burn out city to be the critical path answer to everyone on the team’s challenges. You sound like someone who is very generous with your time and support. In my experience once people have found a critical path there is no point at which they “stop.” This isn’t because they’re trying to hurt you, they’ve just found the answer so to them it doesn’t seem wrong.
It will take even more of your time, but you ought to consider practicing giving less answers and asking more questions of those who seek your help to try to help them unpack the issues themselves. It will feel more tiring at first, but you’ll gradually help the others learn and also create a small bit of friction that will encourage them to try their own solution or two before seeking you.
Management asks are separate/ they can actually reward you with compensation and promotions for this extra work. But the team asking for help won’t stop when you get more comp unless you start teaching that you’re not the answer.
My anecdata is to be vocal about your works, like having short knowledge sharing sessions to show it to your team/manager. Don’t be afraid of little things, your team would learn something from it.
Agreed but selling your work is an art. Sometimes folks try and sell it so often that it makes me wonder if they are really getting anything done.
Having a boss that loves to sell his team's work to his boss also works wonders.
Value is in the eye of the beholder is what I discover. Got to play the game.
If you are an IC, you play it up with the right people. Those right people fight for you instead of the other way around. When the “we need to cut the fat” talks come around, suddenly you are not on the chopping block. But that also assumes the people fighting for you are not on the chopping block.
Soft skills across the industry is highly under valued. Knowing when to pick a fight or hold your tongue is also important (ie, reading the room).
I hate it but you got to do what you got to do to get that cheddar
Quit your job and work for yourself.
If they don't pay you to fix those problems then don't fix those problems until they start paying you for it. You'll have to deal with the psychological torture of watching things go horribly wrong. So it's up to you how to deal with it.
It’s dealt with by knowledgeable managers that recognize and award your skill according.
If you don’t have such a manager, perhaps find them. Or become them.
Your value gets revealed in your absence. To leave and the company tanks- is a expression of the value shadow you throw.
An adjacent problem is how to know that you've really prevented issues/others from getting stuck with whatever you were working on. No system is perfect and engineering is also an art of knowing when to skip which corners. If nobody sees value in your preventive work then maybe there isn't any?
Depending on your context, if you are at the top of the foodchain as described here, minus the recognition for it, one option is to simply add grist to the mill. Creative problems you can leave for the team to sweat and for you to come and slamdunk.
Time.
Your value will emerge with time.
Just keep calm and carry on.
Good and bad times will do their usual sinusoidal dance but your slope will be pointing up if you zoom out.
There will be unfair promotions outside of your team, reductions or freezes in yours (companies have tendency to throw people at problems = inflate dysfunctional parts; if you have well performing team - they won’t let you grow short term; just wait it out, they will eventually).
Work for yourself. The fewer bugs you create, the faster you iterate. And the faster you iterate, the better your chances of finding product/market fit.
Think of it this way: when a principal engineer is hired from outside the company, what is the forecasted value of that engineer? How was that determined, since they have yet to do anything? If Andrej Karpathy joined your AI startup, people won't say this person has unclear value. It's because of the impact they've had previously and what other people have to say about that person. Value is your reputation.
The problem may be in the relative value contribution. A senior eng that rights a project on 3 days which had been going on for 6 months is incredibly high value/he work.
A Sr eng who spends 6 months coaching a junior engineer has much lower value/hour.
I hate to ask the obvious question, but maybe your job title is wrong?
At least you're likely not a person who works at a freelance agency where you purposefully do poor work or break things, so then you can bill more time "fixing" the problem - until the client gets the wiser at least, or there's another more oblivious client you can grift much more easily who doesn't know better?
The above seems to be how the top billing agencies on Upwork function, with fewer staff you can grift more people because the hours being billed aren't honest, and so you can have more clients and reach the "top" earnings position faster and more easily; with the same practices existing at least on one of the two prior platforms before oDesk and Elance merged.
Unfortunately one of the best ways is to toot your own horn. If you work with good folks, it will be appreciated for you to present a tricky problem you solved, or to present a design you put forth that would save everyone time.
Play to your leadership. Exceptions apply, but female bosses tend to appreciate averted crises more than male ones.
If your boss is male, let shit fail. You'll get hero points for responding to incidents.
A female boss will quickly suspect incompetence if things keep breaking (i.e. see how far half-assed DIY home repairs get you with your wife before she loses patience with you). Your hero points will come from mitigation.
This paradox comes up a lot in security. IME in this particular field the gender stuff is less relevant since everyone is paranoid. But when we run stuff up to C-levels, it's only the female execs and lawyers that really stop to consider possible issues-- the men just dismiss everything until it happens.
Well, a problem that hasn't happened yet is a risk, right? So you can apply risk assessment math to it:
Value = Estimated risk of the problem occurring * Cost if it happens
Good leaders. Or, worse: Consultants. That is, it takes either an outsider or a self-critical leader to affect change? Those doing the planning are always optimistic [0] about their decisions, processes, evaluations, and progress. https://en.wikipedia.org/wiki/Planning_fallacy
[0] not always bad: https://en.wikipedia.org/wiki/Hiding_hand_principle
The problem with doing something right the first time is nobody appreciates how hard it was.
If the headline were actually true...
Agree with your point, solving problems gets you points, avoiding them doesn't. My cynical view of the headline is, that a lot of people do get credit for solving problems that never really existed, simply either fabricating (intentionally or not) easy to solve problems or vastly overblowing problems they just happen to have a "solution" for.
If ypu successfully fight a fire, you are a hero, if you prevent fires it is just normal and nothing special.
Really good bosses make up for this. They push teamwork and collaboration, but at the same time they know the details of what every individual is doing and how they each contribute to the whole, and they can ~accurately compensate/promote/terminate. This prevents demoralization of team members. Team members psychologically need to be recognized for their individual contribution.
These bosses tend to be competent ICs who became team leads, they are best positioned to judge the ICs they manage because they themselves are masters at the craft.
Having a carefree life of no technical debt is not its own reward? Firefighting is stressful :)
Other problem on similar spectrum, when you have a good performance and your boss notice you, their expectations can increase and increase until you unable to deliver, then suddenly you're bad.
It's mostly a question of finding organizations that recognize and value that sort of contribution. It also helps if you track who you work with and what you help out on every day, so when review time comes along you can list not only your 'regular job' wins, but all the wins you helped teammates get.
Go into consulting.
1. Donald Rumsfeld was right when he said “A’s hire A’s, B’s hire C’s”. 2. Environments where people can’t tell the difference between 1x and 1000x engineers lead to performative work and arsonist firefighting. 3. When folks in your organization can’t recognize (and reward) high impact, it’s probably time to get out, largely due to the first two points.
Presumably that's not -all- you do? Or did you sit around for 3 months going quietly mad waiting?
Assuming you are doing other things make sure that is visible.
Also why did your colleague wait 3 months to ask you for help? A couple days would be OK, a month would be crazy. You should poll them and not wait for an interrupt.
Think that it is the management's fault if they misjudge you. They have the chance to judge you correctly, yet they misjudge you. Their fault entirely.
No one knows your true value but yourself. Move accordingly.
Me neither. And I've had a similar issue with the first case. You do your work and help someone finish theirs, people still think you only did you job (even if people do credit you, you still have normal impact).
The psychology of workplace is quite subtle and almost backward. There's a motto "squeaky wheel gets the grease" .. to the point I'm thinking of trying some games. Like designers who produce crappy options and one good to toy with higher ups thinking they decide. There are some ideas in that vein so you flip the relationship and benefit from it instead of bleeding.
The lean mantra "Make work visible" is not only good at an organization level but also as professional feedback level. Although if used openly for promotions it will cease to be a good metric.
I think there’s a “this is life” aspect and a “this organization is immature aspect”.
A mature organization respects the process that prevents getting stuck for 3 months. But… they may be more stable and less nimble. Boring orgs don’t like heroes.
I went on holiday for 2 weeks. Came back to find the team had spent a week on a 5 minute task that had been clearly documented. Nothing had moved forward. Spoke to my manager that day and from that day forward it was clear what my value was. Eventually lead the team and helped the team develop processes and initiative so they could be productive in my absence.
Up until that point I knew my value but I don’t think anyone else quite got it.
It's always worth making some effort at promoting your value; no one else is likely to do it for you.