Oh boy, been there. This is Conway’s law in action - they are about to go and create the same code soup again - because the org won’t have changed. Having been in the same boat my lesson is positive engineering initiatives can only come top down - through some very senior champion. You can’t change an organisation from the bottom up - and it’s the organisation that produces the code base.
Its interesting because, although I don't know the linkedin codebases nor worked at linkedin, I've seen a few codebases that sound scarily similar and am familiar with a few companies that sound organisationally and politically scarily similar.
And I generally advocate 'finger gun' approaches.
Finger gun rewrites can be implemented in a good way. Often when you have n clients trying to do the same thing, one of those can be used as a basis for serving the other platforms. Or if you start over, you can do it in a good clean quick concise way.
The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)
The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.
Would be really nice to hear the symmetrical interview from a finger-gunner.
The general success ingredient is putting a small team of veterans on making the new system.
I don't disagree with this approach. But one thing I find is:
Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.
But it's always not the best choice.... and while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away. And it is oh so easy to start projects and make decisions if you never see the outcome / have to deal with the fallout.
Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.
This is very closely related to why these veteran teams end up successful. Existing concepts, tools and approaches should be the default; if your company uses Python with MySQL you better have a really, really good reason for your new project to use Rust and CockroachDB.
while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away.
I disagree with this. The veterans are veterans because they've already spent time maintaining the existing systems. They've handled oncall, major bugs, and seen all kinds of stupid shit that happens with software. They'll get pulled onto other projects eventually, sure, but they have a higher chance of getting a system into production in front of users, IMO.
I disagree. There's no magic to being on call that means you understand something well enough. That's simply not automatic.
Following through with something from start to "finish" is far far more valuable than fixing one off bugs and then doing something else, and then something else.
If anything I think it can lead to a lot of poor assumptions based on very skewed experience.
Depends on the nature of your oncall. If your oncall is just following documented restart procedures when your app gets stuck, you won't learn much. If your oncall involves deep investigation into your system and how it interacts with other components (services, databases) etc you'll learn a lot.
per some sibling comments though, years of experience != veteran, to be clear. There's certainly people who have been working for 10+ years, sometimes even at the same company, and don't really understand what they're working on.
Most software is never finished so I have no idea what arbitrary point you’re deciding they need to see it through before qualifying here.
I think the implication is they’ve seen all problems before and can engineer their new system to solve them.
Some changes are better than others. If your product works in python/MySQL then rust probably isn't a good fit. Maybe Ruby or PostgresSQL would be better fits.
If your main product is C or C++ then you should be learning about things like rust. Maybe you decide modern C++ is good enough, but if so you should be working with the C++ committee to push safe C++ (as a C++ developer I'm hoping this comes in C++26... I'm looking at how I can put rust into my code for now).
The point is you should know what the good next steps to investigate are and what are not. Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing. Both tools have other competitors that you perhaps should be looking at though (Rust - ada, C++. Python: ruby, go)
But it's always not the best choice....
I think it comes down to the personality of the veteran. If they're the "i've seen some shit, so I'll 'architect' the fuck out of this to handle any type of change" type veterans, then it's likely to fail. But if they take their knowledge of how the system has grown over time and architect to handle the types of changes they've seen, then there is likely a greater chance of success.
Personally, I think a having some junior and mid-level engineers on the rewrite is good because it's a good way to see how the system is all put together (as opposed to just one area) and they'll be able to have conversations with the veterans about how the system has evolved as it grew and that is valuable information.
Meh. I’ve met brilliant juniors and worthless veterans. I think it’s more important to recognize brilliance in people… That spark of hard work combined with vision and creativity.
Yeap, good point, tenure != expertise.
If I could edit the post I'd insert a qualifying "expert" before veteran, where the expert veteran meets my definition of the 10x developer :D
Any veterans who aren't gonna get the project done quickly _and_ properly aren't the kind of veterans I have in mind.
putting a small team of veterans on making the new system
I talked to someone who was leaving his job because he did this in order to meet a critical date for a product launch. The fact that his group of 3-4 people did what the company could not in a year ruffled so many feathers he said the board told the entirety of engineering that under no circumstance would anyone be allowed to dictate any detail of future projects.
Yeah, I've been these "cool Skunkworks" teams and I literally made no more money, maybe got a $5k spot bonus (super rare), and burned myself out over a 6mo launch. Spent my entire thanksgiving week on-call putting out fires, literally every hour, while everyone else at the 800 person company was on vacation. Not a single executive or product manager involved in launching was there or checked in on things. Some moron (I hope he reads this, my nicks not anon) let us launch without connecting our credit card fraud system so we (ops, me) got slammed for an entire week with fraud server launches. Jason, you're a moron, and that's an understatement.
It's like being paid in exposure. We'll pay you in cool tech and learning!
Then you see the 20 go devs around you making the same money writing grpc services 8 hours a day home at 5 relaxing.
I'm not really sure how this relates. The story I was told had the guy working as the lead with several less experienced but still competent programmers for around 2 months. There was no crunch, burnout, or anything of the like.
You're being downvoted because most devs are blue collar and hate the idea of passionate talented devs doing something amazing quickly and effectively. Any anecdotes of small elite team success must be crushed.
If you are working more than 8 hours a day you have nobody to blame but yourself I think. Especially in a situation like that.
To me it seems crazy that a board would be getting their hands dirty on engineering "specifics". I would think you would have engineering level leadership making those calls and the board would either trust that person or should move on.
Anyway:
I will see that I always worry about super teams because super teams tend to get free rein, so of course they can be faster and more successful. But sometimes that's about the bureaucracy and not even the team members.
I've started projects with a team and due to time factors decided "I'll just take all of this." and it worked out great. That's not because I'm amazing... it's just the old adage about "What one programmer can do in one month, two programmers can do in two months".
Unfortunately, I've seen places where these super teams / people are setup and endlessly praised and it goes on and on and it becomes very clear it isn't them, it's the structure.
let me tell you a simple story
1. team A developed some really nice looking graphs from customer data we already collected
2. a member of team A pointed out he is colorblind and the palette did not work for him
3. product manager took the time to research this and figure out how we might make the palette configurable for those who are colorblind
4. CEO got wind of this and called everyone involved into a meeting. The decision of the CEO was no one would ever use colors for any data presentation. The only allowable color was a shade of gray
I'd like to point out that using only a shade of gray for "colors" effectively makes the entire planet colorblind
As someone who is colorblind I find shades of gray better than the colors I often see. OTOH, I can see most colors just fine, it is just that some colors most people think are very different look the same to me. There are a handful of people on earth who cannot see colors at all.
The article is not very specific so I can't really tell one way or other
5 year plan is sounding really bad, though
Yeap! We can't predict the future but we can all be very surprised if whatever tech is picked today is still a good fit, or even still exists, in 5 years. Especially in the javascript space.
As a sibling mentioned, this isn't really true in the JS space anymore. The JS frameworks and react/vue have pretty much won, so now you one of those UI kits and whatever build system/ancillary tooling your framework says you use. There might be churn under the covers, but the framework abstracts that away for you (though occasional migration issues are bound to come up with big changes).
it's always quiet before a storm :) It'll be fun to fast forward five years and see whether there wasn't some big disruptive change and everyone is now using rust++ for client dev or something? :D
This is becoming less true than it once was. React has been ‘this month’s JS framework’ for 10 years now.
This is a reasonable take, and I mostly share it. None of us were especially happy about that plan! However, and I alluded to this on the episode, it wasn’t actually a five year plan to get to a very specific endpoint. It was, instead a 3–5-year plan to make steady incremental changes to our entire system in ways that meant that if we realized our end destination needed to change (e.g. from React to Svelte or something not even built yet) 2 years along, that was totally fine.
It's not clear to me what ended up happening. Did they start a rewrite from scratch in React?
The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.
Most places I've worked at, this ends up being the case because the most experienced are needed to fire fight and to keep things running.
(...) because the most experienced are needed to fire fight and to keep things running.
By that account the most experienced started out as the least experienced who built a mess, accrued more technical debt, and built up their experience keeping their ball of mud from crumbling.
They are also more resistent to change because this underlines all the crap they did. It's their baby, and their reputation is pinned on how successful it is.
New is not necessarily better, but old and stable also doesn't mean fine engineering.
started out as the least experienced who built a mess, accrued more technical debt
THIS
"experienced doesnt always mean good. This is why you build teams.
The CFO is your enemy as an engineer.
Stop describing my workplace
Stop being the least experienced!!! (by tenure alone)
I'm also generally a finger-gunner. I don't identify with the stuff OP was criticizing though; in particular I want people to poke holes in the plan: those are usually requirements we need to meet or at least talk about why we don't need to meet them. But yeah, other than that, I generally think it's easier to rewrite things than it is to understand the intricacies of a gazillion-line software project. That probably strikes a lot of HNers as problematic (this is why we keep getting new JS frameworks every day! this is why we keep reinventing email! this is exactly why we keep building gazillion-line monsters!), but I don't really mean it like that, I mean it more as a manifestation of Gall's Law [0].
To me, it's all about energy. If the energy is behind "fuck it, rewrite it", then that's what you should do. If the energy is behind, "preserve the valuable IP in the existing code" then that's what you should do. Actually I don't think it's even "should"; I think it's "will".
Would be really nice to hear the symmetrical interview from a finger-gunner.
I found the initial framing of like "expediency vs. the right thing" not quite right. My read was that Chris just found fault with the finger-gunners' bad engineering. Here's a really good excerpt:
---
Why doesn’t Code Review just solve this? That was a direct quote from Jim. And my answer was, well it didn’t, and it won’t again in the future because people are going to make mistakes and just be better. Again, it’s not an answer, because what happens when it’s some junior on the rotation who thinks, yeah, that seems reasonable, and this PR was made by a very senior SRE?
Why am I going to question whether that value is reasonable? Like, yeah, there’s a bunch of boxes. That seems fine. Those kinds of things happen. And to that question of engineering systems, one of the things I think about a lot is, does our system only work or does this process only work? Or does this tool only succeed?
If I’m acting like a senior engineer, on his or her very best day or, or does it work if I’m a super junior engineer who’s having a bad day and we really want our systems to be workable for the latter case. And that helps all of us because sometimes, even though I’m a very senior engineer, sometimes I have bad days.
Sometimes my brain doesn’t feel like it’s working at all. Does the system still support me on those days or does it punish me? I really would like to not be punished.
---
I'm gonna try to be polite here, but Jim's "why doesn't Code Review just solve this" take here is shockingly naive to me. If he doesn't know that defects slip by even the best engineers with the best code review processes, I have to question his experience and judgement. Chris' response of "well it didn't" is all that needs to be said: we empirically can't rely on this system. Failing to recognize that is malpractice. The length Chris goes to to explain why it's malpractice speaks to his professionalism and patience, but yeah it's clear he was in a situation where someone pretty incompetent was now calling the shots.
[0]: https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
This might be No True Scotsman, but it doesn't sound like you are a "finger gunner" after all.
After (fairly) long time in the industry, it seems to come down to:
- understanding what one's clients need (not what they ask for)
- understanding what's in the realm of the possible (technically, politically, organizationally)
- understanding what are the possible futures (good, great, and abject disaster)
If, after answering all those questions, one still advocating for a rewrite, you've shown the maturity needed to undertake the effort.
"Finger-gunning" is usually related to skipping those steps, not the ultimate decision.
(edit for formatting)
Fair! Yeah I mean, I will say I don't need a complete plan, but unknowns make me nervous, and maybe a true finger-gunner never doubts their ability to figure it out.
I have never heard of a finger gun rewrite - what is that?
From the podcast:
It was completely what another colleague of mine once described as being in finger guns mode, meaning like, yeah, yeah, yeah, this is going to be awesome, man, kind of finger gunning at each other without answering any of the kinds of questions about what does it look to like to operate this when we’re trying to support hundreds and hundreds of engineers…
Gotcha thanks.
Tbh this was kind of my conclusion as someone new to the field (23). In technical consulting, I feel like this phenomenon is even more pronounced since A. most technical consultants don’t get the time to become domain experts in their client’s business, B. most technical consultants aren’t entirely technical experts (consultants first, developers second), and C. consulting management has a vested interested in keeping things this way because they don’t have the tools to make things different. Maybe this doesn’t happen as much in boutique consulting firms but I wouldn’t know (hire me if that’s the case).
It’s given me an idea for a two-way portal where on one side, technical developers join the platform, self-organize into independent, transparent teams (not just developers but any role in development e.g. UX designer, project manager, etc) and on the other, companies join, post their potentially cross-functional work requests which are then distributed to the relevant teams. Another important concept is that these instances can be federated(?) so existing consulting firms can integrate this new kind of work allocation into their pipelines (only their consultants/their invited companies). There are a bunch other concepts I’ve thought through but that’s where I’m at now.
I feel that a system like this provides better value to a company vs. traditional consulting. First, the responsibility of team formation, networking, and allocation is moved to the service. These three things are hard for humans to do themselves and are responsible for so much consulting admin bloat/inefficiency and companies have no choice but to play. Second, it in theory improves the quality of work teams. With this focus on smaller, self-organized teams and a UX that allows for transparency I.e. companies can view teams, their members, and track record, companies know what they’ll get and not need to roll the dice to see what available resource gets assigned to them. Moreover, I’ve read that consultancies have a tendency to throw certain resources on a large work item where they exist to just to cash in budget and write bad code. This problem is completely mitigated with this approach. Finally, it’s a decentralized work environment (I think). Besides OSS, I don’t know what other platform allows for developers to have so much autonomy and self-determination. You could argue sites like Upwork or Freelancer but these platforms seem more focused on small, individual jobs rather than larger work items which justify having larger, more competent teams.
Without negating your idea, I’d add that as a (non-IT) engineer of some 15 years, I’ve found that my first 8-10 years was building my technical capabilities, however beyond that it was really about learning how to navigate (at increasingly more complex levels) the business systems and fit the technical to the business.
Being a principal engineer, my major value is not in my technical capacity, but rather my ability to shape the technical solution to the best fit for the business - be it to do with existing tooling, processes, systems, etc. Though I can do the technical solutions (probably more efficiently, too), my value is much stronger when I steer multiple junior engineers doing the technical work whilst setting the guideposts so they align the solution to what best fits the business.
I would therefore suggest that one major hurdle would be that these self-organising teams might not be able to bring to bear their full capacity insofar as they are not familiar with the businesses to which they pair. Though you can sometimes see parallels between company processes, understanding the nuance and detail that makes them different is where so many consultancy projects fall over. Think about the canonical meme of a team of consultants coming in and spending millions developing a solution that turns out to be a lemon. It’s not for their lack of capability, it’s that it’s devilishly hard to really, truly fully understand a business’s processes, systems and cruft in a sufficient level of detail to truly shape a solution such that it remains maintainable and fit for purpose in a long-term sense.
That’s the real reason why veteran employees are worth their weight in gold. It’s that they are the custodians of the institutional memory and can bring that forward when it is needed.
This of course has the important counterpoint of the necessity of seeding new ideas into a business, and that’s an entirely different essay in itself, however suffice it to say that new ideas need to be integrated in a sustainable and controlled fashion - protected and nurtured from being driven out by the existing ways, but also moulded to integration where it’s necessary. Failure to do either is a speedrun to pain.
Edit to add: If you think that IT is somehow immune from this relative to traditional meatspace engineering disciplines, I’d say two things: 1) tech still goes through this, just sometimes operates on different timescales, and 2) go see if a COBOL greybeard collecting FU consultant wages to maintain old OT systems agrees with you.
There’s definitely a middle ground between ‘we need to know how this plan will overcome every obstacle we will encounter before we even begin’ and ‘there won’t be any obstacles what do you mean?’
I always want to have a sense that the plan gives us realistic options for dealing with the obstacles. And that doesn’t just mean technical options - we’ll need to have the people to tackle it too. Eg it’s not great if the plan involves a bunch of teams running servers, and while technically there are obviously ways for all those teams to operate and scale those servers, if they don’t have the time or skills to do so then that option of having the teams run stuff is not really realistic.
But on the other hand planning a careful course that navigates round every obstacle is just as bad because
1) the obstacles will likely have moved by the time you get there (eg we spend ages figuring out how teams on legacy JS will be affected by the project only to find by the time we get there everyone has already moved to typescript and all that prep was YAGNI).
And 2) there are probably obstacles right in the path you’ve carefully chosen that you don’t know about yet and when you hit them your plan will grind to a halt because there was only one way planned and this was it.
We’re reading a cartoon podcast description of a complex architectural debate here though, so who knows which of these straw men was actually close to what was happening in LinkedIn.
This is a great summary. Obviously I bring my own perspective to this, because, well… it happened to me, and I “lost”. But my read was that the folks I disagreed with and ultimately parted ways with were falling into the bucket you label ‘there won’t be any obstacles what do you mean?’.
The tradeoff around “do we migrate to TS?” is a great example here, by the way. A huge part of our decision-making last year around the big app at LinkedIn was on exactly that question: If you are going to stop and throw the entire thing away, you should waste exactly zero time migrating any of it. On the other hand, if the thing is going to be migrated incrementally, you should accelerate the TS effort because it will make it so much easier to code-mod the code base.
(I don’t agree that it’s a “cartoon” podcast description, but it is definitely extremely compressed and Adam and I chose to focus on the personal dynamics over the technical details, because diving into the latter would have made for a 5-hour episode. Also: it’s really tricky to do any of this kind of public discussion in a way that doesn’t just end up reading as a one-sided self-defense!)
Most of the time, you have to deal with codebase. And when you finally have chance to work on a greenfield project, you will have to compromise between your opinions and wishes with those of everyone else who is involved.
when you finally have chance to work on a greenfield project
the person that comes along next is going to moan about how much of a mess of compromises and hacks your, now, legacy codebase is
the next big system will be built the least-experienced.
This sounds plausible and is consistent with my experience, but I'm curious if you have a reason for why this is? Is it just that the experienced folks are already working on the established systems, and we don't know what the next big one will be? Or is management doing something intentionally that causes this? Or something else?
Perhaps the experienced folk are "difficult" and have "opinions" about the best way to write the new system which is somewhat at odds with what management think they they need. Difficult opinions may include "no, it doesn't need to be 'AI powered' or include a blockchain or be a combinatorial explosion of configurability".
The inexperienced folk will be more "cool yeah let's do it". You can't blame them, it's how to become experienced.
For what it’s worth I don’t think what you’re describing here is the same thing I meant by “finger guns” at all! What you’re describing looks to me like good incremental development.
With my “fingers guns mode” phrase, I really had in mind a blithe disregard for any of the real-world engineering problems and constraints that we knew we would hit (because they just inhered in what we knew the solution needed to do). You don’t have to solve those right up front, but you do need to (a) acknowledge that they exist and (b) have some notion of what your approach to them will be.
I’m extremely in favor of taking a small team of veterans (who are open to doing something novel!) on a new system, as you say—let them leverage all their expertise, while being careful to avoid Second System Syndrome, to build something by using all of the knowledge about those pitfalls/problems/etc. to avoid them. (The risk there is that you can end up overfitting on previous experience! That’s real!)
The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)
Fully agree with this for all scopes of projects. Larger projects are compositions of this principle with functional communications between focus areas.
There's no financial incentives for Maintenance though, which means "leaders" have less and less incentive to fix what they have versus building from scratch. In my experience in these cases it comes down to (like most things) whomever the least flexible person in a power position decides to do.
Fundamentally the problem with all large organizations is that desires and incentives become diffuse the more people are a part of it and the "alignment" of what should happen drifts, until eventually nobody cares to maintain the original purpose.
You can argue this is good or bad or inevitable, but either way it's a structural issue between the broader world changing, and a organization that seeks consistency
I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
Anyways, AMA
Anyways, AMA
What precisely is LinkedIn trying to be?
Seems like it's turning into 2007 Facebook. Is that intentional?
I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users?
I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't.
Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
I don't buy entirely into "FB did it" kinda thing. They found it ... but users LOVE it. That's a choice in the last part.
They made loads of product changes to get people to engage in that way. As an example, showing who viewed your profile in the email vs needing to log in.
Some users love it, the type who want to sell their every thought online to anyone who can tolerate them.
Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.
I suspect the user base has largely driven it.
In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".
At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.
Oh I disagree greatly.
If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.
Whatever the reason they make that choice, at some point that's on them.
If the choice is "use the platform everyone else is on" vs "don't exist", many people will choose the former.
That's not the same thing as wanting to use the platform.
I've always thought LinkedIn was just Facebook for Business™ - has it ever been anything else?
It was always to some extent, but the feed was a lot more formal IMO at the beginning. Content was more like actual articles and links.
But I think users just love / gravitate towards a more facebook style feed.
Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment.
LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
They're trying to be a site that makes a lot of money from ads. To do that they need to get people to spend a lot of time there. To do that they need an infinitely scrolling feed that has addictive characteristics. What else are people going to spend a lot of time on there? Scrolling through their contacts?
Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.
I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad.
The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.
I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
Seeing you use 'Jugaad' make me unreasonably happy.
I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.
Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
But are things ever permanent? Isn’t it to be expected that stuff can change in the future?
LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is.
Theyre still using python 2 and centOS in tons of systems.
They just started using github last year.
Their tech stack is easily over a decade old.
Theyre still using python 2 and centOS in tons of systems.
There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.
In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.
TIL jugaad haha.
Yea, give everything 3x estimate because everything that can break will break along the way.
Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
Why did the codebase end up like this? Is it a lack of hiring for platform or developer tooling teams?
I bet in most places it all starts with a new innovative system of (self) assessment that relies on number of features/VI/KPI tied to promotions.
In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.
It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
Unfortunately, 17min to build is pretty good.
My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.
As opposed to making builds faster, or build infra faster/cheaper.
There are mostly only a few ways to make builds faster.
* Ship less code (Very hard to get a large org to do)
* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.
* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.
These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
Can you expand on "ship less code"? I think I'm agreeing with you but I'm taking it to mean ship smaller pieces of code, not overall LoC.
I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome.
Sometimes doing the same thing with simpler components and fewer components is just better.
These kind of speedups at LinkedIn scale all fall off the chart at the bottom left of this scale:
Nonetheless, leadership will never give it priority.
Whats part two of the story? Did LinkedIn move to React? Did the initiative fail, or is it ongoing?
This sounds like my exact experience at another big tech company
Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality.
There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.
I did work on one voyager task though and I remember it being a nightmare
Okay. Well, I can keep butting my head against this wall. I had conversations with that manager where I was told in literally so many words, ‘You’re too idealistic. You don’t care enough about the bottom line. You should change your values.’ And I was like, no, nope, that’s not how this is going to work, man. Uh, outside I did the politic.
To me that's the most interesting quote of the podcast and that's the impression I had gotten before reading this quote. Sounds like they received valuable feedback along the way and ignored it on purpose.
My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.
I worked on the facebook.com rewrite to React in 2019. I find this story particularly interesting.
I mean, yes and no. I find staff engineers, and especially the senior-staff-engineer-type positions, to be the ones most sensitive to the exact specifics of the org they've joined.
I've seen a senior staff engineer (at a popular unicorn you've all heard of) who was a very smart, but also friendly and reasonable person (how often are those traits in opposition of each other!). They were very good at the staff engineer game. They were vouching for us to upgrade a framework that we pumped 50M of spend through per year. From v2 to v3. It wasn't a huge change, not like python 2 to 3 at all. Positively trivial compared to py2 to 3. But management just were not up for it at all. Even with research done showing we could expect literally a baseline of 10% improvement in performance (so you do the math on that vs the 50M of spend), they just did not want to "waste time on version upgrades." The staff engineer was told they were being unreasonable in pushing so hard. Their manager tried to ditch them, but again, they're good at the staff eng game, so they switched teams due to being in the good graces (due to an excellent track record) with a middle manager above them. They decided, hey fuck you, I'm going to one man army this effort. They did, they had preview versions ready in less than a month and had migrated some of the bigger-win jobs within 2, saving their annual comp (which was prodigious) several times over just in that. Suddenly everyone was keen on moving their stuff over now that the startup cost (both political and eng-time) was paid.
Anyway fast forward a year, the rollout is complete and nearly the entire fucking management chain above him has turned over, a 50/50ish mix of firings vs quittings. But he's still there and the upgrade migration is complete.
So sure, sometimes staff engineers have a stick up their bum and can't dislodge it enough to get work done on the bottom line. But sometimes they're the only sane ones in a crazy world, and you can only do so much to try to turn a balky mule around.
One way to read your story is that upper management applied heavy downward pressure, to the extent that relatively trivial changes in the organization were discussed as 'costly' - in an effort to get engineers like the senior staff engineer in your example to pull heroic feats of engineering in a few weeks all by himself, instead of using a small group of engineers / resources.
I think your argument is that 'it worked for him' because he is still there while the managers are not, but reality may be the managers were there to have that effect in the first place and the 50/50 turnover was because they were in the thick of it.
That'd be my cynical take, anyway.
"upper management applied heavy downward pressure"
Correct
"I'm an effort to get engineers like the senior staff engineer to pull heroic feats"
I think this gives them too much credit. The reality, imo, is that the org had a pathological emphasis on "impact" and ran performance reviews so tight that everyone, managers included, were so scared of being PIPd and losing their grants that they were extraordinarily risk averse in planning and picking projects to bet on. And leadership in general undervalued "infra-y" changes as mostly make-work.
The turnover was partly because of this general effect - people scared, not betting big or smart enough, and optimizing for holding on / not getting fired. And to their credit, most of these people vested 2-3 years of their 4-year grant before it all fell over, so they made out decently lucratively, so maybe they're the real winners in all this after all.
That's an interesting take, thanks for writing that. That gives me something to think about.
Especially if he wasn't rewarded commensurately for that feat.
I’m not American so I don’t know what a staff engineer does, and googling the term is not helping much. I have also not worked in any software company that has “engineering teams” which would maybe have helped me understand the Google results.
What you are saying is that sometimes a senior engineer manages to make good things happen?
Did this person get rewarded for this huge effort they put in? You say they already have prodigious yearly compensation, was it at least doubled given that they saved it several times over?
Did this person get rewarded for this huge effort they put in? You say they already have prodigious yearly compensation, was it at least doubled given that they saved it several times over?
When you're getting paid $500k-$1m/year or more as a staff engineer, putting in huge effort and getting organization-wide impactful results is part of the job description. I'm sure it had a good effect on their yearly comp review, but suggesting that their comp should be doubled because they did their job is silly.
A staff engineer is a level above senior. What they do is make them themselves more valuable than the regular engineer who has been around for 10+ years. As you get experience there is diminishing returns, so most engineers will not make a staff level (though there is plenty of title inflation and so you will find many). A staff engineer needs to know the code (or whatever they engineer) well, but they don't normally write a lot of code. They tend to spend more time thinking about the larger problems (including forcing through upgrades nobody wants to pay for), mentoring juniors, and figuring out the hard architecture problems (software architects have a role, but in my experience they rarely actually create architecture despite the name - possibly because what developers need of architecture is different from management?).
Staff level developers are trusted to figure out what needs to be done without direction. When they are given direction it is figure out how the other engineers break this problem up and do it - typically not do it themselves.
If staff engineers are doing something it is not important to any project. So nobody feels bad about interrupting them if they need help or something urgent comes up. (this also means you are developing your senior developers into staff engineers by giving them responsibility)
Staff engineers (in the FAANG parlance) are, generally speaking, ICs who were previously senior engineers who have proven themselves enough that they're trusted to be roughly autonomous, they're kind of "peers" to engineering managers in a way. While a senior engineer will typically not have the political capital move big politically-heavy rocks, it is typically down in the staff engineer's job description that they are expected to. It's basically shifting from a focus on code and building systems to people/the org and architecting systems that fit the org.
"Did the person get rewarded for this huge effort" Money-wise? Haha. No. Only in the sense that they further solidified their soft power as someone you shouldn't bet against. And of course the sense of pride that comes with shipping something (that further enriches some capitalist and maybe yourself to 0.0013% of the effect, since that's how much of the company your grant commands shares of).
This reminds me of the blog post, "People can read their manager's mind," by Yossi Kreinin.
https://yosefk.com/blog/people-can-read-their-managers-mind....
In particular:
"Corollary 1. Who can, and sometimes does, un-rot the fish from the bottom? An insane employee. Someone who finds the forks, crashes, etc. a personal offense, and will repeatedly risk annoying management by fighting to stop these things."
you gotta a do what you gotta do, but if you have the luxury, and conceptual integrity, there's going to be sacrifices in the name of "alignment" that you're just not prepared to make. I agree with your positioning on lots of "right" solutions, IME it's recognizing and understanding and still not accepting when it violates your foundational values. I've been there and had the ability to not participate, but this isn't always the case.
You're exactly right and to me that's part of finding the 'right' solution for the organization. Sometimes you compromise for 'alignment', and heck sometimes you discover that those compromises were a better solution all along.
In other cases you chose to 'pick a battle' and find the best way to navigate and get buy in. Because again being right -and getting buy in- is the important part. Not later giving everyone an 'I told you so'.
There's an element of truth to this! I alluded to this (and Adam totally reasonably cut even more of it for time) but I definitely was not as effective organizationally as I could have been; that is one of the major things I learned from my time at LinkedIn. The biggest challenge for DX teams remains figuring out how to (a) communicate but more importantly (b) align their work with the key priorities of the business. I got to be okay at (a) but never particularly succeeded at (b) in my time at LinkedIn. Some of that is on me! Some of it… is on LinkedIn.
That said, in this case, the person telling me “You’re too idealistic” literally meant it in the sense of “You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones. The bottom line matters. So, though, do things like user experience, developer experience, and for that matter just basic ethics about what we build (though happily the latter wasn’t in view here).
“You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones
I have a lot of empathy for that feeling. I've come across this often in my career and I have not enjoyed it. I enjoy building software that I'm proud off. Thank you for sharing your story!
Have you ever worked with a BOFH?
BOFHs suck.
Your career lesson is correct.
>My career lesson is that being right isn't the hard part of being a senior staff engineer.
Not only that, but your own definition of 'right' may or may not align with what is 'right' for the people paying your salary - pretending otherwise is just silly.
In any organization, you advocate for what you believe is 'right' the best you can, someone else (or a consensus of others) will likely decide if they agree - you get to decide if you are willing to live with it/compromise, or move on - I have done both in my career.
My previous company had what I thought at the time was a pretty decent sized app, like 150,000 lines of code. And the LinkedIn front end, when I got there had 2 million, it’s just like, I’m, did you say million? Uh, that’s, that’s a lot.
I'd love to see what makes it so large. That's like 1/15th the size of Linux.
Have you ever worked on projects (particularly java suffers from this) where there is generational bloat because most people have very narrow understanding of the codebase so they can only add things and not remove/refactor without breaking it? That’s how
That's not likely to be the issue here tbh, or, that's focusing on low-hanging fruit. You get large codebases from having many features. No amount of coding and refactoring will bring a codebase's size down in a significant and measurable way unless features are ruthlessly removed.
I mean sure, I could understand Figma's frontend or something having millions of lines but what does linkedin frontend do that would reasonably require this?
My guess is that it's because there are 800 different versions of everything because none of the company's 13,000 UI/UX designers are willing to use the same component libraries.
Blaming 2,000,000 lines of code on UX designers, and not the developers, is certainly a new one.
These high-profile apps are elaborate spying devices. Besides the UI code, they probably have thousands of classes, some of which abandoned as dead code, with names like ProfileViewActivityLocationTracker...
Orgs never value engineers that improve the overall codebase so I can't blame them. You're only getting promoted and compensated by delivering the next big feature which is part of the owners vision and being visible.
particularly java suffers from this
Correlation, not causation. Java is often used on large projects, it doesn't itself make things brittle.
It's like the bomber survivorship fallacy - the fact you see such messed up projects mostly in Java is that they would not survive in languages like Python, where even a simple method rename can easily lead to big regressions.
Usually if you peek at the first few commits it all seemed nice and clean. The turning point is usually as soon as the original author/'s leave and new people come in.
That quote is why you can't take anything seriously on the internet about running software or software engineering seriously.
There are to many people that just don't know what they don't know and act they have things solved.
I think the software world has like two bubbles of people around the scale they work with and they don't understand each other or run into each other, except in interviews and on the internet.
It's funny you say that because as someone in the large scale bubble when I read 2 million lines that didn't seem that large. Other companies of similar size are dealing with 2-10x larger web apps.
Ex. https://stripe.com/blog/migrating-to-typescript
Which talks about 3.7m lines at Stripe.
The site makes me sad whenever I visit.
On my beefy desktop, load times for the home page are at around three seconds. On my feeble (but still enormously expensive: a Microsoft Surface) laptop, it’s well in excess of 10 seconds. Both tests run on the very same network, so my conclusion is that load times are CPU bound, aka there’s huge amounts of JavaScript running, for one reason or another.
Including boilerplate code? ASP and JS is far more verbose in those areas.
Think js+html+css, that's a lot
That's 20 times almost the size of my, how, how do you even build that? How long to build? Oh, 17 minutes for a new build. That's, that's not bad. We should probably make that better, but that’s not bad.
Am I spoiled for thinking 17 minutes is too much, even if it's 2 million LOC? I guess LinkedIn can afford it
It is not too much or too little. The real question is whether the value of X% speedup worth Y hours of developer time to make the improvement. Relative to other work that could be done.
Answer could well be yes since build time impacts everyone, but also have to factor in complexity of the CI system they use.
No, that's not the real question. Decisions have more factors than just what cash will they bring in.
Long build times are an operational risk - it's very hard to apply patches to solve incidents with a 20 minute build time.
So reducing this would be a multi-year project of a sizeable team (let's say 5-10 people). How do you get someone to sign off for this kind of expediture and not lay everyone off when the next economic fart comes?
Because THAT'S the hard problem to solve in any bigger organization. Not hacking the code.
So reducing this would be a multi-year project of a sizeable team
Maybe, or just a knowledgable person to dig in for a bit and solve it. On one of my teams when I was back at Microsoft was working on a similarly huge codebase with build times twice as long. We had a senior engineer who was really into build optimizations from a previous job who wrote a tech design, implemented better project splitting in our monorepo, better local caching, and finally cloud caching, all within 3 months. When I left, our builds ranged from 30 seconds to 3 minutes.
It's not guaranteed that it'll be this simple, but I also don't think it's guaranteed it'll be 3 years and several people.
To clarify: when I say value I don't just mean monetary value - though of course everything in a business ultimately does boil down to that, including the total expectation of risks.
I agree though, definitely a factor
It’s not as easy as it seems to answer that question. You’ll have rebound effects meaning developers will probably recompile more often.
It depends on what those LOC are. Is it HTML? Is it JS? It it a compiled language? How much of it is generated?
Aside: I don't understand the obsession people have with throwing out LOC. It's a completely meaningless number unless you're familiar with the codebase, in which case you don't need to be told the stats.
I don't understand the obsession people have with throwing out LOC
LOC is the metric I get to remain employed or fired on.
Aside: I don't understand the obsession people have with throwing out LOC.
Agreed. A better metric is how much of it you need to interact with on a daily basis in order to perform any meaningful work.
If the code is sane, you'd only ever need to interact with and understand a fraction (a subset of the public APIs) of the entire codebase. You could have millions of lines of code and be productive when knowing only a couple of hundred of those.
If the code is insane, the risk is that you'd need to interact with most of it when introducing any non-trivial functionality. Worst case scenario there isn't any clear rules of what is public and supposed to be used, and what is private and isn't supposed to be used, and people have used "whatever from where ever" to solve their problems.
The second one is always bad, but also significantly worse the more lines of code there are. Perhaps those who throw LOCs around also implicitly tell us that they are dealing with the second variety of code.
The more important number for a codebase that size is almost always "what's the incremental build time for whatever I work on". It's nice to have a "build from scratch" be fast, but almost always you can leave it for two hours to do the first build while you're in a meeting or something. More important is that when you make a change, the incremental build is fast.
This is why build systems like buck/pants/bazel are good in larger codebases. You can trust your incremental builds in a way that you can't in any build system that isn't hermetic.
And we have cloud caching for the "build from scratch" case so even that can be pretty quick. This all just takes a little time investment.
Yeah, they could definitely optimize it quite a lot further. With investment in code structure and caching I bet they could get below 5min at that scale. But also for edit/refresh that should be on the order of a second or two, never minutes.
Sounds to me like he made a couple unfortunate moves:
- Propose a 5 year plan (lol) - Don't lead the incident with leadership, but with blame - Speaking about problems more than solving problems (hard to do) - Lack of relationship building
On one hand, I empathize with Chris. On the other hand, this sounds to me like he just didn't know how to perform in this environment. And that's totally fine! Not everything should learn how to perform in the bureaucratic knots -- startups are simpler in this way. And there's a reason the big guys lose their edge over time, and then some exec in a board room is faced with -10% YoY loss without any truthful VPs around the table.
I go back and forth on how to interpret people like Chris (and likewise, myself).
I've been in this situation, and it's psychologically disorienting: I seem to be right, but am I? Are the people around me really as incompetent as they seem, as totally disinterested in learning from their colleagues, etc. etc.
Then a few years after leaving, you look back and: oh, all those people have been fired or left; they still cannot do X as an org; the agility and competence of teams I then joined really does exist etc.
So, on the one hand, it's true that there's a sort of arrogance, political incompetence, and inability to cope with environments with a pathological practice culture --- but, on the other hand, maybe that's the right reaction?
Maybe the institution is undergoing a pathological cultural period, and maybe talented, considered, passionate people should be driven mad by it. Maybe the people who arent driven mad by it are either irrelevant to productivity/growth, or worse, net deadweights.
Who knows? That's what makes being in this environment such a psychodrama -- is the situation really this bad, or am I over-reacting? "Everyone tells me...."
Don't gaslight yourself. Believe! Remember that shit floats, and you have to pick your battles.
Scum rises to the top.
Its very hard to affect real change without power. The people who are genuinely interested in solving problems generally don’t want to deal with the political intrigue that comes with power. So we’re left with a situation where the leaders suck for the most part.
Playing politics is hard. It’s literally the job of business management and the only way they get a paycheck. So they’re well practiced and beating them at that game is difficult.
If you’re technical, you can get a paycheck just by doing the work. Sure, you can play politics and tech, but is it worth it? Unless you are rewarded as a stakeholder, I’d say it’s not worth it.
A friend said something to me years ago that I find more and more wise as time goes on: almost everything is about aesthetics. Any given group of people (a company, a band, an advocacy group, a family) is pursuing its own sense of aesthetics. What does this look like in practice? Let's say using Elasticsearch would save $1M of engineering time and infra costs over using PostgreSQL for search but the people in charge of search have a vague aesthetic belief that Elasticsearch is garbage and they're willing to spend $1M of the company's money to not work with it. Is that nuts? Is that rational? Is that good engineering? Is that judicious use of company resources? Their bosses are the ones who make those decisions.
More frustratingly, this works the other way. You might skunkworks Elasticsearch into your org. You might have cost-saving metrics up the wazoo; you might have multiple engineer testimonials; you might have satisfied customers or growing sales; you might have better velocity metrics; you might have built profitable services on top of it. An Elasticsearch-hating engineer can convince your boss that it's a ticking time bomb--with no evidence whatsoever--and your boss can say, "hey I found out you're using Elasticsearch for all this, which is a big no-no. I love what you're doing here, but you need to migrate to Postgres".
(I know Elasticsearch flunked Jepsen; it's just an example)
You might think a few things:
- can't I show all my successes and change hearts and minds?
- isn't this how we evolve as a company? we skunkworks things, see if it works, build more on top of it, repeat?
- if we don't take risks, won't another company out-compete us at some point?
- won't some manager take notice of the inefficiency and do something about it?
Sometimes the answer is yes! But it's almost never yes for the right reasons; it's merely the case that you found a manager who hates the PostgreSQL team, or hates one particular PostgreSQL advocate, or finds it advantageous to be seen as a manager who's spearheading new initiatives, or thinks that a homogeneous tech stack is a vulnerable tech stack, etc.
This is the way we all work. We all have our irrational heuristics built up of lifetimes of experience that we hate interrogating; we naturally gravitate to others with the same irrational heuristics because they spare us that interrogation; we assault others who force that interrogation upon us because it's painful. You'll see this dynamic in political tribes, in technology choices at work, on reality TV, everywhere. You can watch us tie ourselves into knots in real time, selectively forget events or facts, or wildly misinterpret things, all to defend our aesthetic preferences. This is the psychological disorientation you're experiencing. It's not common to find people who are aware of these tendencies and--successfully--employ strategies to compensate, and it's even less common to find groups of people who do that. The probability is inverse to the number of people.
Like you (and I guess OP), I've fallen victim to this over and over again on both sides. There is no justice, because there is no God and nothing even approaching perfect competition. Some of those groups are doing great, others are deceased. I've only relatively recently figured this out, so I've now added some things to my interview questions like "tell me about a time you were very wrong" and "how do you reevaluate decisions or longstanding beliefs you have". I also try to check myself whenever I react a little too positively or negatively to something, because that's usually an indication that it's aligned or across my aesthetics and something I need to dig into.
The good news is I've had success instilling this into people I've worked with. It's sort of "a minute to learn; a lifetime to master", but it starts creating a culture of psychological safety where everyone relies on the others to help them be the best they can be.
Strong +1. Also hi Zach!
MM lines of JavaScript! That is bloat incarnate.
I've been thinking of re-implementing something like LI, or rather implement my own contact database (without the "FB" features, please).
The only problem would be how to make my contacts migrate over in bulk.
Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.
LinkedIn was originally built in Ruby, containing 60k lines of code
https://queue.acm.org/detail.cfm?id=2567673
Summary: https://www.pixelstech.net/article/1395463142-Why-does-Linke...
LinkedIn migrated to Node early 2010.
So they claim the change to node reduces the 60k lines of Ruby to only 2k of node. So how did they manage to add 1,998,000 more lines of node in the next 10 years and not think something was amiss?
So how did they manage to add 1,998,000 more lines of node in the next 10 years (...)
Does LinkedIn have the same features it had a decade ago?
Search is far more powerful. It also shifted to add an entirely new social media dimension to it.
God knows what features were made available to recruiters and marketers.
The hard part of a web app is not the stuff you see; it's the stuff you don't even know is there.
Developers love to armchair and then you put a bunch of them together and you inevitably end up with soaking wet bloated festering codebases like LinkedIn
Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.
Platform lock in is certainly intended (even though it sucks for users)
Have you tried grabbing JSON directly from the web server responses to the browser to get your contacts?
I haven’t tried this on LI but it’s my dirty trick for exporting conference attendee lists when they’re made public on websites. YMMV.
Call me crazy but I did not think that was a surprise. I worked at big bank that had consumer-facing JS web app with 6MM lines of code, though now I am starting to think that that number is false, given the feedback in this thread.
Poor kid, learning how an enterprise works the hard way. If he stays in software eng long enough he'll be back here eventually, with a little less gumption, but a little more tactics and grit to fight the battles he wants to fight.
Are we belitteling people now because they have intrinsic standards and pride in the quality of their work instead of blindly chasing whatever is best for their replaceable career in a hypercapitalistic environment?
No. We're just being condescending pricks on the internet
Huh, self-aware HN. Don't see that very often.
Not really bro, but you stay angry
Hopefully they'll learn that such standards don't exist and that if they do will deteriorate by the second year.
If it wasn't for it being some big enterprise corporate company no one would give a stuff about the blog post.
Do I care someone left company X, fired from company Y? No.
Do I acknowledge it's upsetting? Yes, Pack your stuff and try again.
welcome to life.
Aside from the GP comment calling the guy "kid", it doesn't seem especially condescending to me.
I read it more like "Yeah, we've all been there. It's a sucky lesson to learn, but in the end he's likely to be wiser (albeit more jaded) to these things."
I spent 12 years at LinkedIn. Sadly, it's not even close to the engineering org it used to be. The era where Kevin Scott led engineering was a really good one in comparison.
Ryan Rolansky said LinkedIn is essentially feature complete.
Isn't it?
They seem to be trying to jam AI into a bunch of things now.
As an aside, I work on a product that should be considered feature complete, but PMs keep dreaming up features so they can get promoted. I wish we could just take some time to clean up the horrendous tech debt accrued over the past 10 years of feature proliferation.
If you consider lots of features that don't work "complete" then I guess
Every engineer of a company of this size says the same exact thing.
IMO more about growth of the engineering teams that lead to any specific culture deteriorating.
Chris: There we go. Okay, those are running now. Crap. Did I screw up my ability to hear you?
Adam: Hello, hello, hello?
Chris: I did. Dang it. Hang on.
Adam: Hello, hello?
Chris: Yeah, yeah, we’re good now. Okay. Let’s try that again.
Truly fascinating dialog
They know you need something to complain about, so are giving it to you upfront.
Can I complain about the podcast website that truncated the right side of the content on my phone and won't let me scale it to correct?
It's a thing. Like how GenZs start their tiktok videos showing fake urgency.
It is a fairly common podcast trope to leave some bloopers at the start to humanise the people there.
It's no surprise that LinkedIn's codebase is a mess if you try to use the website, it's a complete nightmare. Say I see an interesting post, I click on the person who wrote it to learn more about them, then I press back to go back to the post and continue reading it - now the feed refreshed and I've lost the post.
Say I want to scroll back through messages I've received, I do that, the entire webpage starts lagging and takes 10-15 seconds to register inputs.
Why do I get 30 fake notifications? They are literally not real, made up rubbish to force interactions from people - it's disingenuous. The recommendation algorithm is also completely terrible.
The thing about the notifications though, is you can turn off every single one of them in the settings.
When I tried to turn it off though, I was hit with 100s of different types of notifications. I generally like it when apps/sites do this. This way I can turn off the garbage i dont need, and keep the ones I want. But 90% of thosr categories were garbage. It is really shocking that one can take a good idea this far and make it frustating.
The notifications has long been totally useless.
considering they are owned my Microsoft, their windows app app is a technical marvel. It’s a crippled buggy version of their already too buggy web app. I understand why it was made, but I don’t understand why it was released. It doesn’t look good for either brand.
Since when is LinkedIn is fast paced? It hasn't substantially changed in ten years.
The fast pace comes from product deciding there should be 5 different types of “like” button and goofy AI “takeaways” functionality.
LinkedIn is a solved problem but you can’t have that any more.
Probably a lot of work that is being done there is inventing useless stuff that makes people, projects and departments look needed.
What happened to Twitter shows that in most older orgs with a well established product you can throw out ~50% of the people and it still will go on.
No shade on ember as a framework but LinkedIn’s decision to use Ember always baffled me. As far as I remember they switched to it long after it was clear react had “won”.
LinkedIn actually chose Ember well before React had won, and definitely before it had become the new “no one gets fired for buying IBM.” That all preceded my time there, but the evaluation was late 2014 and the LinkedIn.com web app rebuild in Ember was kicked off in 2015 if I recall correctly. React only came out in 2013, and while it was definitely on a rocket ship it was way less obvious in late 2014 that it was going to be the de facto standard that it was by late 2016–early 2017.
Bonus note: One factor that was really not obvious at the point they were doing the evaluation was how much TypeScript and TypeScript-powered tooling was going to matter. TS shipped support for JSX/TSX in TS 1.6, in late 2015, and I contend that while JSX had a lot of pros and cons, the one absolutely unambiguous win it had over everything else from that point forward was that whether you were writing JS or TS, you got outstanding editor support in basically every editor in the world courtesy of the TS language server backing your JS or TS development. Just absolutely massive.
The fact that the TS team baked support for JSX into TS itself and that it's not pluggable is (a) totally understandable and reasonable from a maintenance effort POV, especially given that Microsoft uses React extensively, and (b) extremely frustrating for everyone not using JSX. Vue, Angular, Svelte, and Ember all had to build the exact same infrastructure to get that support, and that was a huge productivity hit for devs using those frameworks compared to React in the meantime.
Some internal stakeholder surely fought for it so they could get promoted.
Happy for that person, wherever they are now.
I was really impressed with how open Chris Krycho was about his struggles without playing the blame game. CoRecursive is one of my favorite podcasts because it explores the complex context behind the code.
He seems like the kind of guy I'd like to work with, personally.
Agreed. And I also think Adam is a great host. He asks good questions and lets the guest talk.
The introduction sounds very GPTy:
His journey was marked by challenges: from the nuances of remote work to the struggle of influencing company culture, and a critical incident that put his principles to the test against the company’s push for speed.
Chris’s story highlights the tension between the need for innovation and the importance of project health. This all led Chris to a pivotal decision: to stay and compromise his beliefs or to leave in pursuit of work that aligned with his principles.
He chose the latter. Join us as we dive into Chris’s compelling story, exploring the challenges of advocating for principled engineering in a world that often prioritizes quick wins over long-term value.
FWIW, it seems pretty consistent with the style of Adam's other intros.
Note: This podcast is designed to be heard. If you are able, we strongly encourage you to listen to the audio, which includes emphasis that’s not on the page
Seriously? You getting more "engagement" because someone is a good actor?
what?
In my career I have learned to listen more carefully to engineers. Sometimes it is not about saying yes and no to them but listening to what they are trying to say. Sometimes if you ask a few questions they learned something about the business that makes their case come apart. Sometimes if you listen to them you find a good idea and it opens up a whole new area of the product. So I think it’s a false dichotomy and you have to be good at listening and asking the right questions to get your point across.
Really feels like mission impossible to do this kind of role remotely.
I have no idea why anyone is still a user on LinkedIn. They've had numerous breaches, and give 2 entire shits about it. If your account gets stolen, even with MFA enabled, they will take roughly 2+ weeks to respond to your ticket and they require that you send in even more information that they will callously handle.
My advice, leave LinkedIn.
LinkedIn is so awful to use.
There's value in the connections you can make and hiring or finding a job... But it's probably one of the worst web apps I use.
I'm surprised it ever got this big.
Although I do listen to Adam Gordon Bell's podcasts from time to time, I'm not sure if I should invest 40something minutes into hearing about why some rando left BigCorp. This may be an unpopular opinion, but isn't unprofessional to spill the beans about your past employment like this? As an external observer and hiring manager, I wouldn't touch this person with a 10-foot pole if they're in the hiring pool.
I personally find this type of content to be mundane, basic and useless. This is the type of stuff most people run into if their engineering career spans more than a couple of years, touches large organizations, they can't handle conflicts and have the luxury of being more affluent than other employees. I don't understand why someone would go out of their way to share something like this.
Can't stay in business swapping dollars. Usually, the ones declaring their idealism are going to find a problem with you regardless. Doing the right thing is what you do, but that's not the sort of attitude that works.
But I bet he still has his resume on LinkedIn. :)
This sounds like one of those "soft leadership" roles, that almost always suck because you're "responsible" for something, but have little or no authority over the rest of the org. The real tech leadership (if any) are MIA or no longer in touch with the actual problems, even if they've been their forever or are the "system experts"
Been there, done that, no thanks.
The big rewrite. Risky even on manageable codebases, and the leftovers never seem to fully go away. Who wants to score points on re-writing a tucked-away settings page several years into these?
I've seen so many attempts at these you'd think we'd have a framework for rewriting codebases. We don't. Automated codemods require a consistency that few adhered to. The code patterns evolved so much over time, it's like looking at tree rings.
We're basically putting code in boxes, re-arranging the boxes, and rightfully saying some arrangements are more efficient. How come we haven't found a better way? Automations operate at the code level, but not at the box level.
LinkedIn comes across as a not-so-great company to work for. I find their product to be very toxic, off-putting and full of life coaches and scammers.
Boeing in action. Luckily nobody is flying LinkedIn MAX, so there are no deaths as a result.
If you want to dig into Conway's law and its implications, I can't recommend this video essay by Casey Muratori enough: https://youtu.be/5IUj1EZwpJY?si=dPxsXieBwZsP0PPP
Fair warning though, you'll lose all hope that companies like Microsoft will ever manage to produce anything that they don't then ruin.
Oh. My. God.
This video explains everything I've seen in the enterprise IT of a huge government org that was the result of a long series of back-to-back mergers and splits.
It's exactly this!
Conway's law, but over time.
Dysfunction not just because of the large hierarchy, but also because of the built up layers of history. Some elements still limping along, some essentially amputated and mere phantom limbs of the org tree now, some outright dead and spreading miasma and corruption from their decay.
I knew or suspected most of this, but to hear it said out loud so clearly has crystallised my understanding of it.
Yeah once it hits, it really explains so much of the issues in large organizations (and explains a lot about small organizations / indie development as well). For example, why is it that mergers and buyouts rarely work out? Conway's law largely explains it! When a product is bought by an organization, the result is an integration over time of both organization org-charts combined. If the companies are both large, this immediately becomes an intractable mess. If the original company was small and the new company is large, it _also_ becomes intractable because now different teams will have to communicate to manage or split components that don't fit the communication patterns of the new organization, and will lead to large refactors and rewrites and in the end the essential qualities of the original software will inevitably change.
Even if you know this, there is nothing you can do - you have to organize yourself somehow and how you do that can't be left up to just mirror whatever this piece of software that was brought in happens to be organized - because _that too_ doesn't reflect an actual org chart but is an integration over time of all org charts that have touched it.
I don't even know how to think about the implications this has for the use of open source software and how open source is developed, like... I think there are huge opportunities for research into Conway's law and how it relates to software development practices, software quality, etc.
I have this naively optimistic hope that AIs will allow orgs to scale units past Dunbar’s number.
We humans can’t effectively communicate with groups larger than about 7 people, but AIs have no such limits. We could all simply talk to one manager that can integrate everything and everyone into a unified whole.
It’s like the ship Minds in the Culture series having hundreds or even thousands of conversations at once.
That's an interesting thought, yeah... technology and communication are definitely interwoven, things are possible today that were impossible before computers simply due to communication constraints. One could imagine a large number of practically autonomous small teams organized "automatically", more mirroring a neural network than a hierarchy.
The problem is always communication because it is the means to cooperate. The root of many issues in software development is the simple fact that instead of letting the required communication pathways define the organization, it is the organization which defines the pathways and through that causes communication obstructions.
"Not Just Bikes" has a good few videos, including https://www.youtube.com/watch?v=n94-_yE4IeU and a couple more that talk about problems that larger roads effectively cause more traffic ("induced demand", "traffic generation"). Organizational structures are like roads, and like roads they can get overloaded, which in turn means traffic needs to be reduced. There is even communication jam, and to combat that something like enforced communication reduction (lower information throughput), etc. to keep this manageable. That also causes decision making being done with less and less information the more steps are included in a communication chain (like upwards/downwards in a hierarchy), which in turn means the quality of decision making is severely hampered by it.
This whole mess is also the reason why the agile manifesto puts humans before processes and other such things, in fact it implies you change even the organizational setup to fit the project, not the other way around. But in the space of "managerial feudalism" (David Graeber) this is pretty much impossible to pull off.
The tragedy of agile is that the practices that are labelled agile in practice tend to exemplify the exact opposite of everything that the manifesto advocates..
The more the initial fade of AI assisted work sets in, and given the inherent vagueness and unpredictability of managing, I'm eagerly awaiting not my job, but my bosses job being replaced by AI. There's no need for exactness, but superficial clarity, decisiveness and seeming coherence.
You might be correct, but the AI minds that you are contemplating don't exist yet, and there is no reason to think that they will be developed from current LLMs.
Once again, seizing the term AI to mean LLMs and other current generative techniques has poisoned clear thinking. When we think "AI", we are thinking about HAL and Cortana and C3PO, which is not what we actually have.
interestingly positive, I think I might agree with you. It seems the nr.1 problem of good leaders in organizations is that they can't be everywhere at once. But an AI can be.
I've watched a lot of Casey Muratori's other presentations but just happened to find this one last week and I wholeheartedly agree. Like many people I'd heard of Conway's Law but always imagined it as a pithy truism and hadn't thought that the effects could run so deep.
Casey's example is focused on a (frankly, ridiculously) large organisation but I've seen the same thing in numerous small companies with just 20-30 developers, and it's hard not to imagine that this is universal, which is a pretty depressing thought.
Recently I've been involved in a new project where teams were set up to be 'domain-specific' with the idea that this somehow avoided the issues of Conway's Law, but this just feels like exactly the same problem because team structures and the software that they end up producing is siloed in exactly the same way as the organisation structures that existed at the time.
Casey's point that the final design is inherently limited by the lack of high-bandwidth communication between the groups of people that need it most is also fairly demotivating, personally.
Tangentially, having been writing more Golang recently this made me think of Golang's mantra (or was it Rob Pike's quote) of "Don't communicate by sharing memory; share memory by communicating.". Go/CSPs concurrency and sharing model is interesting, but I wonder if it's a fair analogue to compare sharing memory by communicating as low-bandwidth communication, and communication by sharing memory as the high-bandwidth communication that seems to be needed for effective designs.
Does Conway’s law imply that an organization of size 1 produces a system with components that are optimally integrated?
I sometimes wonder what would happen if a company hired 100 software engineers, gave them all the same task to program the entire thing independently on their own, and then paid them as a monotonic function of the ranking of their output against their peers. (Obviously, there are limits on what type of product could be developed here, but I think this still encompasses far more company products than you would suspect.)
Fred brooks believes the Max is 1 or 2 (if they have experience together), and explains how to scale an org around those people.
Does he say what to do when they die, quit, or retire?
So clever! The brief answer is you have many staff who have knowledge but not creative authority. If you think about it for a few minutes I think you’ll find some options.
Run all 100 in parallel in production and monitor for inconsistencies.
I remember one time we had to provide some important numbers to auditors. A data analyst on the team wrote some queries to collect this information, but my manager decided to write his own queries independently as a consistency check. The numbers didn’t match at all.
So he asked me to write my own queries as well to see who was right. Lo and behold, my numbers didn’t match either of theirs at all.
An organization of size 1 has other constraints, like time.. But also, Conway's law applies outside the organization as well, communication constraints shape society at large and the way a society is organized limits the communication channels available.
On the topic of Microsoft/Linkedin, aren't they the ones who used to or maybe still assign the same task to multiple teams and let the teams compete for who can deliver? That does sound vaguely like what you propose.
It’s creative work, like anything else. 1 is not necessarily always the right size, but 100 certainly isn’t. If you look at musical bands usually a sweet spot is between 1-6. You can add more in eg an orchestra or choir but everyone’s not involved in the creative work.
Then it depends on how well you vibe with people. If you have extremely unique ideas and thoughts, you may be better off alone. If you find peers with similar values, preferences and experiences, you might be fine with 5 people.
The best way to de-risk a project is to run three at once. Just don't let the teams find out about the other ones.
This feels like something that is just crazy enough to work.
Most teams I've been on or managed were at their optimal efficiency with three people. Just enough to keep any one person from making too many poor decisions, but not to many to spend half a day in meetings.
What’s broken with current office 365? It’s a good product and service.
This is what you see from the outside. LinkedIn also looks good from outside. This is the entire point of the story.
If the user experience is good, then the product is good. Users don't care about build time or internal dependency management.
Making something look good is very popular but entirely different from actual quality.
Build time & internal dependency management are just a few markers used to predict where user experience will go in the future
I partially disagree. IIRC their search is broken, and their job alerts are pretty lousy at de-duplicating advertised positions.
And IIRC they hijack the back button, which I find pretty offensive.
The existence of the product is broken. Office suites should not be web applications.
To be fair, it’s only a web application if you want it to be. I have all the usual office products installed locally. When I get a link that opens a document in the browser I click open in app and off it goes.
Microsoft has moved very quickly to add LLM functionality to multiple products
Literally the opposite of LinkedIn. I’m approximately user 5000 on linkedin, and since I joined almost nothing useful has been added, despite a huge increase in complexity
Come on now, that's not true. Every week LinkedIn adds a new email marketing category which you haven't opted-out of yet, because it didn't exist before.
Exactly. Remember that we are not customers, we are the product. Presumably, LinkedIn is adding a lot of useful features for their actual customers, they're just not visible to us.
LinkedIn and Microsoft are the same, that said I personally think the rush to push LLMs into everything is entirely consistent with my prediction and with the internal structure of Microsoft (current and historical).
I've witnessed what Casey calls "Conway's Nightmare" (at 32:04 in his video) firsthand- it is definitely real. Basically, a software system isn't necessarily just a reflection of an organization, it's an amalgamation of all of the organizations that worked on the software system in the past.
you are implying that we still had any hope left in such endeavors.
I think it is more nuanced, you can also change the organization bottom up, but it needs to be new stuff that does not exist or has it infancy. Changing existing stuff is very hard even top-down.
If it was new stuff, then the people doing it must not be very low at the bottom (by virtue of the org being quite shallow - such as a startup).
Even new stuff in a big company with lots of layers of management cannot make new stuff with excellence. Eventually, probably sooner rather than later, it gets polluted.
Not to mention that in a big org, the people at the bottom do not get rewarded for making excellence in engineering, if the top doesn't appreciate it. SO why go that extra mile, when a mediocre job will earn you the same salary? You'd be more incentivized to move jobs to grow your pay instead, and use this as a stepping stone.
As somebody who has only ever worked at places small enough for individuals to all roughly understand eachother's impact, could I even get a job at a big org like that? I don't even understand the mindset
Yes of course. You have to demonstrate that you are very skilled at building with the particular bricks they use. Don't mention anything else to show your commitment to particular brick usage.
Yikes! I didn't want to understand the mindset after all! Thanks :)
Get one? Sure. Be happy with it? Propably not.
I learned that the hard way, only the other way around.
Have you changed an organisation from the bottom up? In two decades I’ve never seen it happen in organisations larger than 50 people.
You can change something, like how an engineering team works and how an organisation does DevOps and other things that management doesn’t really know anything about and trust their employees on. But moving an organisation into something like team topologies which is a more modern expansion on Conway, is virtually impossible from the bottom up in my experience. Change management in general is very hard both directions, but it’s much harder going upwards because going upwards means you don’t really have the support of management above you. Maybe they’ll humour you but to make an actual impact you’re going to need them onboard. You’ll even have to reach pretty far up top if you want to inflict lasting culture changes as things carried by a single (or few) managers often dies with them.
My career has sort of put me in a role where I help non-tech startups transition from that to medium or enterprise size. I solely focus on the tech journey though, as I know I won’t have too much impact on culture or work processes on the grander scale. Often this leads to a semi-team topologies approach as the tech teams divide systems into independent business related services, but as a whole I don’t expect any changes to reach outside of the development departments.
I'm not disagreeing on your overall point, but it can be done (I've done it with 130+, so there's a lot of incumbent process and numerous senior managers to align).
The key is having a champion who can tie change in with higher level desires.
In these orgs there's often an external (as in stakeholders external to prod eng) perception that "engineering is slow, product doesn't deliver, they're preventing us delivering our OKRs", and that can be used as leverage for change.
You can't go dark and stop delivery, but you can usually carve out a 10 - 20% allowance for change (that doesn't sound much, but is 1 - 2 days every 2 weeks, which quickly adds up). Start small, show success in impacting metrics that external stakeholders care about, then next quarter push for more.
I've focused myself in a similar area as you, but actually lean into processes and people, whilst still guiding tech - maybe we should chat!
Indeed, and greenfield projects are a great way to move upward in the org - as Jim in the story recognises.
That highlights the greatest repeated tragedy to befall software: poor leadership. For some bizarre reason developers almost always think they can fix people problems with better tools. For example if all your developers suck give them a popular framework. It’s just an excuse to avoid dealing with people that provides license for the children to run the daycare.
If you want excellence set high standards defined by rules that to hold people accountable and impose ownership (liability/reward). It’s not complicated but it requires not being terrified of assertiveness and confrontation from the top down.
It requires the professional communication skills that are not taught in a STEM education. That's why this is all so difficult: we are not taught how to communicate effectively with one another, and the entire tech landscape has poor communicators misinforming and misleading one another.
Statements of poor communication always make me wonder if the shirky principle might be part of the problem
I do not believe so. This lack of communication skills is also due to a lack of recognition of the value good communications engenders. An undergraduate business school education pragmatically touches on professional communications, but only as far as how to motivate and control people, not how to effectively communicate - as in a shared synchronization of understanding, which is what professional communications is all about. Colleges of communications teach professional communications as their foundation, before the various media majors one graduates within. The more I look at the levels of communications in general within society, our civilization has not yet recognized how critical communications are to every aspect of life, not just business and careers. Good communications are like fire, capable of destroying and capable of being the foundation greater things are built, and the lack of this recognition amazes me.
That cannot be over-stated enough. I learned my communication skills in the military while building my software career. Those worlds are so incredibly different that I have spent most of my professional life lost.
On one hand my expectations for excellence in software are alien to my peers who just want to complete a JIRA task by any means possible. Yes, it is possible to deliver superior results for an entire day's effort in about 2 hours provided you aren't framed by all kinds of institutional roadblocks and unnecessary abstractions. I am scared to death to actually communicate my opinions regarding software because people tend to whine about how hard life is. For this reason I have abandoned my JavaScript career.
Then on the military side I have to unlearn the high agreeableness that is so necessary for not getting fired in software. By high agreeableness I mean keeping your opinions to yourself, not voicing concerns, treating everything with kid gloves, and making everything as easy as possible. The military side is a much safer environment that expects brutal honesty. Nobody there believes themselves to be god's gift to the universe and they are always in a learning mode, which means they are coach-able and will not whine about corrections and mentorship.
Although I do not have military experience, I've spent some time writing software deployed to military security clients, and my interactions with technical armed forces staff has had remarkably good communications. To the degree it changed my attitude about the armed forces in a significant positive direction, and not just the tech side.
I think well communicated terrible ideas are still terrible.
From the point of view of upper management, the squeaky wheel gets replaced.
On the other hand, the non-squeacky wheel gets to collect their salary for a long long time.
Top down comes with its own risks. The bottom line is big ideas and big change can only happen with both the champion at the top, as well as good understanding at the bottom, and a critical mass of good alignment and competence throughout the middle layers. Though Conway's Law is immutable, it doesn't necessarily depend on a formal org chart, it can be worked with by simply creating ad-hoc communication structures between the right tech leads and competent managers. A handful of technically weak or empire-building managers in the middle can fuck the whole thing pretty easily though, and depending on the lifecycle of your company it may be a lost cause due to Pournelle's Iron Law of Bureaucracy.
How do you hold the middle together? Especially under such turbulence? "Strong" leadership isn't enough.
Reflective practice [0] is one way to build that. It's common in medical and some safety-critical civil/military spheres but strangely missing from many corporate structures. I talk about it here with regard to a more mature cybersecurity stance [1]. It intersects with Agile ideas but doesn't make much noise in software engineering project management as much as it should? Anyone have this in their org?
[0] https://en.wikipedia.org/wiki/Reflective_practice
[1] https://cybershow.uk/blog/posts/reflective
Have fewer middle managers?
Although it’s not always necessary to go through the middle - you can do the old switcheroo and build a parallel org in your image, emerge from the shadows and duke it out with the old structure.
If you come at the king you best not miss. Fail at taking down the old structure and those relationships are so destroyed they will come after you. I’ve seen groups of consultants become unemployable across the whole industry by failing to take down the old guard. Word spreads.
Very good comment
Conway's Law isn't a "law" - it is an interesting observation and there are definitely interactions (both ways) between organizational structure and software "design". But people weigh it too heavily. There are plenty of counterexamples of, for instances, fine-grained teams in one company that build a monolith, and the same type of teams in another company building a microservices architecture. Complex software is complex, and once your require more than a handful of people, that complexity requires a complex organization structure to handle the challenges of communication and responsibility. But it isn't a purely deterministic relationship between organizational structure and software structure. More that certain organizational structures are really bad at building complex software, and some are better.
It is interesting to think about, but there is this tendency to invoke "Conway's Law" as some simple method of fixing the extraordinary complex problem of developing and evolving complex software over time.
Really what it is is that organizations from the 70s, 80s, and 90s were built around building physical things, for the most part. Manufacturing and construction, mostly. And taking those organizations and applying them to software development fundamentally was a terrible match. We had to find new structures that were better suited, and we have gotten a lot better, but we are still working on it. Management was taught by those that managed manual laborers, who's output was easy to measure, and whom you could force to work a certain amount without a significant drop in their output. The same isn't true for knowledge workers. And so new management had to be created, a process we still are working on as well, but many managers these days are way better than managers from 30 years ago.
Read Conway's original paper, it is interesting, but it isn't a physical Law that cannot be violated!
These aren't counterexamples because Conway's law makes no comment on how the teams will choose to deploy what they build. It talks about the overall design of the system, the solution space that was explored. Windows is a monolithic application if the alternative is microservices, but anyone who gives it more than a cursory glance can see the org chart (both past and present!) reflected in the disjointed UX.
I'm not sure why this makes Conway's law not applicable anymore—what you're describing is that new management does a better job of creating communication structures that lend themselves well to creating software. That may well be correct, but the resulting improvements validate Conway's law! If we're getting better at software, it may well be because people are talking about and accounting for the impact that communication structures have on the output.
It’s not a physical law - it relates to people - “law” has a wider meaning outside the field of physics.
I recommend watching the talk! He specifically addresses the very points you bring up. Of course reality is messy and people especially so, but Conway’s law is one of very few theories within CS that has held up in replication studies.
...like ?
Not just one layer, look at the stack: Type script is MS's Conways law. API/microsevice everything is more of a google thing (vs FB monolith)... So you have two other major product of Conways Law that your trying to jam into your own org.
On top of that a 5 year plan is never going to digestible by (almost) any company. How many people stay in one place for that long? How much has the front end changed in 5 years? Is any choice you make in the JS world relevant in 5 years? Though sell there too!
I'm adding "Conway’s Soup" to my vocab as a name for the result of such process.
And even then - the senior champion might as well burn out on it, even when they do manage to pass the initiative. Been there too.