56 replies
21h4m
I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability.
I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promotion and salary/rsu bump, ofc.
People need to realize that code is a liability, anything that is not the business critical stuff that earns/makes $$$ for the company is a distraction and resource sink.
This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun.
I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech.
Code is not a liability. Code is what makes a company move its gears.
Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.
For trucking company owning and developing trucks makes sense.
But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business
The margin on trucking could be so much higher than plumbing that most plumbers could never afford the R&D necessary to advance flushing tech. Big truck operates at a scale where they materially benefit from better flushing, so they take their truck dollars and pour them into their own plumbing lab. Big truck sees this as a competitive advantage that no one else is positioned in the market to unlock. They may one day enter the general plumbing space and disrupt waste management, at their option not obligation of course.
This describes Google and Amazon perfectly - while you can armchair quarterback their biz decisions they are definitely doing well for themselves.
Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular.
The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server
Can you steal something that is given away freely? People are buying those services so they must be providing some value.
I was just replying to the comment above that amazon somehow rolls their own stuff and gives back to the community by open sourcing their systems.
Amazon’s approach is the opposite: steal open source repo and make $$$ off of open source contributors’ labor
I can't find a single line in the Apache software license that would indicate that Amazon is breaking any agreements set forth for Airflow.
Accept they also provide the security, billing/invoicing, IaaC, support, provisioning, scaling, list goes on.
As for pay per server vs pay per usage. Heck you know Amazon actually bills the team who caused the cost. And gives finance a report on how much each team is spending and on what. Good luck doing that on prem.
The question is how much do they give back to the open source community, after making boatloads of $$$ off of opensource contributions and whether their model is sustainable and healthy for the FOSS movement
Trucks are literally an asset - you can't do depreciation on a liability.
The only way a 'truck' could be a liability is a lease for said truck.
There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other businesses.
Actually you can go replace truck with server and you describe what made AWS make initial sense.
Please stop misusing accounting concepts.
A truck also comes with a maintenance liability if you want to continue getting value out of it, just like code.
Liabilities are obligations of a company to pay money owed to a lender as a result of a previous transaction.
You are describing an operating expense which has an entirely different nature than a liability.
'comes with a maintenance liability' is a handwaving statement that means practically nothing without a ton of contextual information. A true liability has a contractual set of obligations to pay defined amounts on a agreed upon schedule. No one is going to come after you for not changing the oil on your truck, try missing payments on a lease.
Several parties will come after you for not changing the oil on your semi-truck that is being used professionally for freight, starting with your driver, your insurance company, and the US Department of Transportation (DOT), specifically the Federal Motor Carrier Safety Administration (FMCSA), with whom you have m have to provide maintenance records. Trucking is a highly regulated industry, and after Crowdstrike, software engineering is only going to get more regulated, not less.
Assets can also be liabilities. The mortgages in a mortgage backed security is both an asset and a liability, as was only too well demonstrated in 2008... It's an asset in the security portfolio, but until you sell the security, it's a liability for whomever is securitizing it.
In the GFC the government literally created the Troubled _Asset_ Relief Program. Those MBSs were assets and didn't magically become liabilities.
The problem was the market value of those assets plummeted because no one expected them to generate the agreed upon cash flows because the underlying loans were going into correlated defaults. Despite all this the only party that saw the mortgage as a liability is the individual who's responsibility it was to make a monthly payment on said mortgage.
Outside of swaps and other derivatives financial instruments and other properties don't magically switch from being an asset to being a liability based on random external factors.
This conversation is like accountants talking about processes, threads, fibers and context switching... very imprecisely.
I'm not using liability in the accounting context, but in the colloquial one.
Code is absolutely a liability. Code deteriorates as conditions change, and unchanged code also becomes more vulnerable in a way that conventional objects can't.
But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.
Especially when the cost of minimizing (long hours, unsafe conditions) is not felt by decision makers, and may not materialize for a while, but the benefits of maximizing their potential is felt directly and immediately.
Incentives are everything. That's why managers are so careful when applying them to their own jobs.
Using open-source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don’t work for you, etc.
Not open source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don't work for you, etc... ;-)
Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore.
They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves.
How are these open source tools supposed to be created in the first place? This is the process, someone has to do it
Usually the corporate needs differ too much and they end up keeping their own fork anyway.
Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and this company doesn't have any reason to review and fix code that is not part of their business.
It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.
You are arguing the difference between the letter and spirit of the law.
you can fork it and continue dev in case they change their mind, it is open source.
any other 'source available' licenses would not (legally) let you do that.
It has the Apache license, definitely Open Source. https://github.com/Netflix/maestro/blob/main/LICENSE
I think the contention here is more about whether it's an open project— does it have an open bugtracker, an open project management structure, clear governance, etc.
It not having those things is fine, and eventually someone may still take the source and create an open project around it. But understanding that is a Netflix project helps calibrate people's understanding around whether the model when you find a bug is going to be "fork, fix, and run the fork indefinitely" or "fork, fix, contribution accepted, drop fork and return to upstream."
Absolutely this. It literally happened to me with a Netflix OSS we were using at work. I found a bug that was biting us, opened a ticket with a PR attached with a possible fix and got an answer after a few months "ah yeah we fixed this in our internal version time ago, thanks, will merge it now".
So Netflix expects open source community to pick up the maintenance tab ?
I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix:
1. They claim it is distributed system, but it is just a regular Java crud with SQL backend
2. Java-like DSL with parser and classloader (why? Just why?)
Projects like these are the perfect examples of Enterprise Grade FizzBuzz (https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...) and this is exactly what I dont like about it
Isn’t this the deal with all open source? They are giving something (the code and access to the project) in return for help maintaining it?
No one is being forced to do anything. It is not like there is some open source contributor somewhere now saying, “oh damn, now I have to maintain this, too?”
If people like it and find value in it, they can help contribute to the project in ways they want. Netflix gets to use those contributions, in return for letting people use their contributions. That is just how open source works.
If Netflix still heavily uses this internally, they should still do the most maintenance. Others contribute based on their own needs.
I think the notion of open sourcing a project, is you are literally asking at the community for help and that the community will naturally help you with the maintenance.
You are making great points. This is power of Netflix marketing and branding that they are considered as cutting edge tech company. In reality most of Netflix Java projects are pretty mediocre Enterprise Java stuff. Last year or so they have mandated Spring Boot as their development platform for all their web services.
This is exactly same stack I have to deal daily and management reason is it is lowest common denominator that works well with 3-month contract developer to deliver Nth micro service whose sole job is to call another service.
People want an alternative to things like Temporal, and don't want to handle DAGs with Kafka Streams.
In fairness, the very nature of open source is that the community is only going to pick up the maintenance tab if the value they're getting out of it is worth it.
Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff
airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively.
And they develop them in a way that works for many customers and use cases, not just netflix.
But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into custom cron?
But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.
Agreed.
To be fair, I doubt Maestro will take off like Airflow did.
Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm.
The field is quite a bit more crowded now.
Seems like you have some experience with the orchestrator offerings. Airflow still the way to go, or would you recommend something else for someone just starting down the path of selecting and implementing a data orchestrator?
That is a huge load bearing statement.
Do you plan on any contributions back to the community yourself?
Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.
If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business.
Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ??
Would the world be better with 10 workflow orchestrator systems or one mature?
Netflix is building a workflow orchestrator not a container orchestrator. The viable alternative would be Airflow or maybe something like Temporal. K8s alone isn't going to meet the need in this case.
Does the world need another workflow orchestrator? Who knows - some folks at Netflix seem willing to pay a handful of engineers $ to do so. Good luck to them
Aren’t most of those things developed in house at tech giants and later open sourced?
I am confused by this comment:
improvement and maintenance is continent on usage, and having been used at Netflix, this project is in a better place to have already faced whatever bug you are worried about (and let's be real, 99% of applications wont ever get the luck to exercise code paths sophisticated enough to find bugs Netflix has not found already).
You might be unnecessarily projecting here. You don't have evidence to support that open sourcing this might have been for any other reason than it is simply good for the community to have.
100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.
I was going to say this. I never mess with random libraries like this, always so much pain.
This sounds like the beginning of a sales pitch.
Code that you own and intimately understand is less of a liability than some 3rd party dependency (paid or free). Stitching together a patchwork of dependencies is not likely the optimal result. The more aligned your codebase is with the problem you're trying to solve the better, and if functionality is core to your business better to own than borrow or rent.
Off the shelf come with its own set of burdens, not always sunshine rainbows and loli's
This is a naive view, other people’s code is even more of a liability. Look at crowdstrike and opensource infiltrations. Using opensource software doesn’t magically grant you security nor stability.
I very much disagree with this take-- and the more I've experienced throughout my career the more I'm sure of it.
Companies spend an IMMENSE amount of time and effort adapting sometimes subpar off the shelf solutions to fit their infra and pay an ongoing tax w/ increasing tech debt trying to support them. Often something bespoke and smaller + more tailored would unlock significantly more productivity if the investment is made consciously.
Any code that is written has both assets and liabilities. But to claim it is a distraction and resource sink is a very, very bad take. Every decision to build something in-house needs to be done thoughtfully and deliberately.
3rd parties are also a liability. Pick your poison. Trust in unknown individuals, trust in megacorps, or trust your own people. Choosing wisely is why people get paid the big bucks.