Tech Debt: My Rust Library Is Now a CDO

The library quality is mostly fine -- however crates.io is basically unmoderated and it will get worse and worse overtime as things bitrot and collect dust or the namespace gets abused (already happening).

The biggest problem is the exploding transitive dependency graph, including multiple versions of the same basic utilities.

Even small projects can suffer from this. It's too easy to reach into the grab-bag at crates.io and pull a little utility out and use it. I've personally built very small things only to find multiple versions or implementations of e.g. sha256 or rand or whatever in my own dep graph, and it gets really hard to prune them out.

At work we're fairly diligent about this but still it's exploding. And when you consider the matrix of minimum-supported toolchain versions for different projects, and if you're stuck on an older Rust toolchain, it's even more pain.

Rust projects starting from scratch could consider vendoring their dependencies rather than pulling from crates.io. But support for vendored dependency in Cargo is weak/bad. You run into all sorts of nightmares especially when using workspaces.

It's heresy and I'll get downvoted for it, but: More and more I lean towards using bazel or gn/ninja for fresh Rust projects, and walk away from cargo & crates entirely.

Cargo (+ Crates.io) was delicious to me at first, but it's a meal that tastes worse the more I have of it.

EDIT: it strikes me that I didn't say why exploding dependencies bugs me so much. There are logistical reasons for sure (competing APIs, compile fails, binary bloat, etc.) but the primary reason is: exploding complexity. Every additional dependency is now a potential problem for future me/future team.

Two dozen dependencies or one and a half?

    └── num v0.4.1
        ├── num-bigint v0.4.4
        │   ├── num-integer v0.1.46
        │   │   └── num-traits v0.2.18
        │   │       [build-dependencies]
        │   │       └── autocfg v1.1.0
        │   └── num-traits v0.2.18 (*)
        │   [build-dependencies]
        │   └── autocfg v1.1.0
        ├── num-complex v0.4.5
        │   └── num-traits v0.2.18 (*)
        ├── num-integer v0.1.46 (*)
        ├── num-iter v0.1.44
        │   ├── num-integer v0.1.46 (*)
        │   └── num-traits v0.2.18 (*)
        │   [build-dependencies]
        │   └── autocfg v1.1.0
        ├── num-rational v0.4.1
        │   ├── num-bigint v0.4.4 (*)
        │   ├── num-integer v0.1.46 (*)
        │   └── num-traits v0.2.18 (*)
        │   [build-dependencies]
        │   └── autocfg v1.1.0
        └── num-traits v0.2.18 (*)

It has a large standard library compared to something like C or C++, but it’s small compared to a lot of more recent languages. Intentionally so, since they wanted to be able to create smaller binaries.

> "the standard library is where code goes to die"

I disagree.

It's more like the code has ascended (https://nethackwiki.com/wiki/Ascension). It has won because it is useful enough and mature enough to accompany the language and serve everywhere.

Depending on what it does, standard library code may be used everywhere. For example, would you say that the Rust's Vec (https://doc.rust-lang.org/std/vec/struct.Vec.html) is "dead"?

It can still get new features added. It will still get bug fixes. But its primary APIs are engraved in stone, never to change because it would break all the code that depends on it.

Granted, some code that goes into standard libraries will eventually fall out of use, or will be replaced by something better. In such cases, the code should be marked deprecated and eventually removed in a future edition of the language. In that case, it is like the code retires or could even be forked into a community-supported library in case it does still need to be used.

.NET really is something special. I guess that’s what billions of dollars buys you? But even still the saying is true about many other standard libraries.

> The Rust team prioritized keeping the standard library small, and allowing important but inessential functionality to live outside the stdlib and keep their own release schedule.

There are crate "recommendation lists" like this: https://blessed.rs/crates

While these help, Rust's small standard library has slowed Rust adoption where I work because our IT security team must approve all third-party packages before we can use them. (Who blindly downloads code from the Internet and incorporates it in their software?)

This means that unless there is a compelling need to use Rust, most teams will use C# because .NET is approved and has almost everything. I prefer Rust to C#, but we are not using it that widely yet.

It's not a fair comparison, but it is how things currently are.

I wonder if there is a scope for a "opt" / "extras" type level where libraries for stuff super commonly used, like JSON/YAML/ssl etc can be placed. It doesn't have to cover everything the defacto third party libraries do (that will be too much and not practical), just including happy paths should be enough for most people to just include the meta package and get the most generally used features instead of including a ton of third-party libraries.

My knowledge about Rust's library system isn't great so there are probably several gaps in my argument.

No, but it's an acronym in common circulation in .NET land.

What kind of decisions driven by Azure do you have in mind? What kind of impact does ASP.NET Core (or EF Core for that matter) have on CoreCLR and NativeAOT?

Technically speaking C# allows custom scheduler implementations, and internally ThreadPool is an abstraction too - e.g. AOT will use kernel-provided threadpool on Windows for size savings by default.

Practically speaking, however, you are right.

Yes.

The release notes for the last release explain: https://github.com/dtolnay/serde-yaml/releases/tag/0.9.34

I found an explanation. Reading between the lines I think he found something less abysmal than YAML.

https://github.com/dtolnay/serde-yaml/releases/tag/0.9.34

As of this release, I am not planning to publish further versions of serde_yaml as none of my projects have been using YAML for a long time, so I have archived the GitHub repo and marked the crate deprecated in the version number.

He does this all the time to libraries. These things are correlated; the more tools you make, the less time you have to maintain any single one of them. You learn to ignore it because they're also extremely well made libraries where the unmaintained status doesn't mean anything.

AFAIK he'd been looking for a maintainer for this for a while, didn't find one, and so just marked it unmaintained. He's not doing it, nobody else wanted to, so it'd be irresponsible to leave it listed as maintained.

Sometimes you just say fork it.

FTFY. And vendoring a dependency is really just a hidden fork.

This is exactly what we all do every time we

  cargo update

I wish audit tools and corporate policies are smart

The ball is in the court of the enterprise purchaser mandating the auditing company and they don’t give a ** if your dev team has to hot swap a yaml parsing library

Been doing this software thing for a bit now… while what you say is true in that it is possible in some rare cases, I think it’s a bit of a straw man exception. Code is, in the general sense, unfortunately never complete, regardless of how complete it seems in its context.

The issue is that the context is always changing. It is almost like suggesting there is an evolutionarily perfect organism. At best, one suited perfectly well to the current context can exist, but that perfect suitability falls off the second the environment changes. And the environment always changes.

I totally get the sentiment, but I think the idea that any code is ever complete is unfortunately just not a realistic one to apply outside of select and narrow contexts.

It's incorrect description, though.

Sorry, indeed it's been about the other library. However, these issues are connected: dtolnay is a co-owner of yaml-rust too, so either way without him YAML in Rust has a https://xkcd.com/2347 problem.

You don't discuss why they do it or the downsides. "Google does this" is not good enough of an argument.

For the why, could be that the build would be dog slow if a network requests was made for every one of a bajillion dependencies. Or to avoid breaking the build if something is janked externally. Could be because C++ has no modules/libraries. Could be for the lawyer's sake.

Is any of that applicable? There are also other ways of vendoring Rust crates, e.g. mirroring them to an internal registry. That can have fewer downsides.

Looks a lot like a Linux distribution, for example, Fedora, which has the same policy to reduce maintenance burden.

This seems like a unique solution for a Google-sized programming enterprise, right? Google is big enough and has enough programmers that they can have their own internal ecosystem that looks a lot like the whole open source universe (other than not being open to the outside world).

I’m completely guessing here but this looks like one relatively technical reason Google keeps killing products.

Author here: the result is now that the debt got absorbed into a project that uses the fork exclusively. It’s less likely that folks will look for security issues. It would in my mind have been better for folks to use an abandoned shared library where at the very least if a security incident popped up we would have all been notified and dealt with it at once.

Vendoring a pinned dependency just looks like dependency theater. You can let it sit there as a pinned dependency and it will do just as much harm as if you vendor it.

I get the sense (only a sense because the author is a bit coy, but sibling comment by him seems to confirm) that this was more to appease these tools that look for “bad dependencies”.

Even if it's maintained, you don't know how good it is security wise.

Great movie- but didn't remember this acronym.

I came of age during the S and L crisis. Been alternating between increasing prosperity, magical technology, idiotic wars kept going by the side that wants might makes right against the side of people that want an ordinary and beautiful life, and scam based economic collapses, against a backdrop steady worsening of global warming and global biodiversity ever since. Most fascinating life, I hope my kids enjoy their time half as much as I have. Not without problems, but the people, the learning, the flowers, the maths and the holding hands, hard to imagine better, if we’d just get out of our own way.

Embarrassingly, was certainly old enough to be reading those same headlines / being familiar with housing crisis, guess I'm just not an acronym person.

Do you think that implication is correct? If so, how do we fix it? Of course, companies should pay for what they use, but how do we practically implement that?

Yeah, finding free labor isn't light work. It's not trivial to externalize costs to unfortunate people. But the entities that succeed in this subjugation are the ones that are rewarded with some substantial wealth and power.

No, there are issues in practice. I have seen many published libraries getting neglected by their creators. Unless the community is actively channeled in this case, the result is a set of lackluster forks by people who only make some random changes. As a potential new user, you don't know what to pick. In the worst case, you need to create another fork, because none of the existing ones fixed the issues you encounter. If you take that public, the mess only gets bigger.

This chaos can be reduced with a single "blessed" fork that actively accepts reasonable maintainance patches. There are fewer incentives for competing forks and dependencies (direct and transitive) of the original package can automatically benefit, too, because releases are made under the original package name. This is good because the neglect problem can be several levels deep.

There is the issue of manpower. There are two arguments here: one is that the bar for package takeover needs to be set such that community interest in a fork is high enough and the fork needs to be managed by the package registry owner for several reasons. One is that maintainers need to be appointed through the registry provider to keep the nodel functional. Maintainer appointments can be passed on after extended inactivity. The other argument is that a package registry provider puts themselves into a trusted position by default and has to actively maintain that trust. Thus, the burden of managing forks and fork maintainers shouldn't necessarily be seen as inappropriate.

think of it as an NMU (non-maintainer upload)

https://wiki.debian.org/NonMaintainerUpload

also, names in a registry should be "leased" not forever eternal perpetual immutable assigned to some guy

Two counter arguments:

The package registry entry isn't the package. At no time is the original creator deprived of their rights to their package. Any fork will have to respect whatever license the original creator assigned. Open source licenses are allow forking by design for a reason.

Also, this whole thing needs to rest on a transparent process that doesn't make takeovers easy. There need to be clear criteria when takeovers can even be initiated, like extended inactivity (think 6 months or more) or unresponsive maintainers when serious issues are encountered. Also, there is not much harm in a mechanism for reclamation by the original package owner.

Yeah, I'd put the blame for that on the tools / systems which consume and report this kind of vulnerability data. They (at least as far as I've seen) don't really differentiate between severities of vulnerabilities. On the one hand, a strong preference toward "just patch" for vulnerabilities can make a lot of sense (the more that mechanism is flexed, the better it moves when you really need it, and rapid integration of fixes reduces time for exploitation by attackers), but here it's applied to a maintenance notice that ought to be treated a little differently.

Nonetheless, I think RustSec's feed is working as intended, and I'd put the expectation on the consuming tools (including `cargo-audit`, which I believe is largely maintained by the same people that maintain the RustSec vulnerability database) to change.

Just a bigger copy-paste from Stackoverflow than normally.