If you’re building an abstraction-heavy API, be prepared to think hard before adding new features. If you’re building an abstraction-light API, commit to it and resist the temptation to add abstractions when it comes along.
You could always do both.
Provide a low-level abstraction-light API that allows fine control but requires deep expertise, and write a higher-level abstraction-rich API on top of it that maps to fewer simple operations for the most common use cases - which some of your clients might be implementing their own half-baked versions of anyway.
If you maintain a clean separation between the two, having both in place might mean there is less pressure to add abstractions to the low-level API, or to add warts and special-cases to the high-level API. If a client wants one of those things, it already exists - in the other API.
Bonus points for providing materials to help your clients learn how to move from one to the other. You can attract clients who do not yet have deep knowledge of payment network internals, but are looking to improve in that direction.
It does double your API surface area, so that's the tradeoff you'll have to consider. It can be the correct decision in a lot of cases.
It doesn't double the security surface area if the abstracted API goes through the low-level API. The outer one is just chrome, and so the risks of screwing something up there is far lower.
Unless you're using a trash language where even simple wrappers could buffer underrun or something.
Isn't there the issue of modifying the state enough through the low-level API such that it breaks the assumptions of the high-level one?
Just say that you don't support mixing both APIs.
Now you’re telling users to check the source code of any tools they use with your stuff? “We recommend using ToolOne only on content you do not also manage with ToolTwo”
See how libfuse handles it with their low level and high level APIs.
That’ll fix it. ;-)
Yes. But that's the high-level API's problem. That's a problem with any abstraction really. "What if there's something in the thing we're abstracting that doesn't fit the abstraction" isn't really a problem with the "two API" approach, it's a problem with abstraction.
The high-level API needs to handle that case, if nothing better than having internal assertions that throw if it hits a case it's not designed to accommodate.
(also I'm annoyed with myself that I wrote buffer underrun in my first post instead of buffer overflow and now it's too late to edit).
You did make me search what a buffer underrun was and I think it was a good read
The high level API shouldn’t care about state. In other words, your “readers” should merely aggregate state and your “writers” should only care about subsets of state.
Think about file permissions in Linux. Running ls just shows you gross file perms (current user, group, and global) but you can also grant access to other individual users, or even make a file immutable that still shows up as writable to ls. The high level api doesn’t know or care about the low level state except where it is relevant.
Unlikely to double. The low level API exposes all capabilities, the high level API exposes a subset of those capabilities under a smaller surface. The high level API will not be as large as the low level.
This reminds me of the Kubernetes API.
This reminds me of the git API.
This. There should be a low level API to be able to do rarer more complicated cases, and a higher level simple API for common cases built on the lower-level API.
Just today I was working with the Web File System API, and e.g. just writing a string to a file requires seven function calls, most async. And this doesn't even handle errors. And has to be done in a worker, setting up of which is similar faff in itself. Similar horrors can be seen in e.g. IndexedDB, WebRTC and even plain old vanilla DOM manipulation. And things like Vulkan and DirectX and ffmpeg are even way worse.
The complexity can be largely justified to be able to handle all sorts of exotic cases, but vast majority of cases aren't these exotic ones.
API design should start first by sketching out how using the API looks for common cases, and those should be as simple as possible. E.g. the fetch API does this quite well. XMLHttpRequest definitely did not.
https://developer.mozilla.org/en-US/docs/Web/API/FileSystemS...
Edit: I've thought many a time that there should be some unified "porcelain" API for all the Web APIs. It would wrap all the (awesome) features in one coherent "standard library" wrapper, supporting at least the most common use cases. Modern browsers are very powerful and capable but a lot of this power and capability is largely unknown or underused because each API tends to have quite an idiosyncratic design and they are needlessly hard to learn and/or use.
Something like what jQuery did for DOM (but with less magic and no extra bells and whistles). E.g. node.js has somewhat coherent, but a bit outdated APIs (e.g. Promise support is spotty and inconsistent). Something like how Python strives for "pythonic" APIs (with varying success).
Tbf Vulkan is not intended for an endprogrammer. It is a deliberately low level standardization to allow directly control GPU hardware. The high-level approach (OpenGL) failed. The endprogrammer is supposed to use a third party middleware, not Vulkan itself.
So OpenGL is a total failure! I learned something today… What should I use?
I like WebGPU: it's slightly higher-level than OpenGL, but the shader language is better. (For the actual web, use WebGL: like Wasm, WebGPU isn't actually suitable for use in websites.)
Someone has to write that middleware…
It's 4 async calls, and it can be done entirely in the main thread so long as you use the async API. https://developer.mozilla.org/en-US/docs/Web/API/FileSystemW...
For OPFS it's still 6 calls. And the async API doesn't support flushing writes.
https://developer.mozilla.org/en-US/docs/Web/API/File_System...
My favorite quote about abstraction is from Edsger Dijkstra:
”The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.”
If only the “A” in API stood for “abstraction”. For many APIs, it probably stands for “accreted”. :)
That's a great quote. This topic turned up a couple of weeks ago in another thread. [0] The excellent talk Constraints Liberate, Liberties Constrain, by Runar Bjarnason, gets at the same point, and even uses this same Dijkstra quote. [1]
[0] https://news.ycombinator.com/item?id=40021696
[1] https://youtu.be/GqmsQeSzMdw?t=874 (this takes you right to the Dijkstra quote)
Problems can sneak in when you use the low-level API to do something to an object that can't be cleanly represented in the higher-level API. You need some kind of escape hatch, like a list of links to or ids of low-level details or a blob (Map<String,arbitrary-JSON> of miscellaneous data that can hold the low-level additions.
Hopefully the top-level important concepts like "amount_due" will still reflect the correct numbers!
Those problems usually present themselves by people overthinking the high level api and trying to be smart.
As an example, you can use chattr to make a file in Linux immutable. ls still shows that you have permission to write to the file, even though it will fail.
When people try to overthink the api and have it determine if you really can write to a file, people will try using the high level api first (chmod) and it won’t work because it has nothing to do with permissions.
KISS is really needed for high level APIs.
Git is an example of this. [1]
There are high-level "porcelain" commands like branch and checkout.
And then there are low-level "plumbing" commands like commit-tree and update-ref.
[1] https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Po...
Came here to say that.
Also, to some extent, Emacs. There are thousands of functions (actually, a bit less than 10k in stock Emacs without packages, and over 46k in my Emacs) performing various low-level tasks, and much fewer commands (~3k in stock Emacs, almost 12k in my config), i.e., interactive functions, often higher-level, designed for the user.
.NET also does this a lot. Here's a recent devblog post looking at file I/O: https://devblogs.microsoft.com/dotnet/the-convenience-of-sys...
That's a fantastic blog post, worthy of it's own HN submission.
I'm particularly fond of this pattern when you can implement the high level API that you want outside the library, which ensures that your low level API is sufficiently flexible and means you're dogfooding your own stuff as a user. It's far too easy to get used to the internal side of a tool you're building, and forget how people actually use it.
It is also important to guarantee that the two API designs are coherent and interoperable, and this kind of strict layering is the best strategy to avoid mistakes.
+1, I like to refer to this post on this topic: https://blog.sbensu.com/posts/apis-as-ladders/
“Make the easy things easy and the hard things possible”
I like this idea a lot.
One level of API for implementation model.
And second level for mental model.
That sounds like the exact philosophy the Vulkan devs took versus OpenGL.
matplotlib seems to have implemented this approach
Compare also exokernels, and how they delegate abstraction to libraries, not the OS.
I’m genuinely curious as to what an abstraction-rich api would look like and why it would be useful.
I’ve mainly worked in enterprise organisations or in startups transitioning into enterprise which is sort of where my expertise lies. I’ve never seen an API that wasn’t similar to the examples in this case.
I mean… I have… but they wouldn’t be labelled as high-abstraction api’s. If they needed a label it would be terrible APIs. Like sending table headers, column types in another full length array with a line for each column, and then the actual data/content in a third array. Even sending the html style that was used in what ever frontend, so that some data is represented as “some data” and other is represented as [“some data”, [“text-aligned”, “center”…],… . Yes, that is an actual example. Anyway I’ve never seen a high abstraction api and I feel like I’m missing out.
Another example of this is AWS CDK. There are a few “levels” of constructs - high level ones that are simpler and apply “presets” that are enough for 80% of users, but the core low level ones still exist if you have a nonstandard use case.