Architecture.md (2021)

Random thought: Every IDE I've used gives me the folder structure of the project on the left as a standard directory tree. Does any support navigating a project as a graph of dependencies?

This approach sounds great as a low-maintenance model for open-source projects with many ad hoc contributors. For projects with dedicated engineers, consider ADRs instead. These require more maintenance, but capture the "why" and "alternatives considered", which can be immensely helpful when rearchitecting.

See https://adr.github.io/

If you maintain an open-source project in the range of 10k-200k lines of code, I strongly encourage you to add an ARCHITECTURE document

I like this idea but IMHO, regardless of repo size, architecture can still has some place in a Readme. For example, I purposely placed a Mermaid sequence diagram[1] in the main Readme because I think it's important that all readers see and understand its workflow[2]

[1] https://mermaid.js.org/syntax/sequenceDiagram.html

[2] https://github.com/hbcondo/revenut-app?tab=readme-ov-file#-w...

One of the lessons I’ve learned is that the biggest difference between an occasional contributor and a core developer lies in the knowledge about the physical architecture of the project. Roughly, it takes 2x more time to write a patch if you are unfamiliar with the project, but it takes 10x more time to figure out where you should change the code.

Here here! This sounds like such awesome advice.

I wish we had better tools to visualize architecture of running systems. It's crazy to me that reading the code or a markdown file are still so state of the art. Maybe if someone's fancy they'll have some nice Mermaid diagrams. I want the architecture to be able to show itself live. Broadscale macroscale observability, baked in.

I think this would help everyone be able to appreciate & grok computing much more, would help humanity augment itself.

My experience was that on every project I was onboarded I was shown such an architecture diagram with a brief explanation of its components.

Now I'm surprised how uncommon this is in open source.

I've always found this to be a very useful practice. Many projects have a few core files (or packages / modules / whatever) where most of the changes happen. Being able to familiarize new contributors (or old returning ones) with those quickly really helps the startup time on a project.

I've added architecture files to projects at multiple jobs now [0], [1] and they've been well received. They're not perfect, but they're better than nothing.

[0]: https://github.com/zapier/zapier-platform/pull/324

[1]: https://github.com/stripe/stripe-cli/blob/master/ARCHITECTUR...

A DEVELOPMENT or HACKING document should be enough for contributors, to cite the coding guidelines. When someone wants to contribute without understanding the code workflow, he should not contribute.

One particular aspect of project architecture I often see people do wrong: a failure to have clear dependency structure between directories (and, often, too much stuff within a directory). This is particularly common if there is a directory named something like "common", "util", or "misc" (I am not actually arguing against having such directories, only noting they are prone to confusion).

I developed the following rules, which can be automatically enforced if you explicitly write the ranks (don't try implicit ranking; that means you won't get sane errors when you violate the rules):

0. Every directory has an implicit dependency on all its contained children.

1. A directory that does not depend on any other directories has rank 0. It suffices to only use rank between sibling directories, and it's probably simplest to maintain, though global ranking does work.

2. A directory that depends on others has a rank of 1 + the highest rank among its dependencies. To ease refactoring you could loosen this to "has a rank that is strictly greater than the highest rank among its dependencies".

3. Thus, circular dependencies between directories are forbidden; refactor (preferably, by splitting directories; most projects are too merged already) until you have a DAG. (circular dependencies between files in a single directory are allowed, subject to language-specific caution)

4. Thus, it is forbidden for a subdirectory to depend on a parent (or ancestor). If you encounter this, move the relevant files to a new subdirectory (since depending on a sibling or uncle is okay; if rank only applies between siblings this means the parent has to add a dependency on the uncle so that it gets the correct rank).

5. Depending on a cousin (or nephew) directory should be treated as a dependency on that cousin's parent, though (depending on what amount of directory structure your language forces you to use) it may be a hint you're doing something wrong.

6. Each directory can produce at most one library (shared and static count as the same library) or executable (at least, user-facing ones; code generators and tests might not count). Note that for other reasons it's generally inadvisable to ship multiple shared libraries or multiple static libraries, though you might use them during development.

7. (YMMV) If a directory contains any generated files (or their inputs), it should not contain any other files. Note that there are at least 3 major workflows for generated files, so the details will vary, but isolating them is useful regardless.

Note again: this is both coarser and finer than build dependencies - we treat directories as units, but add conceptual dependencies. As a general rule, I find it useful to define that the client (which calls `connect`) depends on the server (which calls `listen` and `accept`), and/or the data consumer depends on the data producer. Admittedly I have not deeply considered the case of servers that are worker-like, but note that it is often still possible to satisfy both by splitting directories further.

Note that additionally defining a "weight" (1 + weight of dependencies), although possible, is not particularly useful at the directory level. Long chains are easier to understand than tangled messes, but have a higher weight, and we don't want to discourage splitting a directory into a chain.

=====

In my experience, a project of about 100 kLoC organized itself about 20 immediate subdirectories of src/ with a max rank of about 10. This was roughly follows:

0-1: core support for the language, compiler, and replacements for the standard library; these almost never change. If somehow you have other kinds of code that doesn't depend on these, consider artificially inflating their rank to at least 2. About 1 each.

2-3: most fairly-project-agnostic (but not polyfill-like) "common"-like directories. About 2-3 each, probably.

4-6: semi-project-specific common stuff, most executables/libraries. IME generated files tend to belong here, and may be responsible for splitting a directory into a chain of length 3. There are a lot of these, usually with 1-2 dependencies of the previous rank and several more of the rank before that; likely there is nothing that depends on the entirety of any previous rank.

7-9: dependencies of the most complicated executable. About 1 each, a single chain at this point, though they might still have low-rank dependencies. At this point there's a decent chance that a directory might depend on the entirety of rank 2.

10: the most complicated executable (at least for me, this was the executable that had a conceptual dependency on other executables, even if it didn't have a source dependency. If you have a simple executable that conceptually depends on a complicated one, the simple one would be the highest rank instead). Note that only one of its dependencies is a 9; the rest are in the 3-6 range.

An alternate structure with deeper nesting, which I considered but never bothered to implement:

  src/base/ - the directories ranked 0-1 above
  src/lib/ - everything that either is part of a shared library, or is used by multiple executables (all directories ranked 2-3, and some ranked 4-6)
  src/each-executable/various/ - dependencies of a particular executable (including all directories ranked 7-9 above)
  src/each-executable/main/ or just src/each-executable/ - the file containing `main` and as little else as possible (otherwise we're likely to confuse "code closely related to `main`" and "code that is executable-specific but not involved in many dependencies")

0: scripts/ - executable scripts used during the build that don't have to be built, or that might be installed with no more than a shebang update.

1: src/ - all source code, including that for tests and tool/

2: tool/ - built executables needed for later parts of the build. Refactoring to split this out from shebang scripts is nice for your sanity, but may be noisy, especially if your build system's dependencies are sloppy. In contrast to build/ these are never cross-compiled.

3: build/ (all other output (data and potentially-cross-compiled code); contains bin/, lib/, and share/ at least)

Hopefully, projects that get much bigger than this can be split into further subdirectories so that the dependency ranking only need be done separately within them. I'm not sure if any big project is actually that nice in whole, but you should at least be able to create sanity in part without too much distress.

That said, keep in mind that this is just one approach to this particular problem, and it is just one architecture-related problem. Particularly, if your tooling makes it difficult to split directories (or files for that matter), fix your tooling first (related: "recursive make considered harmful")

I experimented with something similar in one of my larger side projects a couple of years ago:

https://github.com/shipmight/shipmight/blob/master/src/ARCHI...

At the top of each file there was a tree of links to other ARCHITECTURE.md-files in the repo, like this:

    * ARCHITECTURE.md <- you are here
    * backend/ARCHITECTURE.md
        * backend/api/ARCHITECTURE.md
        * backend/cli/ARCHITECTURE.md
        * backend/ui/ARCHITECTURE.md
        * backend/utils/ARCHITECTURE.md
    * frontend/ARCHITECTURE.md
    * internal-charts/ARCHITECTURE.md

I would be wary of extrapolating what the author is writing of here to general software projects. I think it makes a lot of sense on large open source projects, where there are many contributors with little context- it is worth the effort to maintain such a document in this case. But all the developer committed documentation I've seen on smaller work projects have inevitably become unmaintained.

I think that's just more documentation to read which becomes outdated (read: lies) at the point in time someone moves code around with refactoring tooling in their IDE.

What about aspiring to "screaming architecture" instead? Don't hide your application domain in a "crates" directory. Do it the other way around.

Rip_hoan903

I’m 100% on board with this. Even if it’s just a see docs within the repo. I should be able to understand the intent of things without having to go read your website (should you’d remember to write one).

ASCII diagrams of components or your draw.io diagrams would go here. Knowing what you have is half the battle.

I love the idea here, but i loath the example. TMI. Give the core idea, any communication points (sockets, apis etc), maybe an abstract diagram.

I was a fan of all these little docs/diagrams-as-code standards:

  - README-driven development
  - ARCHITECTURE.md
  - ADRs
  - arc42
  - C4
  - etc.

Instead of using somebody else’s standard, I just organize and refactor docs as I go, in the same way as I manage my personal notes in Obsidian.

Initially I wanted to use a common subset of Markdown that will work both in GitHub (GFM) and Obsidian, but then I gave up, and just use Obsidian flavor of markdown with all its proprietary features like Dataview plugin, templates, etc.

Mermaid and LaTeX is built-in in Obsidian, and there is a plugin for PlantUML.

For visual drawings/diagrams there are builtin Canvas, DrawIO and Excalidraw.

This sounds like it's suggesting adding documentation of the major units, what their purposes are, as well as their interfaces.

I agree with this. A map of the code is great. It's like an exploded-view drawing of a mechanical component. It helps highlight what goes where and connects how to what.

...but is this what "architecture" means? I was under the impression that architecture went beyond major-units-and-their-interfaces and had more to do with the decisions and assumptions that lead to those specific units and interfaces -- the why behind the what.

In other words, the "architecture", in my view, is the thing we might go against when we refactor things. Not because names and interfaces change, but because the rationale for having things a certain way might still apply only we don't know about it because those assumptions and decisions -- the architecture -- is rarely documented and still would not be under this proposal.

I try to write docs like this. (Not markdown files, heaven forfend, and definitely not a random file in the root, of course, but these are quibbles.)

A problem, though, is that many things lack an architecture; they were grown and are a mess, and the procedural knowledge can't be put to a document without just pointlessly recapitulating the code itself in what is possibly a more confusing form.

  the shorter it is, the less likely it will be invalidated by some future change. This is the main rule of thumb for ARCHITECTURE — only specify things that are unlikely to frequently change. Don’t try to keep it synchronized with code.

I agree it is the difficulty in grokking a codebase. "Pattern" naming sort of helps, but I end up having to read a far bit.

On github, I always keep thinking the commit messages for each file are descriptions. Would that be more useful?

I don't know, man. I think README.md works just fine.

Discussed at the time:

Architecture.md - https://news.ycombinator.com/item?id=26048784 - Feb 2021 (153 comments)

In my company we called it `COMPASS.md`

Compass cause it helped you navigate. It's short. Every 'module' needed to have such a file. It was also enforced in the CI.