How I write HTTP services in Go after 13 years

There's a name for this anti-pattern: "Stringly typed"

things have different costs.

Types limit you from making some mistakes, but it also impacts your extensibility. Imagine an enum with 4 values and you want to add 1 because 10 level deep one of the services need new value. How does it usually go with strongly typed languages? You go and update all services until new value is properly propagated to lowest level who actually needs that value.

Now imagine doing same with strings, you can validate at the lowest level, upper levels just pass value as it is. If upper layers have conditionals based on value, they still can limit their logic to those values

As a PHP developer I am frankly disappointed you think that we only do that with strings. I've got an array[1] full of other tools.

1. Or maybe a map? Those keys might have significance I didn't tell you about.

Every struct requiring its zero value to be meaningful is probably one of the worst design flaws in the language.

C++ constructors actually make the guarantee, but it comes with other pains

Except Username is not exactly the same as string, and that's important. Username is a subset of string. If they were equivalent, we wouldn't need to parse/validate.

The often misinterpreted part of DRY is conflating "these are the same words, so they are the same", with "these are the same concept, so they are the same". A Username and a String are conceptually different.

One of the major issues with a lot of the outdated concepts in programming is that we still teach them to young people. I work a side gig as an external examiner for CS students. Especially in the early years they are taught the same OOP content that I was taught some decades ago, stuff that I haven’t used (also) for some decades. Because while a lot of the concepts may work well in theory, they never work out in a world where programmers have to write code on a Thursday afternoon after a terrible week.

It’s almost always better to repeat code. It’s obviously not something that is completely black and white, even if I prefer to never really do any form of inheritance or mutability, it’s not like I wouldn’t want you to create a “base” class with “created by” “updated by” and so on for your data classes and if you have some functions that do universal stuff for you and never change, then by all means use them in different places. But for the most part, repeating code will keep your code much cleaner. Maybe not today or the next month, but five years down the line nobody is going to want to touch that shared code which is now so complicated you may as well close your business before you let anyone touch it. Again, not because the theoretical concepts that lead to this are necessarily flawed, but because they require too much “correctness” to be useful.

Academia hasn’t really caught on though. I still grade first semester students who have the whole “Animal” -> “duck”, “dog”, “cat” or whatever they use into their heads as the “correct way” to do things. Similar to how they are often taught other processes than agile, but are taught that agile is the “only” way, even though we’ve seen just how wrong that is.

I’m not sure what we can really do about it. I’ve always championed strongly opinionated dev setups where I work. Some of the things we’ve done, and are going to do, aren’t going to be great, but what we try to do is to build an environment where it’s as easy as possible for every developer to build code the most maintainable way. We want to help them get there, even when it’s 15:45 on a Thursday that has been full of shit meetings in a week that’s been full of screaming children and an angry spouse and a car that exploded. And things like DRY just aren’t useful.

There's a mistake many junior devs (and sometimes mid and senior devs) make where they confuse hiding complexity with simplicity - using a string instead of a well defined domain type is a good example, there is a certain complexity of the domain expressed by the type that they don't want to think about too deeply so they replace with a string which superficially looks simpler but in fact hides all of the inherent complexity and nuance.

It causes what I call the lumpy carpet syndrome - sweeping the complexity under the carpet causes bumps to randomly appear that when squashed tend to cause other bumps to pop up rather than actually solving the problem.

Go now has generics, so I'm confident some smart fellow will apply DRY and make it a generic ValidatedData[type, validator] type struct, with a ValidatedDataFactory that applies the correct validator callback, and a ValidatorFactory that instantiates the validators based on a new valdiation rule DSL written in JSON or XML.

...Easy!

It would need a getter here. Probably good to keep it immutable, if you want guarantees that it will never be changed to something that violates the username rules.

I'm not a Haskell developer, so it's possible that I misunderstood the original "Parse, Don't Validate" post.

If you are doing the validation inside of a constructor, you are still doing validation instead of parsing.

Why that would be considered validation rather than parsing?

From the original post:

Consider: what is a parser? Really, a parser is just a function that consumes less-structured input and produces more-structured output.

That's the key idea to me.

A parser enforces checks on an input and produces an output. And if you define an output type that's distinct from the input type, you allow the type system "preserve" the fact that the data passed a parser at some point in its life.

But again, I don't know Haskell, so I'm interested to know if I'm misunderstanding Lexi Lambda's post.

The problem is that pattern "fails open." If anyone on the team forgets to define an untrusted string as UnvalidatedString, the data skips validation.

If you default to treating primitive types as untrusted, it's hard for someone to accidentally convert an untrusted type to a trusted type without using the correct parse method.

Why? You can easily call NewUsername inside NewAccount for example, just return the error. Or did I misunderstood?

But if you then create a constructor / factory method for that struct, not setting it would trigger an error. But this is one of the problem with Go and other languages that have nil or no "you have to set this" built into their type system: it relies on people's self-discipline, checked by the author, reviewer, and unit test, and ensuring there's not a problem like you describe takes up a lot of diligence.

The problem there is that you lose the guarantee that the parser validated the string value.

A caller can just say:

    // This is returning an error for some reason, so let's do it directly.
    // username, err := parsers.NewUsername(raw)
    username := parsers.Username(raw)

    var u Username
    u = "<hello>" // Implicitly converts from string to Username

If you do that, people outside the package can also do Username(x) conversions instead of calling NewUsername. Making value package private means that you can only set it from outside the package using provided functionality.

I mean, you may end up just wanting something like,

    type UsernameError struct {
      name   string
      reason string
    }
    func (e *UsernameError) Error() string { 
      return fmt.Errorf("invalid username %q: %s", e.name, e.reason)
    }

This is fine for lots of different cases, because it’s likely that your code wants to know how to handle “username is invalid”, but only humans care about why.

I have personally never seen a Go codebase where you parse error strings. I know that people keep complaining about it so it must be happening out there—but every codebase I’ve worked with either has error constants (an exported var set to some errors.New() value) or some kind of custom error type you can check. Or if it doesn’t have those things, I had no interest in parsing the errors.

Implementation-wise, yes, but the interface you're exposing is indistinguishable from that of a parser. For all your consumers know, you could be storing the username as a sequence of a 254-valued enum (one for each byte, except the angle brackets) and reconstructing the string on each "get" call. For more complex data you would certainly be storing it piecewise; the only reasons this example gets a pass are 1) because it is so low in surface area that a human can reasonably validate the implementation as bug-free without further aid from the type checker, and 2) because Go's type system is so inexpressive that you can't encode complex requirements with it anyway.

The validation is not completely thrown away, since the type indicates that the data has been validated. I understand "parsing" as applying more structure to a piece of data. Going from a String to an IP or a Username fits the definition.

I push my team to use this pattern in our (mostly Scala) codebase. We have too many instances of useless validations, because the fact that a piece of data has been "parsed"/validated is not reflected in its type using simple validation.

For example using String, a function might validate the String as a Username. Lower in the call stack, a function ends up taking this String as an arg. It has no way of knowing if it has been validated or not and has to re-validate it. If the first validation gets a Username as a result, other functions down the call stack can take a Username as an argument and know for sure it's been validated / "parsed".

I read through that piece and strongly disagree with the premise that their insight is somehow at odds with leaning into the type system for correctness.

The legitimate insight that they have is that anchoring the state as close as possible to the user input is valuable—I think that that is a great insight with a lot of good applications.

However, there's nothing that says you can't take that user-centric state and put it in a strongly typed data structure as soon as possible, with a set of clearly defined and well-typed transitions mapping the user-centric state to the derived states.

Edit: looks like there was discussion on this the other day, with a number of people making similar observations—https://news.ycombinator.com/item?id=39269886

In Go, capitalized identifiers are exported, whereas lowercase identifiers are not.

In the example I gave above, clients outside of the package can instantiate Username, but they can't access its "value" member, so the only way they could get a populated Username instance is by calling NewUsername.

Which (in Java) were then abstracted away in... interesting annotations.

I used that pattern for a while but stopped using it. I first encountered it from this blog post: https://commandcenter.blogspot.com/2014/01/self-referential-...

It's a lot of boilerplate to create something that's not actually immutable. It also makes it harder to figure out which options are available, since now you can't just look at the documentation of the type, you have to look at the whole module package to figure out what the various options are. If one of the fields is a slice or map you can just mutate that slice or map in place, so it's not really immutable. The pattern as Pike describes it has the benefit that supplying an option returns an option that reverses the effect of supplying the option so that you can use the options somewhat like Python context objects that have enter and exit semantics, but in practice I've found that to be useful in a small portion of situations.

I wrote a static config class that reads configuration for the entire app / server from a JSON or YAML file ( https://github.com/uber/zanzibar/blob/master/runtime/static_... ).

Once you've loaded it and mutated it for testing purposes or for copying from ENV vars into the config, you can then freeze it before passing it down to all your app level code.

Having this wrapper object that can be frozen and has a `get()` method to read JSON like data make it effectively not mutable.

I just use a struct literal, and then I have the type define a `func (t *Thing) ready() error { ... }` method and call the ready method to check that its valid. I prefer this over self-referential options, the builder pattern, supplying a secondary config object as a parameter to a constructor, etc.

Not the OP, but I mitigate the issue rather than use a different pattern. Like so:

type Server struct { val bool }

type Config struct { Val bool }

func NewServer(... config *Config ...) http.Handler { if config == nil { config = &Config{} } return &Server{ val: config.Val } }

It took me a long time to settle on this pattern and I admit it's tedious to copy configuration over to the server struct, but I've found that it ends up being the least verbose and maintainable long term while making sure callers can't mutate config after the fact.

I can pass nil to NewServer to say "just the usual, please", customize everything, or surgically change a single option.

It's also useful for maintaining backwards compatibility. I'm free to refactor config on my server struct and "upgrade" deprecated config arguments inside my NewServer function.

As you say, having a logger attached is one of those pragmatic and acceptable exceptions to the rule. In a perfect world we'd have the time to go to the trouble of implementing loggable types and data flows and associated higher order functions, in practice taking the compromise means getting the real business valuable work completed while still having the necessary (but usually "low priority") non-functional requirements like logging and metrics implemented.

I agree that too many arguments to the constructor may have the smell of too much coupling.

But if I really feel I can't avoid the need to pass a good amount of external context, I create a dedicated "options" struct and pass that into the constructor as a pointer. The purpose of the pointer (rather than pass by value) is if I want default arguments, I can pass nil.

    type ServerOptions struct {
        logger    *magic.Logger
        secretKey string
    }

    func NewServer(options *ServerOptions) (*Server, error) {
        ...
    }

Is this a Go thing? In C# land this is trivial.

Basically, not all the handlers will use every dependency the server (which is the entire program in this pattern) has. Not every handler will use a database, for example.

While I may prefer a struct for this instead of separate arguments, I do agree it's useful to capture "the world" as the set of all dependencies, even if some handlers don't use them (yet).

That's a nice way of putting it. When exploring a new codebase for the first time it can be very helpful to have main.go give you a high level idea about the overall structure of the program.

If you have ever topologically sorted 100 components connected in a complex graph by hand or found the right spot to insert the 101st, you'd quickly appreciate more help than a compiler check.

Its a bit icky but LLMs / copilot can speed up the creation of openapi specs a lot.

Agree it doesn't fix the "root" problem that the overall syntax is not ergonomic.

it lacks flexibility but i really enjoy grpc-gateway for 99% of my work

https://github.com/grpc-ecosystem/grpc-gateway

This is kinda missing the point; each handler needs a lot of deps to do it's job, and the most obvious place to put them is in the parameters of the function. That is what I want. I do not want more indirection for aesthetics; I want clarity, even if it's brutal clarity.

Whether all the deps are in the method receiver (the parent struct) or in a struct that's a param; it's all just more indirection to hide all the "stuff" that we need cause we think it's ugly. I dream of a world where we don't do that.

out of curiosity, why no sort-of-established pkg and internal dirs? What do you think of https://github.com/photoprism/photoprism structure?

Dreamhost too. Dreamhost will let you run a continuously running process.

The amount of work you can get done on low-end shared hosting is really quite impressive.