Pains of building your own billing system

I worked on a billing and AR system for a small pharmacy chain years ago. Our company would get pretty large random checks from ins carriers totally out of the blue. As in, we had billed them nothing and they would send us $150k checks in snail mail. They wouldn’t realize their mistake and come asking for it back for months. We had a special account named “magic money” for it. The billing world is crazy town.

This is a great article for people who are in the situation where they have to make decisions about billing and don't have much experience (and as a handy reference for those who do, but want some reminders, too.)

However, I will admit, I had a hearty chuckle at this line:

"why can’t we just dump a file of what we need to bill on S3, and have a CRON job pick it up and collect payment?"

Under no circumstances does my engineer brain think this is a good idea. At all.

But, I will dump one aspect of my engineer-brain thoughts: My favorite "billing architecture" decision is to try to decouple billing as much as possible from credit in a system. For example, if you have a subscription system where the user pays ahead for a given billing period, I prefer to have the entitlement itself just store the expiration date and the details about what entitlements the subscription grants during the time period it is active. The billing system can store the subscription and sync back to the entitlement as-needed. This makes both manual billing by human operators (not to mention debugging and patching around momentary issues) and something like a Stripe integration very easy. You should, of course, be very careful to leave it open for extension in the future, but this seems to be a very nice decision that doesn't, in itself, limit you too much.

Obviously, this is not my original idea, but it's still something I've grown to like a lot, especially after having tried other things less successfully.

What are y'all using for entitlements? Do you use your feature flag system? A different saas like Stigg? Or a separate internal system?

I was just coming to hn this morning because I wrote about using FF for entitlements: https://prefab.cloud/blog/modeling-product-entitlements-with... I took some inspiration from another of Arnon's posts about SKU format for the post.

FF don't seem like the perfect place for entitlements, but in my experience they're often the best tool at hand to deal with the challenges. I'd love to hear alternative opinions.

AFAIK, in EU countries you cannot build/run "your own" billing system. Unless you have a special type of company/enterprise that is accredited and registered specifically for this purpose. This is officially named "Payment Processor".

I'd argue that the pains of building a billing system are not the right approach to the topic. If building your own billing system is a path so much sought after, well, let that be.

Billing systems are of high complexity; I recognise that. However, if Chargebee, Solvimon, Stripe, Recurly, Orb, Metronome, Lago, Togai or anyone else has that body of knowledge, we could instead collect that knowledge in one place.

Indeed, there's no better approach than the one that serves you. If you're a subscription-based SaaS, you have specific solutions for your business. If you're a usage-based API, you have specific solutions to the billing.

But we could have all that knowledge, approaches, paradigms, programming patterns, better and best practices in one place, instead of discouraging the practice. There are also edge cases where a company is not U.S.-based or European, and a billing solution like Stripe wouldn't work, e.g. your company is based in Venezuela, and you can't have a Stripe account. What do you do in that case? You must forcefully build your own billing solution and connect it to the local payment gateways with their arcane SOAP-XML APIs.

--

On a separate note, "building your own billing system" reminds me of the topic of "rolling your own SIEM" with the typical Elastic + Grafana setup.

I don't recommend it, but I understand why it's such a hot path for an IT Security department to do it.

I don't know about this advice, you ain't gonna need half the features it talks about, and you absolutely can build it out gradually as your business grows.

You do need to understand the concepts of invoices, credit, tax periods, pro-rata billing changes and so on... but all of that knowledge can be used to make an informed decision on build vs buy, rather than an automatic reason to outsource.

The only external API you need for software-as-a-service is a credit card processor, two if you're fancy. Sure after the first year you will probably have a bunch of manual work to do and your accountants will tell you the dumb stuff you've done, and you'll learn a lot about accounting :)

(I would still shop around to start a new business today, but with the confidence that building isn't very scary)

My rule of thumb: Always ask about rounding. If they don't have a sane answer, it's basically guaranteed that their system is broken in countless ways. Pretty much the only sane answer: Until you realize that every jurisdiction, bank and organization you interact with financially can and might round pennies differently, you don't understand financial systems enough to implement one. i.e. rounding is configurable or you are doing it wrong.

You should never build your own billing system because you do not want to do the global VAT work.

itch.io is a good alternative if you sell digital assets and even services as you can verify payments with mail.

For such systems that are commonly used by many, it’s best to open source it. Don’t reinvent the wheel.

Worked for a large travel website that rolled its own billing...

Absolute nightmare when we blew up, had to hire 6 people to beef up the billing side of the website and it definitely cost us way more than those 6 people due to mistakes.

I ended up working on parts of it for our customer service portal, lots of band-aides applied in a rush as we scaled.

The 14 pains of crossing the street and why you want to pay me to read my book.

Many people think it is a good idea to close your eyes and just start walking, resulting in untimely injuries or even death.

1. Countries and even cities have different rules for crossing the street. 2. You may not know this yet but you need to look both ways. 3. Many people do not realize they are color blind, resulting in death and injuries because they cannot tell red from green. 4. You may look both ways, but do you look up as well? In cities people can drop things from windows, in the country birds may fly over head. More and more space junk is falling to earth. 5. Thing may come at you from multiple directions. 6. etc.

Not saying this article is that bad, but I would appreciate an article framed as a how to. I am much more likely to read articles that are not framed negatively.

I built my own hybrid billing system and yes it was painful, but it opened the door to more payment methods and allowed prorating across the different payment methods seamlessly. I think it increased my revenue, but I can't know for sure because maybe everyone would have just used credit cards if that was the only supported payment method.

2 problems not talked about here that are things I've had to deal with:

1) Month/Quarter close. While this post talks about the account ledger, when you start dealing with a public company that has to report numbers, you have hard-cutoffs and have to make sure everything goes smoothly for month or quarter close.

2) Cash-in-transit accounting: this post assumes your payment processor is 100% correct and nothing goes wrong. When you get big enough, you need to be able to match any money that lands in a bank account with a given invoice/billing entry. You need to be able to detect any invoices that do not have a paired bank statement line. And as others have called out, the reverse can happen where you get paid for something that may not have an invoice item associated with it. Being able to deal with credits on a bank account not associated with a billing invoice can be equally important.

Maybe you can say that these are all accounting departments problem, but they are tightly coupled and the 2 teams need to work together that there are no discrepancies between their books.

I know the founders of bunny.com (which tries to solve this very problem). They sold a different b2b startup after 10 years and billing was such a pain they created a whole nother startup just to try and solve it. Lol.

I'm sure the person that invented mantras like "seperation of concerns" and "single responsibility principle" did so based on their trauma of building a billing system.

The biggest mistake I see companies making is not properly appreciating the fact that there is in fact 4-5 entirely independent business processes going on inside that monolithic concept they refer to as a "billing system", and they REALLY need to be kept seperate.

IMO the fundamental seperation that is absolutely critical to make is distinguishing between your "entitlements system" that tracks what SKUs you offer and which customers have what SKUs (and in what quantity), the "accounting system" that takes input from the entitlements system and tracks the over-time net balanced owed by the customer, and finally the actual "billing system" which periodically takes the balance from the the Accounting system, zeroes the account back to nil, and generates an actual point in time "bill" for the customer to pay.

So many systems don't properly respect these boundaries and suffer the unbelievably painful consequences.

On the other side of this, understanding usage based consumption billing as a customer is very tough. First got exposure to this working on https://ec2instances.info.

I don't buy the overall structure of this argument: system X is very very complex, so you should never try to do it yourself, instead use an off-the-shelf solution for X.

Perhaps the general case of X is enormously tricky and complex, but in my use case I only need to handle a specific subset of the complexity. Therefore I can build my own solution that only handles the complexity I need, and it will be much simpler than off-the-shelf tools.

I absolutely adopt this stance for X=datetime. My approach to datetime requires two function calls to be provided by the library: convert an epoch time to an ISO formatted time string in a particular TZ, and the inverse. I never touch any other library code; I do all other time manipulation in my own code, in terms of those two functions.

I don't know what this person is talking about, our billing was brain dead simple, it's only got like 20 lines of Stripe Billing code. :-)

But in all seriousness, we looked at Stripe, thought about "aw man is it worth spending the .5% extra" and then spent about 30 seconds thinking about how much time it would take to roll our own, and it was an absolute no brainer.

(replace Stripe with any of the other similar competitors, not trying to be too biased, but stripe is kind of the mind-share default for SaaS startups, aren't they?)

I'm on the billing team and we just shipped our new platform send me your regards!

This is why I'm so disappointed with Stripe.

I found the documentation both overly verbose and lacking, so it took me longer than it should have to understand basics like subscription item ids and the 50-100 id types which people have collected into lists. Then there are simple bugs like how the go live button only shows in test mode, with no simple way to copy live mode data back into test environment(s), so developers have to manually export/import stuff that business people set up in their attempt to save time. The most basic features are missing, like a way to copy any object as json from the browser dashboard, forcing us to look down in the logs/events to copy the latest version, often with wildly inconsistent/unpredictable formats.

And Stripe does little to support actual real-world use-cases, for example: subscription events come through webhooks ok, but subscriptions have a state like "active" or "incomplete" (meaning that payment hasn't gone through yet), causing subscriptions to become stateful. Meaning that instead of a user being subscribed (yes/no), the app has to consider additional criteria and edge cases around missing or lapsed payments. And there doesn't appear to be a synchronization mechanism for eventualities like a backend server being down for a day, causing events to be missed. Stripe does a best-effort resend some low number of times, but then the events are lost. It's up to the developer to diff the backend's database with the Stripe subscription list and then sync individual subscriptions manually.

These are all just exactly the kinds of bugs/features I predicted would be there before I used it, which is why they felt like such a slap in the face. I suspect that there are conceptual shortcomings throughout nearly every service provided by Stripe, and I'd be over 50% confident that any ones I predict here would be found upon first use. These are artifacts of favoring agile over waterfall for formal engineering challenges.

Basically what I'm saying is that I consider the Stripe API to be what an MVP payment processing system might have looked like back in the 1990s under older paradigms like SOAP/XML. But nobody has created a wrapper for Stripe yet that works how people expect, more like Venmo/PayPal that maps customer use-cases to CRUD operations.

I thought that maybe Laravel Cashier would do some of the heavy lifting, but it appears to be just a thin wrapper over Stripe, with its own set of oversights.

I know that everyone uses Stripe and apparently loves it, and I applaud its efforts and accomplishments around reducing the friction of payment processing. But I can't help but feel that there is an element of having to drink the kool-aid here. I would still recommend Stripe though, so this rant is directed more at its developers who may not be aware of these issues. Stripe would do well to perform some new user testing like Apple used to do when designing its human interface guidelines, to enumerate the pitfall(s) in each step of Stripe onboarding.

I built one for the government keeping track of licensed professionals and receiving their payments for things reported in the field (mobile first web app). It collected over a quarter million dollars in 30-40 dollar payments with different tiers, refunds, overrides, and even penalties for non-payment all while being PCI compliant. One thing that helped a lot- create an auth system that lets the admin impersonate the customer to walk them through it. It’s a tough paradigm to start with but pays off immensely. Another was generating excel files and reports on demand from any view of the data in the app. One of the developers on the project implemented a simple state machine for payment histories and stored it in a number of tables with FK constraints. Do not do this! That means in the future your app will need to deserialize every customer’s history and after a few years the app will grind to a halt. This was the one issue with the app looking back. A state machine looks like a good fit for billing but if my future self could go back in time and warn everyone it would be with this one common problem or I wouldn’t even make this comment in the first place. If you do consider using a state machine just create thousands of customers with long detailed histories up front and if you can load them all up quickly then that is a good sign. Your billing system will only perform as well as you can transform and index data from customer history in bulk.

Can't be worse then needing to change a SAP system to work with the business processes than instead what is the cheaper and quicker way of changing the processes to better match SAP.

Sometimes you had to reimplement whole systems that SAP comes out of the box with. Thanks not to be named fashion brand.

My first job after dropping out of college the first time was working for a Tae Kwon Do management firm, writing software to help people run martial arts studios. The job had me doing ColdFusion and Adobe Flash [1].

Most of my job involved working on their marketing pages, but at some point they had me work on the credit card processing platform, and I've sort drawn a soft line in the sand that I won't do that ever again.

Part of it was just that the code was really messy (it was ColdFusion after all), but a lot of it boiled down to having to deal with the million edge case conditions required to achieve PCI Compliance. Somehow, that company had managed to get a PCI Compliance label, and I have no idea how, because the code was held together with duct tape and prayers; there were hundreds of nested if statements, and if's nested inside else nested inside other ifs.

I'm not saying that it's unnecessarily complicated, but I know I'm not smart enough to deal with it.

[1] I'd like to point out, this was 2012; it was considered kind of outdated even at the time!

(killbill.io co-founder here). Building a billing system is indeed quite complex, and for a variety of reasons: For once thing, most departments within the company will be impacted (finance, accounting, product, engineering, legal, ...), so the cognitive load on the team building the solution is quite high, and requirements are usually poorly defined. Then, as mentioned in the comments above, there are tons of complicated use cases and edge cases (time zones, pro-rations, entitlement v.s. billing, in-arrear/in-advance models, usage-based, credits, coupons, tax integration, refunds, chargebacks, and on and on ...). Another aspect is that once you have built the billing system, this is just one part of a bigger quote-to-cash picture, and the solution needs to fit within the rest of such platform, so it needs to be made flexible enough to integrate with such systems. I would typically not recommend building your own unless you have acquired lots of knowledge over the years, and have a motivated team in place to execute on it.

"Let your ERP handle your RevRec/Accounting (or use what’s built in with something else)."

With that approach, how do you:

1. Make your ERP recognize revenue correctly? An ERP system isn't magically going to know how your revenue should be amortized based on certain events, or what should happen when a partially-amortized order is later upgraded/downgraded to a different package/plan.

2. Ensure that your refund/upgrade/downgrade calculations are consistent between product systems and the ERP system.

Even if you emit all the necessary events (and each contains all the necessary data), there has to be some system that's programmed (or, at least, configured) to apply your revenue recognition rules to the event stream. Given how much difficulty people have even describing how things should work, I don't think you can just abdicate the revenue recognition to another system and just assume whoever configures/programs that system will do it right. Even if you can assume that, it's equivalent to saying "it's complicated so you shouldn't do it yourself; have someone else do it".

If you're building a product, is the billing component a part of your USP?

Really?

Odds are that it's not, and therefore you should farm out that work to someone who _does_ make it their job. There are reasons why companies implement software from Microsoft, Oracle, and SAP, including that it's better to reduce costs on things that don't differentiate you in your market (and it's nice to have someone to "pointedly talk at" when things go wrong).

One thing this article doesn't mention are affiliate sales which depending on how far you want to go can be a decent amount of work. This is the system that lets other people get commission based payouts for sales within your platform.

You need to track affiliate codes back to sales and a user, handle sending payouts to an affiliate at their configured payment provider and if you want to go all out then handle metrics around visitors and present a UI to the affiliate so they can see their conversion rates and payout history.

Fortunately most of that can be built incrementally. As long as you associate a unique code to a user and wire up associating that to a sale with a specific commission % amount everything else can be done manually or skipped. For example I pay affiliates out once per month where I goto Zelle or PayPal and send out the payments. It takes less than 10 minutes. There is no front end for tracking conversions and it's also never been a cause of someone saying they don't want to be an affiliate because of that.

I worked on a fairly complex invoicing system and couple of things not (directly) mentioned:

  * Forward billing vs billing in arrears. We had to treat different customers differently. 
  * Tiered billing -- e.g., if a customer used > X amount of services, they got a discount.
  * Special rules around when billing starts for new customers (e.g., no bill for the first month). 

I think a lot of the problems which companies experience are all related to them not being able to find good software engineers.

The amount of money I've seen startups forking out annually for various billing and platform subscriptions would have been enough for me to build them a better system from scratch in under 6 months. The amount of money that companies waste on rents is ridiculous and it's sad because that money could have gone to skilled engineers instead of hotshot entrepreneurs.

For my own SaaS startup, I built the billing system from scratch, it measures every millisecond of CPU time and every operation used by each account and adds them to the current active bill. At the end of the month (or whenever), I manually trigger a process to close off active bills and the system will automatically display that one as due in the UI and will open up a new one and new usage will be recorded against that new active one.

The trick is to use a function which automatically figures out what bill to use based on a specific flag so you don't need to worry about the implementation details of which bill to record usage against. This function should handle all possible situations; including the few milliseconds of delay which can exist between the old bill being closed and the new bill being opened.

That's where idempotence can be useful as mentioned in the article; that way if the system fails to record usage against a new bill because (for example) that bill was already just created concurrently by a different process then it will retry again later and record the new usage stats against that existing bill.

The trick I use is to have deterministic IDs for bills that are based on the nearest whole unit of time in UTC (rounded up or down); the amount of rounding I use allows me to control the allowable delay between concurrent processes and determines the amount of pending usage data which is kept in-memory within each process before it is flushed to its bill in the database. If the chosen unit of time is a 'whole minute' then it means that it's not possible to generate two bills within 30 seconds of each other by concurrent processes. This is greater than my database update timeout so it effectively guarantees that two processes cannot accidentally create two bills for the same period. If a specific process took longer than 30 seconds to update their usage info, that process would reconnect and either realize that an active bill has now already been created by a different process or, if not, it would try to create it again with a new ID corresponding to the new 'whole minute'.

I have full flexibility to change my billing to any time interval I want. It doesn't even have to be the same for all users.

Anyway, as you can see, it's very simple.

Summary: just use picodollars!

I helped build a billing system for a smallish family business. It had cost overruns and was way behind schedule. A year or so after planned release and after bringing in some more senior leadership it was a huge success for them. Integrated their customers database with their new SaaS products and facilitated very unique workflows that a legacy business needed to get through a digital transformation.

Very unique circumstances but huge undertaking.

Whenever someone mentions building a billing system, I feel compelled to share Mark Dominus' delightful article, "Moonpig: a billing system that doesn't suck":

Sometimes I see other people fuck up a project over and over, and I say “I could do that better”, and then I get a chance to try, and I discover it was a lot harder than I thought, I realize that those people who tried before are not as stupid as as I believed. That did not happen this time. Moonpig is a really good billing system. It is not that hard to get right. Those other guys really were as stupid as I thought they were.

https://blog.plover.com/prog/Moonpig.html

There are actually quite a lot of solutions out there that fly under the radar but do a great job. Billsby for example has slowly crept up to becoming a significant player in the field by starting with a focus on one segment: recurrent billing. I can see a world where there could be multiple unicorns just in the billing sector of fintec.

What about using merchant of record solutions that pays taxes instead of you? It is a good solution for a self bootstrap startup, or it would be better to do it yourself?

Seems simple enough for a startup idea but prolly has been done thousands of times

I'm about to implement https://www.getlago.com/ on my SAAS. Spoke to the team, great bunch

The general rule is not to build anything that is not your core expertise.

why can’t we just dump a file of what we need to bill on S3, and have a CRON job pick it up and collect payment?

Sounds like an engineer with not enough experience. We all did thought like this once.

This is a solved problem. Don't roll your own queue / job system where you inevitably run into transactional problems. Use something like Airflow.

And mentioning S3 in this sentence has no value. S3 is merely a way to store the data temporarily. But using S3 immediately makes us not consider (more suitable) alternatives.

Host own email and host/run own billing is definitely on my list of things I'd rather not do

Each individual problem listed isn't that hard on its own. Some are even trivial. However there are so so many constraints one has to keep track of so system complexity quickly snowballs into something unmanageable. Each cog is simple, but the whole clock is insanely complex. And by the way, the clock can't stop and has to be running 24/7/365 while you keep changing it. Your code will quickly turn into a tangled up mess of stinky petrified spaghetti, so you will end up re-building your entire system from scratch every few years. And with each rebuild you will keep having the same trivial problems you already solved again and again and again.

Billing problems in general are a subset of problems one has in banking because banking system basically = billing + interest calculation + payments + accounting + multiple equally complex things and it has to work in real-time in a super-regulated environment while working in perfect sync with other systems you don't control.

If you do everything just right and manage to keep the can rolling down the street for sometime, there will be a moment when regulation changes or your business changes and your architects suddenly resign and you will end up with a mess which requires more engineering dollars than off-the-shelf system would.

Don't build your business on such foundation. It is XXI century already, no need to re-invent stuff which is readily available for peanuts.

Yet, every new neobank comes up with their own shiny corebanking ledger because they can't be bothered to look into somebody else's petrified spaghetti.

What really impressed me was the need to be able to run the entire Rube Goldberg machine backwards.

You really want to have "the order" as static as possible, for "the invoice" there is even legal obligation.

Then it goes something like...

I bought 5 and got a discount but I want only 3 and ohh there is a typo in my name and one in my address - sorry about that.

No problem, you credit the invoice, make a new one for the same amount with the name and address correction, then you wait for the items to be returned and checked and credit it again, make a new invoice again. The return shipment stays in limbo for a while, they order 5 more, one arrives broken, they want a refund rather than a replacement. With 2 delivery dates the system doesn't apply the 5+ discount.... and all of a sudden you have tons of transactions and "paperwork" for what you initially imagined to be 2 simple orders. They also forgot their password so they made a new account using a different email address. Looking over the logs a year later it is hard to figure out why they got a discount for just 3 units.

I haven't really explored it but one idea I had (for small volume) was to make everything manual data entry then put suggestions and warnings with the form fields. Even if you perfect automation there will be exceptions and mistakes made. By doing it manually everything also gets reviewed.