Amazon S3 will no longer charge for several HTTP error codes

As an HN reader it added valuable context for me that I was unaware of.

To be fair it is quite remarkable customer service story & relevant to the article.

I did, but I think it's worth stressing that this didn't actually have the quick two week turnaround you might assume if you first heard about it from the billing horror story posted on here recently. It's been known about forever and only became a priority when it turned into a PR issue.

It gave me pleasure.

Adds historical context as to a duration of an extremely long-lasting problem?

So, another way of saying this is, it took more than a decade to get attention?

Why have things stagnated?

twitter support tier is the highest of all!

But it's kinda the same as a trial where you have to put in a credit card number.

If they auto-charge once the trial is over, I don't like them. That is a dark pattern.

Equally, with this, AWS could very well ask you, the user, what you'd like to do if you surpass the free tier. Charge? Or turn it all off.

On top of that they could instate default thresholds so that you, the person who just started their free trial, does not get bill shock when you forget to turn of that $200/h machine.

My threshold is $0. I was never expecting to get billed on the free tier.

You have to manually set this up though.

There's so much UX for a prospective new AWS dev that could be improved. Say a 1 click "do you want billing alerts with that?" Template, or a "do you want to lock down expensive nonfree stuff?" option to set some soft limits to zero out of the gate (nescessitating a self serve support case to unblock).

It's frustrating. It's been this way for over a decade yet you'll still see new customers cutting themselves on the nuances of free tier. I get that AWS is 'enterprise real deal do what I say', but I don't think that means you should completely exclude the customer story of any new developers just getting their feet wet. It's an area of customer obsession the business has regrettably lacked, and if you go by the continuous stories of people messing it up on HN/twitter/reddit/etc, it only becomes more glaring how little the new guys are being taken care of.

"It's not too hard to figure out, though, if you take the time to read through their rather large set of documentation"

I don't even know where I would start with that. https://docs.aws.amazon.com/ lists documentation for 305 different products!

Ooh, that sounds nice. Beats my current slew of emails around week 3 of the month. Got a link to any docs to get set up quickly?

I prefer your version: Barr replies to a tweet before gatecrashing the next S3 planning session. "A customer is hurting, folks!". The call immediately falls silent with only occasional gasps heard from stunned engineers, and the gentle weeping of a PM. I wonder if Amazon offers free therapy following an incident like this

I was thinking this too. You're giving AWS a lot of credit if you think they're not going to do some kind of analysis about how much they were making (albeit illegitimately) from invalid responses. I'm just surprised that they either didn't do the analysis beforehand or that if they did do the analysis beforehand (like the parent commenter suspected), how they were able to get the report for that analysis out so quickly.

I find this hard to believe because this issue was known for years.

As someone that works at AWS (but not on S3), that's wrong in like eight different ways.

But the only way that matters is the core one - analysis, data, and instrumentation.

AWS does not make these kinds of decisions without a look at the metrics.

Metering feeds into billing and they are some truly epic levels of data volume. You can kind of see the granularity they're working with if you turn on cloud trail.

do lots of useful work for free

Have often wondered about this in terms of some of their control plane APIs, a read-only IAM key used as part of C&C infrastructure for a botnet might be interesting, you get DNS/ClientHello signature to a legitimate and reputable service for free, while stuffing "DDoS this blog" e.g. in some tags of a free resource. Even better if the AWS account belonged to someone else.

But certainly, ability to serve an unlimited URL space from an account with only positive hits being billed seems ripe for abuse. Would guess there's already some ticket for a "top 404ers" internal report or similar

Simple. Just encode all of your app's data and logic as a massive lookup table, each bit of which is represented by an object that either doesn't exist (a zero) or is unauthorized to access (a one).

When you read a sequential series of keys (404 403 403 404 = 0110) it will either tell you the data you were looking for or the next key name to begin reading from.

Well, you can probably send out one bit a time by updating your ACLs on a clock (with which your clients are also roughly synchronized) and distinguishing between 403 and 404.

take an awful lot of time to get that data out, though.

It said "never incur request or bandwidth charges". I assume this means you don't pay to compute the response or for the bandwidth to deliver it.

Seems you could compute the response, store it somewhere (memcached or something), and then return an error. Then have the caller make another call to retrieve the response. (To associate the two requests, have the caller generate a UUID and pass it on both calls.)

That doesn't make it entirely free, but it reduces the compute cost of every request to just reading from a cache.

(This does sound like a good way to get banned or something else nasty, so it's definitely not a recommendation.)

I don't know if this counts, but I had something that I would call parasitic happen to me once.

I administered a VBulletin forum, and naturally, we installed all sorts of gewgaws onto it, including an "arcade" where people could play games, share high scores, etc.

This arcade, somehow, came with its own built-in comment system, one where users could somehow register without registering for a proper VBulletin user account on our instance, and thus without admins being notified.

One day, we discovered this whole underbelly community that had apparently been thriving under our metaphorical floorboards, and promptly evicted them. In hindsight, I probably should have found some way to let them stick around, but recently several things had happened that hardened our stance to any sort of un-wanted users.

It has the same cost as a successful response which can quickly add up to a few hundred dollars per month with a couple of DNS enumeration scans.

Google Cloud and Azure also bill DNS like this. Unless you need some of the advanced features you really shouldn't host your DNS in the big cloud providers.

It’s still a huge problem for people that have purchased domains. I bought one that apparently used to be a BT tracker, and gets on the order of several hundred NXDOMAIN requests per second.

I understand it’s still hitting Route53 infrastructure, but I’m not using it, and it’s not commonplace to charge for NXDOMAIN records. Because of this, I’m unable to host at AWS (prohibitively expensive for my use-case).

It’s worth mentioning that DNS infrastructure for things like this are very cheap (I used to self-host the DNS infrastructure for this domain for ~$2.5/mo), so the up charge is even worse that what AWS is charging for bandwidth. If they brought it in line with actual costs, I wouldn’t have as much of a problem.

Agreed. Don’t know what made them wake up, but I did file a complaint about their free tier dark patterns and the Luxembourg EU GDPR office got involved after my countries GDPR office tried it first, and apparently is busy with some bigger investigation, so that investigation might’ve spooked them (not my own application I don’t think)