23 replies
2d22h
I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?
8 replies
2d20h
Looking at AWS's price page for us-east-1 [1]
* PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests
* GET, SELECT, and all other requests are $0.0004 per 1,000 requests
It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be unable meet durability goals. Successful read operations are an order of magnitude cheaper because they likely move only one of several heads, sometimes none at all (because they're cacheable).
Authorization failures should never hit an HDD.
Other folks have mentioned that these auth failures are costing AWS something. Sure...but very very little. Far less than the cost of a successful GET, much less a successful PUT. They really shouldn't be charging at all for this, or if they do, it should be like $0.00001 per 1,000 requests (another order of magnitude cheaper than GETs). Not even that really; the actual cost for them to process this request is likely less than they're (over)charging for the bandwidth alone.
6 replies
2d17h
What's crazier is that turning on Requester Pays does not actually mean requester pays when the request is a 403 [1][2].
Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill...
That's a platform level security issue for AWS to resolve. We do not need another Bucketgate.
[1]: https://twitter.com/Lauramaywendel/status/178507498639629151...
[2]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
3 replies
2d11h
For me it looks like AWS has mainly different scenarios to handle. First, unauthenticated users are a big security pain for AWS and S3 because S3 is used for everything. But S3 was not designed for "secure" public access. As a result billing attacks are possible (e.g. [1]) and will not be fixed (by design). Second, authenticated AWS users are kind of trustworthy and these are never assumed to do any DDoS stuff or similar. Here, we have even more surface for billing attacks. I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.
My personal suggestion is that you should never use S3 for public access. Public buckets are an open gate to your AWS bill. If you have private buckets the attack surface is reduced but even in this situation you cannot prevent billing attacks. Try to reduce the damage by activating Cost Anomaly Detection.
[1] https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-a... [2] https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-...
1 replies
1d20h
private in the sense that the connection will never make it to the S3 bucket, is that correct?
private in the sense that it is locked down to allowed source IP's or IAM-- then the connection still gets there and AWS incurs a charge for the Access Denied response, etc
0 replies
12h16m
Yes, private in the sense of "block public access" (your second point) (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access...)
Sadly, the private in the sense of "never make it to the S3 bucket" is nearly impossible. May this can be achieved by choosing some "secret" S3 bucket name.
0 replies
16h17m
I am waiting for the moment when this assumption breaks because credit cards do not make users trustworthy.
IMO it is already broken, or beginning to crack at least, with the advent of throwaway virtual cards.
Try to reduce the damage by activating Cost Anomaly Detection.
Yeah, this sounds like it could be a better solution than just using CloudWatch Billing alarms and billing alerts that I mentioned elsewhere. It's hard to tell at-a-glance if Cost Anomaly Detection is built on top of CloudWatch Billing alarms, or if they're just totally separate products. I don't seed the latter mentioned in the docs for the former.
[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori....
0 replies
1d8h
What is bucketgate? Your comment comes up when searching...
0 replies
10h18m
That's why AWS doesn't allow pentesting of this (S3) service.
0 replies
1d8h
Other folks have mentioned that these auth failures are costing AWS something.
This should be entirely irrelevant, as those same failures were happening even before the bucket existed, they were just hitting a different failure path.
8 replies
2d22h
This isn't obviously a scam to me. Cloudflare have guarantees around pricing per request, and I'd expect them to honour it. AWS have different prices. I'd expect them to honour them.
The problem is the cloud priorities availability over cost savings. We used to run our own infrastructure, that would just fall over if it got too many requests. This infrastructure never falls over; it just bills more. I don't know the right way to solve this, other than don't stick your S3 bucket on the public internet.
7 replies
2d22h
The right way is for unauthorized requests to not cost anything.
6 replies
2d22h
That makes no sense, though. Some network and compute are running for this to happen. Someone needs to pay the bill. It's not obvious it should be the cloud service, if that's not the agreement.
1 replies
2d20h
They charge ten times as much for an unauthorized PUT as they do for an unauthorized GET, and I feel fairly confident in saying that it does not cost them ten times as much.
0 replies
2d12h
That seems like a different issue to "who should pay at all".
1 replies
2d6h
AWS rakes in multi billion dollars in profit every year. They can definitely eat the costs of unauthorized requests to s3 buckets.
0 replies
1d8h
At this point I believe that convoluted pricing is their business model. I got hit by their “tiny font price” twice, paying extraordinary money for trivial things. And I barely used AWS.
0 replies
2d21h
This is part of doing business for the cloud provider.
The cost of running compute to deal with unauthorized requests for an arbitrary extant S3 bucket, is the same as the cost of running compute to deal with unauthorized requests for nonexistent S3 buckets.
If I generated a billion requests to an S3 bucket that did not exist, Amazon would have to decline that traffic in the same way. Since the recipient did not exist, there would be no one to bill.
I as the attacker should not be able to add on a sticky note saying "btw you can charge X user for this malicious traffic" and have Amazon actually honor that.
EDIT: Here's an analogy. Say a business in a city has a front lawn that they must pay to maintain. Sometimes people walk on the grass as a shortcut to the business. Some subset of those people also enter the business as a customer. Would it be fair for the business to charge those people more to account for the extra landscaping bill they cause?
But the answer doesn't matter, because that's not what happened. What happened here is more like, someone left a note on the front door of the business saying "My name is X, and I walked on your grass last night", and so that day the business charges more to any customer whose credit card says their name is X.
0 replies
2d9h
When thy receive a request to a non-existent bucket why don't they charge it to the lexicographically closest customer?
7 replies
2d22h
Ok I'm actually going to delete my web site and cloudfront distribution over this. It gets no traffic (at the moment).
The bucket is the same name as the domain so I can imagine this could be exploited quietly very easily as PUTs are so expensive.
Literally a $5 VPS could cost someone $1k.
3 replies
2d22h
i doubt it. i would imagine any single ip address would get rate-limited/throttle before doing much. the situation the article is describing is much more like a botnet — there were hundreds or thousands (or more?) of unique instances making requests.
0 replies
2d21h
i would imagine any single ip address would get rate-limited/throttle before doing much.
Does AWS have default per-IP rate limits?
0 replies
2d21h
i would imagine any single ip address would get rate-limited/throttle before doing much
Yes, you would imagine, because without relevant WAF rules in place there would be no throttling.
0 replies
2d22h
I very run big buckets in production that have single addresses hammering them hard all day every day. I've never seen any restrictions or throttling.
2 replies
1d22h
A $5 VPS should have a $10 spend limit or something, at which point it should shut down. In a reasonable world.
1 replies
1d8h
I think they meant the other way around, someone pays $5 for a VPS to spam unauthorized requests to your S3 bucket to get Amazon to bill you $1000.
0 replies
1d
Thanks. Damn, I've misunderstood some things recently so I'll take some rest and slow down I think.
6 replies
2d22h
A quick answer is to put the bucket behind cloudfront so that the denial is cached, but that's not totally free either.
If they actually charge for unauth'd requests that's absurd
3 replies
2d22h
That doesn't work, the standard endpoint always remains available.
2 replies
2d22h
I assume random names are the only way forward. Unfortunately a bucket name can’t be longer than 63 characters.
0 replies
2d22h
Surprising people are not already doing this. Something that has been beat into me for at least a decade.
0 replies
2d16h
You can stuff >256 bits into 56 BASE32 digits. The 63 character limit seems like it would only be constraining if humans want meaningful bucket names.
0 replies
2d22h
You can't protect your bucket with CloudFront if it's being access using S3 api.
0 replies
2d22h
Other than deleting the bucket, there’s nothing you can do to prevent it. You can’t protect your bucket with services like CloudFront or WAF when it’s being accessed directly through the S3 API.
5 replies
2d22h
Quite terrifying. Anyone know if competitors also bill requests which result in a "permission denied"? GCP, Azure, Backblaze B2, etc.?
1 replies
2d18h
90% sure GCP charges for rate limited API gateway. i.e. doesn't matter if it is above rate limit. Never got 100% confirmation either way though - their documentation on pricing is too vague
0 replies
1d16h
Yea they do, it says always charge when listing buckets.
Any way to mitigate this on AWS by restricting access to it via VPC only?
0 replies
2d1h
This note at https://cloud.google.com/storage/pricing suggests otherwise:
Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
0 replies
2d18h
Gcp def bills requests but not sure about unauthorized requests - this needs testing
0 replies
2d21h
It seems nobody replied to you, so let me try.
Yes for GCP and Azure, also for GPT...For Backblaze I am still investigating...
Two quick examples in different contexts:
https://learn.microsoft.com/en-us/answers/questions/1064419/...
https://community.openai.com/t/cost-for-failed-gpt-requests/...
Edit: It seems for Backblaze that would be a no, see below:
https://www.reddit.com/r/backblaze/comments/16o4bed/charge_b...
4 replies
2d22h
uf, aws should definitly not count the requests against a bucket that is not configured for public use.
Edit: if they do that, they should make it possible to firewall buckets via ips so that they are only accessible via certain ips
3 replies
2d22h
Or at the very least the requester should b charged for the failed auth.
2 replies
2d22h
How is that possible? AWS can't bill anyone on the planet.
1 replies
2d22h
This is an unauthorized S3 API request. If there is a valid access id and secret attached, bill them for the request.
0 replies
2d12h
I doubt that anyone's agreed to pay that. Also that there's often a valid access ID and secret. Also that the cost of scanning through all the client IDs across all secrets is worth the money.
4 replies
2d19h
Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
My first thought is that can be abused to multiply the damage if you want to engage in some cost based denial of service.
About every other week AWS gets out of their way to make us remember that they are the worst cloud around.
For reference https://github.com/ZJONSSON/node-unzipper/issues/308
3 replies
2d16h
To preface this comment, I'm not super familiar with range GETs or the library you're using. Are you saying that the library was performing range GET requests but instead of sending a byte range, it was only sending a starting byte number?
Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.
Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.
FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4
Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.
https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
2 replies
2d15h
Yes. It made many request with Range star- (i.e. without end).
500$ is not making a huge dent in the monthly bill for this. I just got a few laughs at my expense from colleagues..
I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then. I could generate 10s of gbps of "apparent" billable traffic with a puny VPS.
0 replies
16h31m
I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then.
I suppose that would be easy enough to test.
I was curious how video streaming services handle the case where a stream is started or partially consumed. Mux meters it in terms of [seconds of] video segments delivered:
Cost is per minute of video delivered. To calculate video delivered, we measure the number of seconds of video delivered to a video player. Note that if a segment of video is delivered, it is charged, even if the viewer doesn’t actually watch the video.
0 replies
2d13h
You could also generate 10s of gbps of actual billable traffic with a puny VPS by forging TCP ACKs and getting AWS to send you data faster than you can actually receive it, but that's closer to an actual DoS attack and more likely to get you in trouble...
4 replies
2d8h
Unfortunately you can't always keep your bucket name a secret, if you use presigned URLs to let your users directly download objects...
3 replies
2d4h
It’s a good idea to use a reverse proxy like Nginx to handle the transfer through an internal redirect, which will hide the bucket name and authentication credentials.
1 replies
2d4h
But this means data transiting your network instead of being directly fetched from AWS infrastructure, which for me mostly defeats the purpose of presigned S3 URLs.
0 replies
2d2h
If your server is on AWS, then the egress cost should be identical whether the transfer originates from S3 or EC2. But if your server is on premise and your host charges for egress (or has impactful bandwidth restrictions), I can understand where you're coming from, as you could end up paying twice.
0 replies
1d8h
Introducing complexity, failure point, scalability issue just to work around broken billing. You are probably right but I feel it’s wrong architecturally. I always liked S3 precisely because I could offload file serving to them.
3 replies
2d22h
Crazy, I wonder what AWS's margin is on these unauthorized requests? Questionable incentives.
2 replies
2d21h
Guess what? It seems is the same with GCP and Azure. I am willing to be corrected if it's not...
1 replies
2d21h
Yes? AIUI all 3 are regularly accused of wildly overcharging on egress; what's your point?
0 replies
2d21h
My point is that we need to talk about all three.
3 replies
2d21h
I wonder why does not Amazon be responsible and lock these accounts clearly attempting abuse? Logically either they are compromised or malicious. At least after certain number of failed request. And thus should be treated as such.
0 replies
2d20h
This case is especially problematic because this person was being hit from many distinct and legitimate accountants.
0 replies
2d18h
One problem here is that you can execute these unauthorized S3 requests without having an AWS account at all.
0 replies
2d21h
Anybody can do this.
1 replies
1d8h
Many are speculating that this will not be fixed by AWS (by design) however now that this has been discovered AWS will "need" to repair this flaw or they will start incurring customer flight to more secure or cheaper services.
The question is more about how long AWS are going to take to fix this issue and how many DDoS bills will they forgive.
0 replies
1d8h
1 replies
2d17h
Since you need to create a unique bucket name any way, I like to include my AWS Account ID in my bucket names. Be advised that bucket names need to be 63 characters or less. Another reason I do this is that CloudFormation supports the use of:
${AWS::AccountId}
0 replies
2d13h
This does not save you because AWS Account IDs are not private, e.g. if you are using S3 pre-signed URLs anywhere, those URLs include your account ID as part of the X-AMZ-Credential URL field:
https://medium.com/@TalBeerySec/a-short-note-on-aws-key-id-f...
1 replies
2d13h
Just waiting for a botnet group to abuse this, buy puts on the expected shortfall in earnings and profit.
0 replies
21h16m
Why would there be a shortfall in earnings? Failed (due to being unauthorized) HTTP requests cost AWS ~nothing.
0 replies
2d1h
Well this was an unwelcome attack vector, since I wrangle a fair few S3 buckets. I hastily threw together an S3 bucket name generator in Bash under Linux [1] to create names that follow the AWS rules to start my migration to my new S3 bucket names, and welcome any help to ensure it creates names that are strongly resistant to this attack, and follow various S3-service-like vendor rules, so I can do what I can until AWS and others come out with a stronger defense.
[1] https://github.com/automaticit-anthonyyen1/s3-bucket-name-ge...
0 replies
2d21h
Eye opening; I deleted my (empty) S3 buckets in response to this.
0 replies
2d22h
Crazy. Would not have expected that.
0 replies
2d10h
already a meme https://i.imgflip.com/8ohpwh.jpg
0 replies
2d22h
There are so many things that can make your AWS bill explode, so it probably should be included into your budget. Like, $1000/mo is not actually $60K in 10 years, but $60K + some random $20K that may or may not be waived by AWS support.
0 replies
2d22h
could you solve this with a "requester pays" bucket?
not viable for a public bucket, but seems like it'd do the trick on private buckets.
edit: hahaha, i'm wrong https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
0 replies
2d12h
For a long time, bucket name is believed to be non-sensitive so they are sometimes placed in public just for convenience (like in CI scripts of public Git repositories). It's easy to collect examples by searching "s3://" on sourcegraph. I believe it's much easier to find a bucket charging a company/open source project/person than we expect.
0 replies
2d21h
Use a password generator to create your S3 bucket names.
0 replies
1d11h
I was in the same situation as OP a few months back. My bill was $3000 after creating a "vanity bucket" which saw 500M accessed denied POST calls over a month. I took a while to get a one time refund.
0 replies
2d
For me, this is not only a cost thing, but a security issue as well. Considering the IAM auth or any kind of auth method, the goal is never to be "impossible to break-in", but "the spending of break-in is higher than the potential value it protected". Yet if the price of break-in is paid by the victim, this will be a whole different story. Someone can spam the S3 API with brute force IAM credentials can try out everything until find the right credentials, and the failure attempts are paid by the victim, which is insane.
0 replies
2d22h
Great find. This has serious exploit potential.
0 replies
2d20h
AWS shouldn't allow bucket identifier reuse if they're going to charge for unauthorized access.
0 replies
2d13h
Are Lightsail buckets which have predictable prices at risk too?
If there are limits which are surpassed, S3 charges may be incurred.
0 replies
2d21h
However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs--
Aaaah! CloudTrail data events are priced in such a way that each S3 request gets logged--at a cost of 20x that of the S3 request itself. I really hope OP got THAT part of the bill waived as well.
A bored or malicious actor can also just DDoS your CloudFront endpoint and run up an effectively unlimited bill on your end. That exposure is par for the course with AWS, they don't have a spending hard-stop and at this point they probably never will, you just have to hope that they have mercy and waive the bill if you get a surprise extra few zeroes on the end.
Shield Advanced claims cost protection, although as always, the devil is in the details... - https://aws.amazon.com/shield/features/
"AWS Shield Advanced comes with DDoS cost protection to safeguard against scaling charges resulting from DDoS-related usage spikes on protected EC2, ELB, CloudFront, Global Accelerator, and Route 53 resources. If any of these protected resources scale up in response to a DDoS attack, you can request Shield Advanced service credits through your regular AWS Support channel."
Shield Advanced is $3000 per month with a 1 year commitment, plus extra bandwidth fees on top of your existing bandwidth fees, I somehow doubt that many of the users who would be impacted the most by an unexpectedly large bill (individuals and smaller businesses) are proactively paying that much for DDoS insurance just in case.
I think CloudWatch billing alarms [1] are the only viable "solution" for an average joe here. Of course, they do not actually stop the usage, but at least they bring automated awareness.
Even with that, there's no easy way to rename an S3 bucket [2], plus doing so would commonly require updating config + re-deploying any apps using the bucket too...
All of this to say, the customer level really feels like the wrong layer of abstraction to solve this problem.
[1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
[2]: https://stackoverflow.com/questions/41511034/how-to-rename-a...
Agree on that. Was not claiming that Shield Advanced is the solution for this nutty request Rodeo...or maybe we should call it 404 festival...
and their bills cost up to 100x what they should (and then other services that run on top of AWS may inflate it 10x again)
First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....
Not in GCP if https://cloud.google.com/storage/pricing is to be believed:
Heh, charging for 5xx would be even more insane than charging for 403. "Our system is not working correctly at this moment. That'll be $0.25."
I'm less familiar with AWS. But on Azure, you can have private storage accounts which aren't internet routable so they wouldn't even be able to hit the endpoints to rack up charges. It's a bit ridiculous that you'd have to go that far to avoid being charged for unauthorized access, but I'd be surprised if AWS and GCP didn't have similar capabilities.
I don't know if Backblaze charges for failed requests, but it lets you set spending limits, so the worst case is that the attacker drains the daily quota that you set and later requests fail (DoS), not that you get a surprise $1000 bill.
This kinda happened with Netlify recently (https://www.reddit.com/r/webdev/comments/1b14bty/netlify_jus..., https://news.ycombinator.com/item?id=39520776). They got a bill over $100,000 via spam requests.
Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own.
One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.
They cost a lot less than a multiple disk write (PUT), a disk read (GET).
They cost the same to AWS whether the target bucket belongs to me, Amazon themselves, or does not exist.
And in this case, the customer definitely didn't "screw things up".
Charging for unauthorized requests is just weird and customer-hostile. Like, how am I supposed to prevent that?
I get that a service provider might want to "punish" a customer who misconfigures their systems and DoSes themselves with e.g. no backoff on 403 error, but I find it hard to believe that that's the common case -- and how is typo in authentication supposed to be so different from typo in bucket name?
Who pays when the unauthorized requests contains truly random data, i.e. they point at an org and bucket that do not exist? Amazon does. Part of their cost of doing business is building the infrastructure that routes authorized requests correctly, and if a request is for invalid data, it's dropped. That should be par the course.
Large numbers of unauthorized requests can be effectively considered a DDOS on the service itself. That the attacker can add a note onto their attack that says "btw bill X customer for all this" and have that honored is ridiculous.
This is exactly where my mind went. I assume most people, myself included, didn't realize this. This is absolutely going to turn into the economic equivalent of a ddos.
I'm far too lazy to do the math, but I do wonder how many typical residential connections it would take to make a noticeable dent on a big company's bottom line with this method? I'd wager it's a surprisingly low number.
That's why, even though during the day I work on AWS for people who pay me to do so, in the evening I work on my startups hosted on Hetzner where I know in advance what my bill will be not only this month but also this year.
Same here, it might be slightly less convenient but boy how well I sleep at night.
That's exactly it. I just spent the morning cleaning up my buckets.
I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private.
Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already.
So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances.
Obviously all accounts should have budget alerts, but there is no budget ceiling feature.
Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.
You cannot shut down an S3 bucket with a lambda tho, the mentioned issue above is only solvable by deleting the bucket.
Press F12 to screw competition
It's not just "kind of insane", it's completely bananas. Like what the actual fuck?
We really need some kind of regulation to ban predatory cloud pricing.
Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly