return to table of content

Netlify just sent me a $104k bill for a simple static site

bobfunk
150 replies
8h57m

Netlify CEO here.

Our support team has reached out to the user from the thread to let them know they're not getting charged for this.

It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.

Apologies that this didn't come through in the initial support reply.

aurareturn
59 replies
8h51m

There are only two questions everyone have:

1. Would Netlify forgive the bill if this didn't go viral?

2. How do you plan to address this issue so that it never happens again?

Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.

bobfunk
51 replies
8h36m

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral

2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages

noir_lord
31 replies
7h56m

Any cloud platform should have a spend-stop amount built in.

i.e. if I know I average $10 a day, I should be able to put in a "If it hits $50, email me and take it offline".

Of course the opposite problem is then people setting that limit too low but since the user defines the limit that's on them not you.

This is one of the reasons I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.

raiyu
25 replies
6h53m

We did this at DigitalOcean for similar reasons, wasn't a feature that was commonly used. Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.

What Netlify is doing here is really the best approach for both parties. And typically speaking a $104k bill would be hard to get paid up regardless if the customer's typical transaction balance was $5/mo and their credit card limit wouldn't be that high.

Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.

ohnoitsahuman
8 replies
6h35m

A debt is still a debt even charged back.

There would be many attorneys interested in collecting a $105k debt.

safety1st
5 replies
2h15m

No, not really. Not really what attorneys do. There might be collections agencies interested in recovering the debt, but if it's some rando guy who doesn't have the money, even that is open to debate.

I mean I'm not familiar with every debt collection scenario under the sun but Internet randos seem to think this is a real thing where like a cloud/hosting company sends an army of lawyers to repo some guy's house and runs him into bankruptcy because of a traffic overage. I've never seen it work that way, what happens like with most business debts, is someone at the company negotiates with the debtor to try and get as much out of them as they can, and failing that, possibly refers it to a collections agency which does the same but plays a bit more hardball.

In the case here with Netlify even before it went viral they reduced the amount from $104K to $5K, no lawyers, collectors or repo men involved, and while I'd hate to be stuck with that $5K bill, I dunno, that does feel closer to the mark of something that maybe you should be on the hook for if you're responsible for 200 TB of bandwidth overage over 4 days? Is this so bad on the part of Netlify?

All that said I'll just add that I've never given my credit card to any sort of host/cloud who had terms where they could bill unlimited overage fees like this. Never will unless there's a cap. Not Netlify not AWS not nobody. That goes for my personal life as well as for the business I operate. The terms is the terms and the answer is to not use these services unless you can afford them imho.

nickjj
2 replies
1h50m

I'd hate to be stuck with that $5K bill, I dunno, that does feel closer to the mark of something that maybe you should be on the hook for if you're responsible for 200 TB of bandwidth overage over 4 days?

The responsibility part is the tricky part of the equation.

If someone hits your site with a DDoS attack, are you responsible? There's literally nothing[0] you can do as a customer of a cloud provider here because anything you can do is limited to the servers and services you're given access to. For example even if I had access to billions of requests and built an anti-DDoS tool it would still need to run within the cloud provider's provisioned server which means I'd be on the hook for all traffic costs because it's something running in my account.

That doesn't seem reasonable to me as a customer. It means a cloud hosting provider can put an extreme financial burden on a customer and make a killing in profits because of the markup they charge on bandwidth. The incentives are terribly misaligned.

[0]: I mean you can sign up for DDoS protection through a 3rd party company but in this case I'm talking about taking actions within your hosting provider.

safety1st
1 replies
56m

All fair points but do they apply to the Netlify situation? As I understand it they generally won't hold you liable for resource usage generated by a DDoS, the guy on Reddit said this was a DDoS, the Netlify CEO said the traffic "didn't match attack patterns..." I think telling a free tier customer that they owe $104K was a pretty stupid PR move either way, but we don't really have enough info to say whether this was a DDoS or not

nickjj
0 replies
24m

As I understand it they generally won't hold you liable for resource usage generated by a DDoS

From personal experience as a customer of a cloud provider (not with Netlify btw), usually cloud providers who profit from bandwidth costs will write their TOS in such a way where almost nothing qualifies as a DDoS attack unless it's truly a distributed and targeted large scale attack specifically on your site.

A random person on the internet who spins up a few VPSs around the world and slams your site with looped curl requests won't count as a DDoS attack even though from your perspective that will result in a massive bill increase due to bandwidth costs.

In other words, I'm not surprised "didn't match attack patterns" was used. I'm guessing that will be the case most of the time.

coryrc
1 replies
1h7m

Traffic doesn't cost money. Bandwidth costs money. Unused bandwidth doesn't cost less than used bandwidth. So, no, you shouldn't pay so much for something that doesn't cost them any money?

dboreham
0 replies
50m

Traffic doesn't cost money.

Mostly false. Either transit is billed on a 95th%ile basis (so...more money for more traffic), or if it is flat/netted, you're still paying for the capex for the switch ports (fatter connection to support more traffic means more $$$ for the gear to support it).

quickthrower2
1 replies
6h16m

Yes if a corporation says I owe them $100k I will lose sleep. Even if they don’t have my card details.

dwighttk
0 replies
2h47m

The secret is to bump that debt up to $100B

aurareturn
4 replies
3h49m

  Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.
But that's on the user. The user shouldn't get upset in that scenario and has no right to. You're giving the control back to the user.

typon
1 replies
3h16m

The infantilization of the user is common in tech now. For good reasons? Maybe. But it is common.

pierat
0 replies
11m

The site user/admin saying "If this spend goes over $100, shut shit down" is called being a fiscally responsible adult.

The fact that most cloud operators don't have actual hard cutoffs to maintain financial responsibility is intentional. Azure does, but only for specific account types. If it's PAYG, you can't do it. The end result is if you do something "weird", or someone DDoS's you, you're liable.

With a hard limit, a DDoS just takes your site offline.

gwright
0 replies
2h38m

But that's on the user. The user shouldn't get upset in that scenario and has no right to.

I agree. I also agree that when dealing with large numbers of people, there will be people who don't understand this and/or will actively try to social engineer their way out of their own decisions.

Setting customer expectations and meeting them successfully isn't easy.

conductr
0 replies
2h1m

How about just fix the pricing formula to account for massive surges.

Instead of forcing user to set a low cost limit and missing a viral opportunity or the platform writing off the massive bill the customer can't afford... just put the billing mode into a reduced price mode or have some more nuanced configurations. Sometimes is just asking the question the right way. Instead of "max spend limit" or similar "If your site goes viral, how many requests do you want to serve before going offline? 1M=$20, 10M=$100, etc" at this point, I feel like bandwidth consumption is a bad metric for billing; just use requests/visits/actions and price for those.

This is not prescriptive just illustrative. The point is make a better pricing formula to account for these massively unexpected events. Couple it with an aggressive notification policy when this traffic event gets triggered. The user should know the traffic pattern has changed and a high traffic event is happening. They can login and change the configs and decide if they want to keep it going or not.

noir_lord
1 replies
4h46m

Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.

So your suggestion is to issue a chargeback.. to get money back that should under the terms of whatever service you signed up for be owed?.

That seems like bordering on fraud tbh.

Additionally, when you set that limit people then get upset because usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down.

Legit concern and something I mentioned, I'm gonna guess there are broad two camps on that one - mine which is "I want a safety ripcord" and "whee, nice problem to have".

However since this entire conversation is around a guy who got a massive invoice because of a bill he wasn't expecting and couldn't have set such a limit I'm still gonna go with a "I want a way to constrain the financial downside - hell turn it off by default but give me the option".

Since broadly a lot of cloud stuff doesn't, I'll constrain it a different way.

mattferderer
0 replies
1h50m

So your suggestion is to issue a chargeback.. to get money back that should under the terms of whatever service you signed up for be owed?.

Funny story... One of the big cloud providers actually has you do that on purpose as a remedy for an account you've lost access to.

nico
1 replies
2h38m

usually when they go over it for a good reason, like going viral, they aren't anticipating it, and just when their traffic is most valuable the site is down

Of course. And that’s why any limit against a dynamic variable should also have alerts linked to it

Send an alert to the user when traffic starts spiking, especially if a simple projection shows it’s going to go over their limit

Then the user is aware, hopefully with enough time to lift the limit if needed

ygjb
0 replies
1h58m

That's a level of responsiveness that doesn't exist for the vast majority of organizations.

If your customer is aware enough to notice they are being hit with a DOS or legit traffic while it is happening, then great! They can respond, and if needed, engage proserve to get support for scaling or defense depending on needs.

If your customer is not alert enough, then their site is offline, and they won't hear about it until their customers are screaming at them, which will result in a P1 ticket to look at a vendor who won't turn them off during an unexpected peak.

It's a catch 22, and if you have to choose between: a) a PR hit because you have to go on a forum and post about waiving the fee, or b) a PR hit because someone posted a blog post about how you killed their site during a moment of critical growth

any reasonable business will choose A every time because A is far more supportive of customer growth and has drastically better optics. Anyone who thinks A is worse is probably too inexperienced to have an opinion.

ahmedfromtunis
1 replies
3h52m

Here's an idea: let users set a spending limit.

If they're about to go over, shoot them a quick heads-up and give them 24 hours to sort things out or level up their package.

After that, if they haven't made any changes, temporarily pause their site access

skeeter2020
0 replies
2h6m

AWS budgets are good for the first part, but they have no interest in a hard cap for obvious reasons.

patmorgan23
0 replies
2m

They still need to make tweaks to avoid sending $100k bills to people who signed up to a free service.

Like let them go over one month and then make them sign up for a paid plan for the next.

jedberg
0 replies
1h4m

Also, that's the benefits of credit cards - that you can still issue a charge back, and credit card companies very much favor the consumer rather than the merchant.

That has not been my experience. I've had to do a few chargebacks for services not rendered, and I've never won. I will submit my evidence, then the vendor will submit 100 pages of random emails, and then I will have my claim denied. Then I will appeal, will point out that they sent 100 random pages of email, and then they will reply with the same 100 pages of emails and I'll get denied again.

It seems that the vendors have found the hack for chargebacks -- just inundate the credit card company with so much data that they assume the vendor must be right.

It makes sense -- the vendors pay the credit card companies a lot more than I do. They'd rather keep them happy than me.

hex4def6
0 replies
2h10m

It seems like that could be an option like:

Hard limit: [$1000]/mth or [$500]/24hr period

Notify me if traffic exceeds thresholds: [$800+]/mth or [$400]/24 hr period

Notify me if the traffic forecast for the month looks like it will exceed my hard limit before the end of the month.

charcircuit
0 replies
2h42m

that you can still issue a charge back

Most users don't want to be banned from their hosting provider.

benterix
0 replies
50m

Just want to chime in:

1) Thank you for allowing us to set the limit.

2) I understand your opinion that you prefer chargebacks but I disagree with it.

The very reason I stay with Hetzner is that I know in advance what my bills will be for the whole year. Heck, I even charge my account in advance so that I don't worry about any charges!

kej
2 replies
2h15m

This is something I really like about Nearly Free Speech.net. Their model is that you deposit funds up front, and they will deduct from those funds as you use services. It helps that they actually are nearly free so that a single $20 deposit can last for months or years in many cases.

It's bizarre to me that more services don't support billing this way, since there are tons of situations where I would much rather have a site or service go down than be hit with a surprise bill and have to depend on social media and magnanimous corporate PR.

nazka
1 replies
1h27m

Yes it’s nice like that. Specially for side projects on AWS that could go wrong on your personal credit card. Also I heard they forgave bills sometimes.

pierat
0 replies
20m

Amazon will, but they also gauge their discount in how many prevention and security measures from their 5 Pillars you follow in your environment.

You can do stuff like "disallow any of these instances to be used in your env", so if you never use graphics cards, disallow the whole class.

You can also set limits like "no more than 20x m5.4xlarge".

But again, AWS is the worst about no actual hard limits, cause each system generates bills. Ive also seen the hell of "hidden system AWS Billing doesnt have is still submitting billing and we dont know what it is". Again, AWS enables basically infinite liability.

Ive also discussed with C levels that "every engineer and dev with AWS logins have an unlimited credit card to of which you're on the hook for". Lets just say that 'heartburn' doesnt even begin to describe the terror on their faces.

listenallyall
0 replies
1h34m

If a cloud platform offers such a limit, but the user fails to set it up, then uses $100,000 of bandwidth, is the platform then justified in NOT forgiving the bill?

conductr
0 replies
2h19m

I still in 2024 rent physical boxes and run the modern stuff on top of them directly, yes it costs me more per month but the price is hard capped.

I still prefer this too. Kinda funny how server resource limitations became a feature and not a bug when it was one of the problems the cloud sought to overcome

Hamuko
5 replies
7h3m

If forgiving bills for this kind of a thing is a standard practice, how come this was the customer support's first reaction:

We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I've currently reduced it to about 5%, which is $5,225.

20% and 5% are quite a bit higher than forgiven.

CogitoCogito
3 replies
4h50m

Given this has been asked here multiple times without response from /u/bobfunk, it’s hard to conclude anything except that he is lying.

hirako2000
0 replies
4h32m

Lying but a good talker for sure.

I wouldn't want to be CEO these days. A lot are trained and paid to do damage control.

hirako2000
0 replies
4h23m

I will put there the other obvious offender: Vercel . Not sure about bandwidth, but dark patterns, keeping serious RBAC procedures we'll hidden until asking a fortune even for startups, to provide not things like SSO, just reasonable RBAC.

With all that money they then can finance the free tier until they get too far and become platform locked-in.

Surge.sh Im not sure. But shows all the sign of some greedy acquisition, regular long outages , as if I have been sitting as a free tier for too long, quick nudge to pay. For barely accessed sites even behind CDNs, steep. I even worry they one day just wipe all my buckets (they did for a few already) and support would recommend me to be a "normal" paying user .

Nothing is free. And nothing too good to be true is true .

Jgrubb
0 replies
4h30m

Or that usage and billing is a difficult space and this particular thread has been dogpiled by a bunch of folks that have never actually worked in it.

op00to
0 replies
6h26m

This needs to be much higher up.

oneeyedpigeon
2 replies
7h32m

2. is obviously what should have always been the case, but it's good news to hear you've now gotten there. Every single hobbyist website would always choose downtime over a hundred thousand dollar charge.

rjzzleep
1 replies
7h29m

With a properly configured nginx, you can easily serve 10's of thousands of requests a second on vserver type hardware. Netlify just offers these build pipeline kind of static site with cms UI.

But this is a good reminder why my gut feeling always made me avoid these overengineered solutions.

hirako2000
0 replies
4h36m

They aren't engineered, they subsidize (more) enginnering effort , and (are meant to) cost less as a result.

They do. But of course maximizing profit is the sole true prerogative of capitalist enterprises. And the market is not totally competitive. So yes your intuition was correct, to be cautious against over enginnered pricing to get y'a.

I mean those companies cater to hobbyist. Then ...

Render seems more fair-play. Until a change of mgt occurs of course.

charles_f
1 replies
2h38m

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral

Sequence of events doesn't support this answer:

1. User gets charged 100k

2. User complains to support

3. User receives discount to 20k, then 5k. Support states policy is normally 20k

4. User discloses to the world. Goes viral.

5. Invoice is forgiven

While you might forgive "lots and lots", fact is that you still presented the invoice to a free tier customer, and when they complained you gave them a discount, but still charge. Only when it went viral did you forgive it.

sanitycheck
0 replies
2h22m

Quite... It does seem that either the story we're getting isn't completely accurate or the support people who handled this need a little reminder of what's supposed to happen.

I'm a paranoid person by nature so "It's free... just... give us your card details" is always suspicious.

pierat
0 replies
2h16m

Give that there are free stressers/booters , and reasonable prices to rent a DDoS cloud.... https://stresser.su/#pricing

1. What are you doing to prevent DDoS's from hitting your network?

2. Why do customers have to allow an unlimited credit burden to use services?

3. Why arent there cost controls to "if $$ exceeds X, shut acct down"? Azure can do this.

Long story short, why are you by default (except for social media escalation) passing fraud costs to customers?

hirako2000
0 replies
4h45m

1. Forgiven many, is Netlify forgiving all obvious anomalies? Is the question, which if so but you said many so it is a no, it would make you reconsider the next point 2. Favoring keeping people site up ? Would you go as far as keeping them up if they stopped paying for the meter? If not you simply should not let that meter go overboard.

Hey I'm a taxi driver. Hailer fell asleep on the back, so I kept driving all night, once he woke up I dropped him to his place and asked for my monthly wage. I "forgive" many, but just a few are juicy income so I adopted the policy to never wake any customer up. If people ask I say it would be impolite, principles prime.

awill
0 replies
4h8m

I’m sorry. You are working on changing things so FREE sites don’t get charged???

That’s the elephant in the room here. I understand an enterprise plan where you state billing is $xx per GB, but billing someone with a free site??

Give me a break.

Sephr
0 replies
8h1m

Do the changes you are working on that will cause "the default behavior to never let free sites incur overages" involve providing users with spending limit controls?

Solving this only for the free site use case doesn't address the core problem that people are bringing up about a lack of spending limit controls.

SXX
0 replies
6h22m

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral

No offence, but this sounds like "trust me bro" billing and it is not good enough. Someone could literally get a heart attack from getting $100,000 bill - this amount of debt can literally ruin someone financially.

2. While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages

I hope you understand that chance someone who used to pay you $20 / month unlikely want to ever get $10,000 bill. Yeah people might dislike that their website went down due high traffic, but it's not gonna bring this much negative PR as incidents like this. There should be some sanity check at least.

FlyingSnake
0 replies
5h22m

Regarding #2: I would rather have my hobbyist website go down rather than facing the daunting task to raise a query on HN and hope the bill goes away.

EasyMark
0 replies
21m

You should probably consider a daily limit (up to some max n days) rather than a hard one time limit. If your engineers can set a 1 and done they can set an n and done and it would be a much better solution and more customer friendly. The guy using 5 gigs today as a poor college student will likely have a position in a small to mid-size company in a few years. I assume non-free (but low tier) customers would much prefer a reasonable limit set as well. Maybe a max of 2x (or so ) bandwidth so no huge surprises. Remember they're your customers and not your paying adversaries

CogitoCogito
0 replies
8h27m

1. Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral

This isn't what you said in your first post, you said:

It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.

So forgiving "lots and lots" doesn't move the needle. Do you or do you not forgive _all_ such cases where your DDOS protection doesn't take down the site? What was your employee referring to when saying that the usual discount is 20%? Are you saying that you _never_ discount 20% and instead always discount 100% i.e. "forgive"?

gaza3g
5 replies
7h36m

Thank God for social media that the user was able to get attention about this on Reddit which he was then advised there to post this on HN. It must have been stressful to see a six-figure bill and then get told that that, no worries, you’d ‘only’ be charged $5k instead for a static site. It’s just ridiculous to me to be sent a 6-figure bill in the first place.

herbst
2 replies
7h23m

I hope this is not one of the cases that get simply forgotten and in a week or two their beginner unfriendly platform gets recommended again without a second thought.

With models like this and AWS people will get afraid of success

quickthrower2
1 replies
6h12m

I think fly/netlify/vercel/render etc. get a decent enough flak on here for costs and/or reliability.

The average HNer seems to be recommending colocating your physical server :-)

ptdorf
0 replies
1m

This is the way.

mlrtime
1 replies
5h24m

Well, it's still debatable for the history books if social media is a net good.

Before the internet, these issues would be handled by local news journalism, and still sometimes do!

romphl
0 replies
4h0m

I mean, social media is pretty much an inevitability once mobile phones/internet became mainstream. Just like the invention of the gun and gunpowder, I think we are still debating if it was good for society right to this day.

mchinen
0 replies
8h31m

From the 5% reduction it seems (1) was less likely.

To bobfunk, the response needs more empathy and explanation around the obvious frustration around why there is no slider for cost limitation.

As it is, it feels like the minimum viable corpspeak apology and damage control.

altin0
21 replies
5h7m

Lol this deescalated pretty quickly, went from $104K to $20K to $5K to $0 Which basically means you almost scammed the customer for $5K or $20K. Super negative practices. I for one could never trust a company operating in that manner. It would be much more honest to say "unlimited bandwidth" and set a hard-limit for maximum budget, then people know they won't be charged, than to go through all this crap and then pretend you're doing a favor to the customer (you're not). If I'm normally spending $10/month any idiot out there would know for sure that I'm not going to spend $104K instantly. That's a very basic filter to have. But you don't place such filters because obviously you're working on the principle to scam people many thousands of $ if they fall for that. Heck, for all we know you might send that amount of traffic to your customer and the try to scam them and if it doesn't work then pretend you're doing them a favor.

hasty_pudding
10 replies
4h39m

The fact that the CEO had to step in after this blew up online otherwise they were going to try to extort that poor dude for thousands of dollars!

Moving my sites off of netlify ASAP.

BigJono
9 replies
4h30m

Tell you what is a good question, why is this thread on page FIVE of HN (ranked #125) with 1000+ upvotes, 400+ comments and only 7 hours old?

dang
6 replies
3h26m

This is in the FAQ: https://news.ycombinator.com/newsfaq.html. See "How are stories ranked?" and "Why is A ranked below B even though A has more points and is newer?"

About this specific case: it set off the flamewar detector (a.k.a. the overheated discussion detector) and also got moderation downweights. We sometimes turn off that penalty, but I don't think we'd do so in a case like this, because HN gets so many posts of this nature. They flare up with Big Drama that is sensational for a while but not particularly interesting, and therefore not really what the site is for.

In fact HN gets so many posts of this type that it has become a joke, and not only that but a cliché, so much so that the top comment of the Reddit thread repeats it [1]. That's about as repetitive as anything gets. The basic idea of HN is to gratify intellectual curiosity [2] and avoid repetition [3].

[1] https://old.reddit.com/r/webdev/comments/1b14bty/netlify_jus...

[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

[3] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

BigJono
5 replies
3h12m

I don't really buy that to be honest.

I read this whole thread before the CEO posted and after, and neither time thought any of the comments were out of line or even that the general mood was any more heated than any other random HN thread. People are politely asking pertinent questions.

And I think once the CEO makes a statement which contradicts the company's support response, that becomes very interesting. Particularly to anybody that uses their service. I'm certainly not finding the conversation very repetetive or cliche.

dang
4 replies
3h2m

You're welcome to disagree, of course. My main concern is to explain what the principles are. I'm not saying we apply them perfectly—sometimes we make bad calls.

I can tell you pretty much for certain though, that we'd hear many more complaints if a Reddit thread about a customer support shitstorm stayed on HN's front page for very long.

Btw, the Customer Support Fuckup category is one of several $X where HN has become known as the place for $X, but only because HN is not actually for $X. Another example is the Site Is Down category—people often come to HN to find out what's going on when some $Site or other is having an outage. But just as HN itself isn't a site monitoring platform, it's also not a customer-support-of-last-resort platform.

If the community feels like this customer support fuckup is altogether more interesting, I'd consider reversing the call, but again, my gut feeling is that we'd get even more complaints that way.

BigJono
2 replies
2h53m

I agree completely with the underlying principles, I just think once the CEO has commented and stirred up some interesting discussion that's relevant to a large segment of your userbase, the thread doesn't really belong to some generic "customer service shitstorms" category anymore.

I learnt more about Netlify, Vercel etc and how they operate from this thread from the last 100 "customer service" threads combined. I learned about Cloudflare's offerings, and a bit about Hetzner. And it was all very interesting.

You said you sometimes turn off those penalties, I think this thread would be a good candidate.

dang
1 replies
2h50m

Ok, let's try that and see what happens!

pvg
0 replies
2h17m

"Is that you, Rabbit?" said Pooh.

"Let's pretend it isn't," said Rabbit, "and let's see what happens."

croemer
0 replies
1h8m

Thanks for explaining.

Something that makes me feel uneasy about the fact that the post gets hidden is that this strongly benefits Netlify. It seemed like lots of people moved off Netlify after reading the post.

I'm not suggesting that HN actively took an action in Netlify's favor, but the potential is there. Is the algorithm for flame war detection open source? Or do we essentially need to trust you that there was no interference from Netlify? (I do trust you but others might not).

sergiotapia
0 replies
3h36m

I found out about this from twitter - weird how it's so buried on HN.

canyonero
8 replies
4h37m

This is very weird take. I'm struggling to understand why this is incident as a reflection of "super negative practices" or is somehow a "scam". The CEO came here and publicly apologized for the mistake and mis-communication, and the issue is resolved for the user with no charges. What am I missing?

altin0
4 replies
4h23m

It's only a weird take if you don't have any common sense. It's super simple: either offer unlimited bandwidth(since you're not charging these anyways), like Cloudflare Pages does, or put in place controls that will allow customer to set a top limit for their budget. You can't just all of a sudden send them a $104K bill and expect them to pay when the've never spent more than a few bucks. And then even worse, you can't pretend to expect them to pay 20%, then 5% then pretend you're doing them a favor by completely liftig it off. That's just arbitrary billing and preying for any victim that would fall and agree to pay 20% or 5% etc. I'm just asking for common sense and practices that build trust, not arbitraty billing rules.

chomp
3 replies
4h2m

"Pay for what you use" is an arbitrary billing rule? Come on now.

OP was ignorant, and got tossed a lifeline. Also “just make everything zero dollars bro” is a ridiculous proposition.

wpm
0 replies
3h41m

Pay for what I use works for airline seats and reserved compute/storage resources.

I have no control over how much traffic my public sites get. There is zero value in me signing up for a service which charges me based on traffic if I can’t control the maximum they’ll charge me. Would you sign up for an infinite bill?

notnaut
0 replies
3h38m

In New Jersey I have to let an attendant pump my gas. If I have a heart attack while he’s pumping gas, but I never explicitly say “please stop once it’s full” and he, innocently enough, takes the still-flowing gas hose and pops it into a sewer grate once my tank is full, you’d be hard-pressed to find a reasonable person agree that the attendant was throwing me a lifeline when he refunds me after I come back complaining about my $2k gas receipt.

This is a dumb analogy, but the point is there is very obviously a pattern in this payment process that is ripe for abuse. The question of whether or not you aim to be an abusive business, plucking every shady profit where you can put the onus on the customer to try to come get their money back is one that many companies are deciding, and many are erring in the direction of the dark pattern.

By not working to avoid this problem from the get go, there is an implication about how a company wants to make their profits.

altin0
0 replies
3h47m

the CEO said they're "forgiving any bills from legitimate mistakes" which effectively means "just make everything zero dollars bro". And no, he didn't use all that bandwidth, he was victim of a DDoS which the hosting provider should have measures in place to prevent or limit the service if it happens.

herbst
1 replies
4h33m

What price would the dude have to pay if he didn't publish it? How often does this happen and why is there no protection against charging free customers 100k out of the blue. Why charge it and shock the customer if practice is to waive it? The CEOs response kinda just made the situation worse.

canyonero
0 replies
3h6m

Yeah, I don't buy this conspiracy theory. The reason why they charge it could be as simple as they calculated the bandwidth usage following a ddos attack. It amounted to 104k worth of bandwidth usage. There system is not sophisticated enough to recognize it was a mistake due to attack on their site. Thus a manual intervention was needed, and now it's resolved.

jmgaicra
0 replies
4h21m

Any person seeing a user that normally has a $0/$10 per month bill suddenly spike to $104K would see that this is obviously a DDoS.

If it has always been a "policy" to forgive bills, shouldn't it have been 100% forgiven immediately after OP contacted support in the first place? Why go through the trouble of playing the hero by offering "discounts".

chatmasta
0 replies
4h26m

Heck, at that point, why not "send some traffic" to your customer? It's not like they have any way of verifying its source. Hmm... why even send traffic at all? Just add a multiplier to their metrics!

valine
15 replies
8h38m

I’ve been a netlify user since 2017 and I just deleted all my sites. I can’t risk receiving a $100k bill for toy projects. Your “current policy” is not good enough.

ks2048
3 replies
6h1m

Same. I'm looking at alternatives to get off netlify ASAP.

herbst
1 replies
5h17m

Any cheap VPS and nginx should work for 99% of their customer base I guess. If you want easy deploy for static or dynamic just use git hooks.

ryandrake
0 replies
1h4m

I've run all of my hobby projects, including personal web pages, a Wordpress site that serves a local club, a small single-JS web app, and E-mail hosting for my family and a few other domains, on a single $5/mo VPS, and have never received a bill higher than $5 for the past, I don't know... 15 years.

If your web site makes you money in proportion to the amount of views or bandwidth you use, by all means, go with a provider that increases your costs when your traffic rises. But if your web site does not make you money, why not host it somewhere for a flat rate?

eric_cc
3 replies
5h35m

Same. Toy project and it’s not worth the risk of using netlify. What’s a good, simple alternative for a VueJS app?

victorbjorklund
0 replies
1h26m

Cloudflare pages is pretty much drop in for netlify. And it has unlimited bandwidth for free (at least in theory. Guess they might call you if your site does 1 petabyte per hour)

lelanthran
0 replies
7m

What’s a good, simple alternative for a VueJS app?

I'm not sure about VueJS specifically, but I run everything I can off a $6/m digital ocean droplet (static sites, web apps, git repos, RDBMS, some other custom apps I've written) and it hasn't broken a sweat yet[1].

My understanding used to be that requests will be dropped if my virtual server can't handle it, and I'll have to transfer 10,000TB to get to a $100,000 bill.

In practice, my server will not physically handle the load to serve more than maybe $1000 of data a month; it will fall over before that.

In summary, using a VPS is sorta like an instant hard cap.

[1] Until I tried using Jenkins. Which crashed constantly because apparently 512GB of RAM is too little for what it does. I'm now in the process of writing my own little CD tool that isn't going to go over 30MB of RAM just to run my deployment scripts.

geon
0 replies
2h59m

Github pages.

jug
2 replies
6h29m

I agree and also delete my account.

The only "fix" here is to act like Hetzner and null route upon DDoS, price cap the thing, or offer unlimited bandwidth on the free tier like e.g. Cloudflare Pages.

Uncapped but paid is a recipe for disaster and you'll always be subject to the will of the support staff when something happens. If they can grasp to a straw leading to suspicions that it's not in fact a DDoS attack, you can for example be sure they'll do just that. Just no.

zrn900
0 replies
1h43m

Hetzner dedicated servers are (true) unmetered with 1 gbit connection. (can be upgraded).

https://www.hetzner.com/dedicated-rootserver/matrix-ax/

With a 48 core Epyc or 80 core arm server, one really shouldnt need much more for a middling project. There are enterprises who run entire services on such hardware.

livrem
0 replies
2h43m

How does price caps work on Hetzner? I never managed to figure that out from reading their price lists. It looks to me like they charge for each TB, and the only thing I can see is that you can set an email alert to go off when close to some threshold?

turtles3
0 replies
6h24m

Same, as it stands you the user are legally liable for the full bill unless netlify graciously forgive it. Even in op's case, they didn't (still charging 5k!).

If there was an option to cap billing, or at least some legally binding limit on liability, then I can countenance using netlify.

Until then, it's just not feasible nor worth the risk.

listenallyall
0 replies
1h39m

Starting to wonder if this whole thing was an elaborate ploy by Netlify to cull the herd of longstanding, non-paying accounts.

jcalx
0 replies
2h28m

Same. I will (almost certainly) never incur a $104k bill, but switching to Cloudfare Pages looks free and I don't want to depend on unwritten policies of goodwill to mitigate the potential risk.

erremerre
0 replies
7h35m

Same boat here.

the fact that once it arrives to the limits does not display an error page.

At this point I honestly do not care about they changing their policy, they should have thought that a normal person receiving a 100000$ bill on a free tier shall not been at all on the table in any circumstance, even if they forgive the bill, nobody needs to stress out like that.

croemer
6 replies
8h50m

How long has this been the "current" policy? 2 hours?

swyx
5 replies
8h44m

ex employee here, left 4 years ago. was policy back then too.

aurareturn
3 replies
8h42m

  but instead forgiving any bills from legitimate mistakes after the fact
What are these legitimate mistakes?

hnfong
1 replies
6h52m

Presumably the "mistakes" mean failures to detect/recognize "attack patterns".

optionalsquid
0 replies
1h42m

Wouldn't that imply that a person whose site legitimately went viral would be stuck with the $100k bill?

kjs3
0 replies
53m

Anything Netlify deems them to be, of course. That's why these sorts of T&Cs use weasel words like "legitimate", "reasonable", "expected", etc., instead of giving specifics you can action against. That way they can claim every thing they've done is legitimate and reasonable no matter how fallacious that claim is, and double-dog dare you to spend the time/money to take them to court (or worse, imposed arbitration with an arbiter of their choice) and prove them wrong.

CogitoCogito
0 replies
8h25m

So the original support worker just pulled 20% (and then 5%) out of thin air? Given your internal knowledge, can you maybe explain why a support worker would ever do that if policy is simply to forgive the debt?

7moritz7
6 replies
8h0m

How does a 60 TB in a day peak for a site that previously never crossed the free tier threshhold not qualify as "attack pattern"?

This is a static site. To reach that sort of bandwidth out of nowhere you'd need to publish the blueprint for a teleportation machine

hnfong
5 replies
6h54m

To be fair, these days, things can become viral literally overnight.

That said, instead of depending on unreliable heuristics, they should just allow an option to change the behavior. The "current policy" to charge small sites on the free tier thousands of dollars instead of just throttling/shutting down the traffic is really predatory.

quickthrower2
2 replies
6h7m

60TB with each request for 1Mb say would be 60000000 visitors. So guess this is possible but hell of unlikely.

Izkata
1 replies
2h3m

"static site" doesn't necessarily mean "small". It would be easy to go way over 1Mb with a couple of pictures.

frontalier
0 replies
1h4m

at 60MB it is still one million downloads

pants2
0 replies
2h24m

Most people won't want to fork over $100k to support a hobby project that's gone viral either.

brnt
0 replies
4h28m

Anyone exceeding their plan with a factor of 10 or hell, let's make it a 100, almost certainly didn't anticipate it and thus isn't prepared for the kind of bill that apparently comes with it (or even knows that there would be a bill). On top of that, there currently is no way to state such rules up front! Moverover, according to their own explanation, it was almost certainly not organic traffic!

I wager the vast majority of people in the free tier would gladly cap their traffic at the (generous!) bandwidth offered by Netlify. Even to the majority in paid tiers, 100k bills where there previously was none must be unwanted and unintended.

I mean, we all know dark patterns are a thing...

tledakis
5 replies
8h45m

It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.

Well, giving the option to users to plan ahead would be best, no? Like a setting to choose whether they want a potentially unlimited bill versus downtime. Instead of that, you are choosing to stress and make people scared/anxious/homeless even (if they don't think of raising the issue on HN).

Seriously, this is not rocket science. This must have been discussed before in your company, and someone actually made this decision to stress people about such bills.

CogitoCogito
4 replies
8h23m

Frankly the only reason I can even come up with that Netlify wouldn't have such controls in place is exactly if they do _not_ simply forgive these sorts of jumps in costs (as the CEO here seems to be claiming). I'm pretty sure if they'd be left holding the bag, they'd manage to find some way to cut off these kinds of jumps in usage.

op00to
3 replies
6h25m

Maybe it’s a tax dodge! “Forgive” 100k of “overages” which cost Netlify next to nothing, then report it as a write off on taxes.

SXX
1 replies
6h16m

I doubt IRS would buy that BS.

op00to
0 replies
6h3m

They’d have to be properly resourced to identity the BS first.

kjs3
0 replies
1h1m

That would potentially make this situation much, much worse (in the US tax system...YMMV). If Netfly forgives a business debt and reports it to the IRS as uncollectable so they can write it off, the IRS can consider all or part of the forgiven debt as income to the person who is forgiven (there are lots of details, IANATA/IANYTA, YMMV). I don't want a blindside 100k bill from my hobby site, but I sure as phuck don't want the IRS thinking I made an extra 100k of taxable income. I might be able to shame Netlify into forgetting about it, but the IRS is not usually so easy to deal with.

op00to
2 replies
6h27m

I’d rather be shut down than have a heart attack from a $100k bill. That could literally kill me from stress, even if you pinky swear to refund any oopsies.

pants2
1 replies
2h28m

See the Robinhood user who committed suicide after misunderstanding his liabilities from selling options.

phone8675309
0 replies
1h7m

Honestly I'd probably commit suicide if a hosting provider gave me a $100K bill.

Collect this from my corpse you scummy fucks.

jari_mustonen
2 replies
8h43m

"Current policy?" So, you will retain a right to change such fees when you feel like it.

This is a serious matter. We are building a new site for our company with Netlify, but we can't open ourselves to this predatory practice. And even if you do not mean to be predatory, even the option of such is enough.

If not resolved with a clean, legally binding promise, our company (and probably quite a few others) must move our business to Cloudflare, Amazon, or some other competitor of yours.

rafram
0 replies
2h51m

Presumably your company’s site won’t be on their limited free tier.

karaterobot
0 replies
2h33m

"Current policy?" So, you will retain a right to change such fees when you feel like it.

Is that unreasonable?

silent_cal
1 replies
2h22m

Good on you for reaching out, but getting a bill like this in the first place is enough to send someone to the psych ward, lol.

baggachipz
0 replies
1h49m

Email:

"You've got room to grow!"

ohfuckohfuckohfuck

jpambrun
1 replies
5h46m

By the time you forgive the bill you may have caused significant psychological distress, maybe even irreparable. This doesn't feel like a responsible approach.

is_true
0 replies
1h8m

This is the way most companies work unfortunately. Paypal limits your account and makes you wait 6 month to (maybe) give you a way to get the money back.

fabian2k
1 replies
8h32m

You can't rely on such a policy if it is not part of the actual contract. This doesn't address the enormous uncertainty and risk that is present here when using Netlify.

shit_game
0 replies
7h37m

This is what sticks out to me about the situation. I would much rather a site go offline due to service overage triggering at some limit that I set - simply relying on the good faith of a host to subjectively waive fees is not reliable nor does it instill confidence that I won't be financially ruined by malicious third parties (like nearly happened here). I would imagine that the good faith of Netlify in this case would mean very little to a court when there is a contract that stipulates costs for services, and the worst case scenario for a user is that Netlify could take the issue to court with the contract the user agreed to and demand full payment. Even the possibility for this situation to occur without any tools existing to prevent it is terrifying and is a terrible value proposition for a service.

bluerooibos
1 replies
7h11m

I assume you'll be offering this user a good amount of credit on their account for having to deal with this BS and the stress of being told they owe you $100k?

phone8675309
0 replies
1h7m

Fell out of my chair laughing

wendyshu
0 replies
2h19m

Is the support employee going to be fired for making such a traumatizing mistake? Or was 5% ok until this went viral?

underdeserver
0 replies
6h15m

You guys see a lot of traffic. Why not offer DDoS protection for the free tier by default?

tacone
0 replies
39m

Hello bobfunk, thank you for acting on this.

One question though, what is Netlify gonna do to ensure this doesn't happen again?

I understand it's a hairy question, but the general consensus seems to be some policy must be changed or at least some line should be drawn.

spacecadet
0 replies
1h26m

Never used "netlify", but to me a product is broken if you are using the words free and bill together.

I wont touch a fake free service if it requires a payment method. Want my money, give me a reason to pay you, dont trick me into paying you.

Temped to go fuzz your product and document other dark patterns...

pier25
0 replies
1h45m

instead forgiving any bills from legitimate mistakes after the fact

That's terrible for marketing.

phyzome
0 replies
36m

« Apologies that this didn't come through in the initial support reply. »

"Didn't come through" doesn't actually match the user's report of having support explicitly offering 20% and then 5% payment. It sounds like maybe you have a training problem? That seems like one of the important points to speak to.

kiprou
0 replies
8h35m

That customers must seek forgiveness at Netlify's discretion is not comforting. What's comforting is dependable spending controls.

jotaen
0 replies
3h0m

One additional feedback, for consideration: to me, your Pricing page[1] doesn’t make it sufficiently clear that the “Starter” plan may incur costs at all (let alone in this ballpark). It’s now more apparent when looking at it in hindsight, but you have to either read very carefully, or go to the separate “View Features” page to understand this.

“0$ to get started, then pay as you go” reads to me: “0$ to get started, and then you can order add-ons and extra features as you need them”, not “$0 to get started, but we may start charging you virtually unlimited amounts at any point without prior notice”.

When signing up for the “Starter” tier initially, I completely misunderstood this. I didn’t have to enter any credit card or invoice details, so I thought as long as you don’t have that info from me, you can’t and won’t bill anything.

[1]: https://www.netlify.com/pricing/

huxflux
0 replies
6h11m

So this one got attention due to some good Samaritan on Reddit who told OP to post here. Now, to the real question here: have others not received as good advice and just paid up?

heavyset_go
0 replies
7h41m

It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.

That doesn't square with the 5% fee on the original $104k that your company told the OP to then pay.

gizmo
0 replies
58m

That is an outrageous and inhumane policy. People get panic attacks when they get told they owe 100k they don’t have. People will be terrified your internal process wrongly determines the bill is legitimate. Imagine you have to study for an important exam or that you have a paper due. How can you possibly focus with this nightmare at your doorstep?

Truly shameful.

dboreham
0 replies
53m

traffic spikes that doesn't match attack patterns

I interpret this as "we always charge for traffic served, but we attempt to block illegitimate traffic" which means of course that the worse their traffic discriminator performs, the more money they make!

brnt
0 replies
7h42m

I understand that you need to pay bills, but auto-billing over the bandwidth budget just isn't OK, or at least not unless the user specifically configures that that's OK. I for sure didn't understand your bandwidth tiers that way.

You can avoid this sort of bad press and disgruntled users and your support cost by just giving users the option to shut down the site once the bandwidth budget is up.

bb81
0 replies
23m

So netlify is a major scammer organization now!? Uh oh time to look elsewhere

anonanoa
0 replies
7h28m

So what's the policy?

Do you forgive 100%, 95%, or 80% of the bill?

Is the 100% only available when a story about a bill goes viral?

ZeroMetaCool
0 replies
3h28m

Made an account here to also let you know, I too am removing my websites from netlify ASAP. Thank you for bringing this to light.

Sephr
0 replies
8h50m

Can you respond to the allegations that Netlify has inadequate spending limit controls? Are there plans to improve this situation?

CogitoCogito
0 replies
8h30m

It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.

The legitimate mistake sounds to be on _your_ side if anything. You failed to match the attack pattern after all.

Apologies that this didn't come through in the initial support reply.

The support email said you normally discount the attacks to 20%, but in this case it would be discounted to 5%. Are you here publicly claiming that your policy is to in fact to forgive (i.e. discount 100%) these bills? Was the support reply totally incorrect in claiming that you normally discount the attacks to 20% or are you lying when saying that your policy is to forgive the bills? You might want to clarify your position here.

ji_zai
85 replies
11h13m

This is my worst nightmare as a bootstrapped founder. And that there's no way to put a limit on spend is ridiculous. Someone that doesn't want me to do well can simply ddos me into bankruptcy out of nowhere.

Just went through Vercel's docs:

---

"Vercel helps to mitigate against L3 and L4 DDoS attacks at the platform level. Usage will be incurred for requests that are successfully served prior to us automatically mitigating the event. Mitigation usually takes place within one minute.

Usage will be incurred for requests that are not recognized as a DDoS event, such as bot and crawler traffic.

You should monitor your usage and utilize Edge Middleware to protect against undesired traffic based on its IP, User-Agent header value, or other identifiers."

---

That doesn't help me sleep well.

I feel that by now, these hosting providers should simply adopt best ddos protection practices and take responsibility for failure to protect.

"You should monitor your usage and utilize Edge Middleware to protect against undesired traffic based on its IP" - there should be some really good defaults for this right?

Clearly it's possible - Cloudflare's ddos protection is worded more strongly.

I'm willing to pay more for the service for peace of mind. Like, even $10/mo more to insure against getting smacked out of nowhere.

ignoramous
38 replies
10h44m

Cloudflare's ddos protection

Yeah, we got hammered once with over 10TB/mo and noped out of Netlify as fast as we could: https://twitter.com/rethinkdns/status/1370342245841342466 Had to pay the bill in full.

Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.

pacifika
28 replies
9h58m

That’s a free tier that doesn’t sound sustainable then, so that raises alarm bells to me.

DontBreakAlex
9 replies
9h51m

That's because amazon and big telecom convinced you that bandwidth is expensive. It isn't. Once the equipment is there, you might as well use it.

whatshisface
6 replies
9h35m

Wouldn't there be at least a handful of competitors if the economics worked out that way?

oefrha
0 replies
9h17m

A good number of small hosts offer very cheap bandwidth compared to AWS. With Cloudflare’s economy of scale, their costs should be even lower. You only need a ~100Mbps link to serve 30TB/mo, which would cost them ~$10, maybe less.

They’ve written about it before: https://blog.cloudflare.com/aws-egregious-egress

neurostimulant
0 replies
7h10m

In EU, yes. EU cloud providers offers bandwidth on the cheap, much cheaper than anywhere else.

happytiger
0 replies
8h36m

A good data center can sell you a sustained 10Gbps for, and I’m guessing at going rate, but like 4-7k a month? If you’re making a commitment cheaper, and that’s basically a retail pipe for someone in a colocated facility.

For larger providers, bandwidth cost drops tremendously, especially if you’re well connected as transit is much cheaper and if you are really large or a network provider you may even be routing between your own facilities or in some cases from one customer to another and every large scale isp is going to want a “direct link” to your facility (a peering relationship). Those costs are astronomically small at scale for bandwidth.

The ISP or similar then turns around and sells a sustained network throughout as GB transferred, which isn’t how wholesale bandwidth is sold at all. So the get to charge for the data the pipe moves while they only pay for the connection itself — the markup added to this process is considerable.

For someone operated a global CDN, which is basically what they do, they have racks of storage and computer collocated all over the world and optimize the living crap out of their network to reduce their costs and make it run on as many peering relationships as possible. It’s an expensive and complex business to set up, but once it’s set up you get a fairly good and consistent return out of it.

The reason for this article is related to the nature of that business: it’s the issue of liability.

When you have policies where you protect your clients from downsides and excessive use on the network, you suddenly have to assume the role of paying attention to what’s on the network and policing it’s contents. That’s not possible with a massive system like this generally, so they push the liability down to the customer and discount the mistakes that come up. That’s why things are set up like this… this kind of stuff isn’t their business at all really. They are looking for the customers that convert and pay, which is very profitable, and the free tier is often thought of as a sustainable cost if you are large enough scale, as it substitutes for the rather massive expense of marketing and sales which is one of the largest expenses in a bandwidth focused business. CAC is the free tier.

There also competitors, but the benefits of scale are tremendous in terms of cost efficiency. A large provider might be paying just a very small fraction of a penny or less (even “free”) compared to what a small provider is paying. So that’s why you end up with fewer competitors because it truly is a business that benefits from economies of scale.

There are other smarter people on here who can correct any mistakes I’ve made or provide better pricing or whatever, but that’s the more in depth answer.

bravetraveler
0 replies
1h39m

Have you not... looked? They exist - arguably too many of them. Clouds aren't a good indicator of reasonable pricing.

Saris
0 replies
37m

There are tons, the big providers like AWS, GCS, etc are really the only ones who charge ridiculous amounts for bandwidth and everything else.

Those big providers have pretty much normalized high fees and convinced people that's what it costs, the reality is any normal provider like Hetzner for example gives you tons of bandwidth for essentially zero cost included with your servers.

immibis
1 replies
9h14m

Well, they have to pay for the amortized equipment cost. Which, yes, is much less than you think. The big 3 clouds have set their prices in an age when services were much more expensive to provide, and they make a big deal out of the fact they've never raised their prices - but they rarely lower them, either. Now they have insane profit margins.

The invisible hand of the free market has come to fix that, *but you have to opt into the hand by shopping around.* If you don't, you don't get its benefits! You have to willingly take the choice to move to cheaper providers instead of overpriced ones.

Hetzner Cloud: $1/TB (20TB free) Digital Ocean: $10/TB (few TB free depending on server size) AWS: $90/TB (0.1TB free, used to be 0.001TB free) Netlify: $550/TB (0.1TB or 1TB free)

If you move up from $5/month VPSes, to real dedicated servers, you are now spending a lot more money and therefore you get more free perks. A huge number of providers exist that will give you unlimited or unlimited† bandwidth depending on how much you spend. Renting a powerful server with unlimited 1Gbps should cost a few hundred to several hundred dollars per month, and a powerful server with unlimited 10Gbps (i.e. 3000TB/month) should cost a few thousand dollars per month. You can even get some with 100Gbps (for tens of thousands).

Also consider asking your local ISPs and datacenters. If you live in a central area, you can probably get a comparable connection to a nearby datacenter if not straight to your office, for a comparable price. Data center connections are their bread and butter and they should be able to give you a quote quite rapidly; to your office will be a more custom thing.

Recently I got a quote for AMS-IX peering in Berlin, i.e. a peering in Amsterdam plus a link from Amsterdam to Berlin, about a 600km distance. That would cost 950 euros per month. If 1Gbps, it would cost 300 euros per month. Even though it's not really got anything to do with internet access (transit), I include this number to give some indication of the "true" cost of "raw" bandwidth.

blibble
0 replies
7h46m

Now they have insane profit margins.

"your margin is my opportunity"

lifthrasiir
4 replies
9h56m

I have heard that they rather drastically constrain QoS instead, which does sound reasonable. So you are still not charged for abusive traffic, but your service will be much slower than what is actually possible with paid tiers.

throwaway290
3 replies
9h49m

So you'd be either slow or pay them "for protection". Something that reminds me of;)

stavros
2 replies
9h21m

Capitalism? Mob-style "protection" would be if Cloudflare were the ones who DDoSed you if you didn't pay.

throwaway290
0 replies
9h3m

How naive if you think the mob would disclose when it's affiliates trash your shop.

gkbrk
0 replies
8h51m

Yeah. Instead Cloudflare hosts the websites of DDoS sellers and refuses to take them down or tell you who they are. A lot of these DDoS-for-hire services use Cloudflare to hide their real IP.

ilogik
3 replies
9h56m

it's 100% not sustainable. Use it while it's good, but don't get vendor locked in, because sooner or later they will increase the prices

ignoramous
0 replies
8h24m

it's 100% not sustainable

As a business for Cloudflare?

  Cloudflare in 2014 blogged about how they work relentlessly to bring down bandwidth costs by peering aggressively where possible [2] (which apparently means $0 for unlimited bandwidth [3]). And where they can't / don't [4], egress is 5x (est) the ingress (one pays for the higher among the two), but this creates an opportunity for an arbitrage and give away DDoS protection for free.

  This is pretty similar to Amazon's free-shipping offer for Prime customers despite it being one of the biggest loss makers to their retail business. Prime basically has since forced Amazon to bring down costs through building expensive and vast distribution & logistics network that spawns the globe. Doing so was a considerable drain on the resources in the short-run, but in the long run, it has become an unbreachable moat around its largest business.

  Analysts like Ben Thompson (stratechery.com) and Matthew Eash (hhhypergrowth.com) have written in detail about Cloudflare's modus operandii over the years, with both agreeing that Cloudflare's model is so brilliantly disruptive that even Clayton Christensen would be proud of it.
https://news.ycombinator.com/item?id=33337183

dbbk
0 replies
6h23m

They've been going for at least 10 years...

EasyMark
0 replies
14m

This is why we still use services on VM's and open source containers. We can move our services anywhere, including selfhosting. AWS and Google offer some amazing solutions, but lock in ain't worth it if you can manage your own stack via serverless/vm solutions.

danogentili
2 replies
9h56m

I believe it's quite the opposite, cloud has normalized absurdly high traffic fees, and that is what should be raising alarm bells.

rnts08
0 replies
7h41m

Yes, cloud services have inflated both bandwidth and amortized hardware costs to absurd levels. You pay for not having to know what to do in order to run something online. Until it breaks.

raffraffraff
1 replies
9h41m

Their stock performance would agree

actionfromafar
0 replies
9h36m

While a funny comment, stock performance is at best loosely coupled to sustainability as a company.

underdeserver
0 replies
9h54m

These guys know what they're doing. If and when Cloudflare dies we'll find something else.

kkielhofner
0 replies
4h23m

Peering.

Here's how it works:

1) I have a big network and I exchange traffic with another big network. Think of "eyeball" networks like last-mile ISPs (Comcast, mobile providers, etc) where a substantial portion of end-user traffic is going to handfuls of well known networks - Cloudflare, AWS, Netflix, etc.

2) Comcast and Cloudflare say "Hey, I send you X TB/PB/etc and you send me X TB/PB/etc. We both currently pay another provider to route that traffic between us. Let's not do that."

3) In locations where it makes sense they basically throw a cable across datacenters, POPs, internet exchanges, etc. The cost for this is typically extremely low - it's basically a port on a switch/router on each side and MAYBE a "cross connect fee" from the facility. This is usually billed in the tens of dollars/mo if at all. It takes very little time/effort to configure this but of course the details are more complex - multiple ports, multiple facilities, etc.

4) Both sides start routing traffic between their networks over their new shiny direct cables and extremely high speed ports. Faster throughput, lower latency, improved reliability, frees up bandwidth to the transit provider they were using previously, and most importantly the cost of bandwidth between the two networks goes to zero.

This is all well known and publicly available because it's visible in the global routing table(s). Cloudflare, for example[0].

All of the large providers do this and AWS, etc charging in bandwidth per GB (especially at their rates) is more-or-less pure profit.

I have a theory that AWS, etc capitalize on people not really understanding this anymore. AWS is 20 years old - that's an entire generation of CTO/CIOs on down that are completely unfamiliar with these details and think $0.10/GB or whatever is "just what bandwidth costs". It is not.

[0] - https://bgp.he.net/AS13335#_peers

blitzar
0 replies
9h54m

By the time it isnt sustaninable I will have IPO'd and be the next offensive new money tech billionaire writing threads on twitter telling you the secret to success is the 5am grindset and everyone who isnt sinking 5mil into the next big thing (tm) can have fun staying poor.

EasyMark
0 replies
17m

I think a lot of people don't understand how cheap bandwidth is and is decreasing in cost practically every day. Amazon and Google have a lot of people fooled. Go ask someone operating in China and East Asia (and Japan) how much they're paying for local solutions.

cube2222
3 replies
9h34m

CloudFlare pricing is indeed positively ridiculous.

At OpenTofu[0] we’re using CloudFlare R2 to host the providers and modules registry[1]. Bandwidth is free, you only pay for requests.

This already would be great, but there’s more - you only pay for requests that actually hit R2. So with an almost 100% cache hit ratio, we barely register any billable requests.

Recently someone decided to load test us and generated ~1TB of traffic over 1-3 days. All but a few of these requests were cached, so the whole situation probably cost us less than a cent.

[0]: https://opentofu.org

[1]: https://github.com/opentofu/registry

EE84M3i
2 replies
9h4m

Is this in line with the TOS? I thought there were restrictions on serving non-website content in the free tier, or does that not apply to the CDN if you're using R2 as an origin?

ignoramous
0 replies
8h26m

R2 as an origin

We front our distribution service with Cloudflare Workers fronting R2 fronting S3 / Lightsail Object Store (https://blog.cloudflare.com/cloudflare-r2-super-slurper/). That brought our costs down from $500 to $2 serving the same amount of traffic.

sph
1 replies
2h20m

Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0. Makes it hard to move to any other platform. As a small tech shop, this is my Hotel California I'm happy to never leave.

Yeah that's how Cloudflare can reach total control over the Internet. With thunderous applause by people that should know better.

I know that my position is outright blasphemous in this day and age, where even self-hosting a static site has become black magic and we need a third party to do it for us.

kirubakaran
0 replies
41m

I dread the day they go evil

nickjj
1 replies
5h33m

Cloudflare's free tier is ridiculous: We do over 30TB+ of genuine traffic for $0

It's not really ridiculous if you think about what you're giving them.

You are massively benefiting their platform by providing them data which they use to train their services and then sell those services to other customers.

I'd make a case that the data they collect is the most important part of their business and the free tier is a major component of this.

dim13
0 replies
2h6m

If you are not paying for it, you are not the customer; you're the product being sold.

dmw_ng
0 replies
49m

I don't think it's fair to call it their free tier - it's their discretionary tier, there are numerous cases of the rug being pulled as and when it suits their business requirements to do so. Being left homeless vs. urgently coughing up is exactly the wrong problem to be dealing with mid-attack, I can't see any way to consider it free by any practical definition

WyvernDrexx
10 replies
10h41m

Imagine you lost your job. So you are here enjoying creating and hosting your hobby projects in theses services. Now, suddenly one fine morning you get slapped with $104K bill because someone decided to randomly ddos your one page dog lover website.

Now, who in the would would be thinking of having ddos protection for their hobby project? This is just absurd thinking.

bbarnett
3 replies
10h23m

This may seem weird, but I believe ToS ae the real problem here. I call it the "car rental" problem.

When I rent a car in person, I am often given a contract. And this contract is filled with tiny print, and pages of it.

There are often people behind you, waiting, and bored/annoyed people behind the counter, waiting. This is beyond unreasonable.

A point of sale contract should be short, in readable text, and understandable. For example, renting a car? Under a page, easily parseable, and if the person behind the counter cannot explain it, it is null and void.

From a legal side, you can do this. And you can explain legal terms. Of course this means you are describing intent, which limits one in court, oh boo hoo Mr Lawyer. Cry me a river.

Well the same should be true of any retail contract. Sign up for a service? One page with costs listed.

At least then, there is hope of an end-user sort of understanding. And as one could claim that a DoS was actually targetting the provider, and not the website, that should be described too.

So back to the topic at hand. I would write a demand letter, insistong Netify explain the charges, and ask them if they and their IP ranges were DoS, and if so that the charges be reversed.

Because you shpuld not be paying, if someone attacks Netify.

This letter should also be sent by mail, sig required, to the corporate address too.

ryanjshaw
0 replies
8h47m

This applies even offline! Have you ever tried to get a hold of exact insurance policy wording before going through their entire sales process? Impossible, in my experience, whether it's long-term insurance, vehicle insurance, pet insurance, etc.

lifestyleguru
0 replies
9h53m

Every rental and service is so optimized against scammers and abusers that being a perfect legit customer ie. simply want to pay, use the resource, then return the item or terminate the service, you're walking along the edge of a cliff. Annexes, penalties, fees and charges, exclusions, "sign this one more form, everyone signs it". Housing rental is another extreme example, one is simply unable to just get a job in new location and rent something long term.

cxr
0 replies
8h18m

When I rent a car in person, I am often given a contract. And this contract is filled with tiny print, and pages of it.

As someone who reads the agreements I sign, one thing that has become prevalent is that they're so used to people not paying attention to what they're signing that they're sometimes not even giving you an accurate copy to review. For example, you read the thing and think, "Okay, I can work within these parameters," then you sign, and later get an email containing your "agreement", but it turns out what's in the email is a different set of terms with a bunch of stuff that wasn't in the terms you actually agreed to when you signed. Or someone hands you a pad with an "I agree to the terms" box checked beside the signature line, and when you ask to see the terms you're agreeing to, they're caught off guard (being totally unequipped to let you do that), which turns into being flummoxed with how to proceed, which turns into getting angry with you for asking.

signaru
1 replies
9h29m

Can't hosts just make a site unavailable once it reaches its plan's bandwidth limit, DDoS or not?

I think being offline is a lesser headache than a large bill, especially for those who are inclined to a free tier to begin with.

cxr
0 replies
8h41m

Folks regularly show up in HN comments during these discussions stating the opposite—that it's categorically better for all sites/projects, now matter how inconsequential, to stay online. It's weird.

This includes some of the TPTB, too. Occasionally, though, someone'll say the quiet part out loud. E.g. re fly.io:

putting work into features specifically to minimize how much people spend seems like a good way to fail a company

<https://news.ycombinator.com/item?id=24699292>

herbst
1 replies
9h36m

No. This is absolutely common. I remember well how shared hosters 10 years ago already put caps on cheap packages and took the websites offline in case of traffic. And today it's Amazon who bills small players into dept.

There are many provider who don't tho.

CalRobert
0 replies
9h32m

I always loved nearlyfreespeech.com for this, (prepay, and if you run out of money the site goes down) but found it to be a pain for projects that really needed a VPS

underdeserver
0 replies
9h49m

It shouldn't be like this, but it is.

Unfortunately, in today's world, DDoS protection is the equivalent of basic hygiene, foid and road safety. It's just a travesty that the hosting providers don't feel like it's their responsibility to address it.

aembleton
0 replies
9h16m

I always run mine through Cloudflare, at least in part for this reason.

rozenmd
9 replies
10h34m

This might be a good time to point out Cloudflare Pages: https://pages.cloudflare.com/

Under the free tier:

Unlimited bandwidth
itake
5 replies
10h26m

I'm moving everything to Cloudflare.

rubymamis
2 replies
9h44m

I'm trying to sign up but it keeps saying "Verification is taking longer than expected. Check your Internet connection and refresh the page if the issue persists."

Does anyone else experience this as well?

xrisk
0 replies
9h12m

Might be iCloud private relay if you use that

Arnavion
0 replies
9h35m

That's the Cloudflare user experience in a nutshell. Your users will see the same thing when they visit your Cloudflare-hosted site.

cod1r
0 replies
10h11m

Just looking at pages.cloudflare.com now and I think I'm going to be using cloudflare from now on.

neillyons
0 replies
8h55m

I didn't even know Cloudflare offered a JAMstack platform. I'm going to switch as I already use Cloudflare for domains.

jskherman
0 replies
9h32m

Yeah, I'm already using Cloudflare because of Google Domains got de facto killed by Google via transferring it to Squarespace. Why not Cloudflare Pages, CDN, and R2 (S3-compatible storage) too? I'm even considering paying for the paid tier in the future if I ever go above the limits of 20 000 files per static site and the 25 MiB single file size limit [^1] (more than enough right now or in the near future).

[^1]: https://developers.cloudflare.com/pages/platform/limits/

Winsaucerer
0 replies
10h9m

I was looking for a static site hosting option recently and tried out cloudflare pages. Fit my need perfectly. The generous free tier and the reasonable pricing model were the big factors.

Oh, and the ability to put some authentication in front of it was a big feature for me.

baq
4 replies
10h53m

Host on a provider which bills per hour. This caps your cost. It also makes your users pissed because you will go down, but if you’re small, you can afford that. If you’re big, you already have scaling options and should have a team to handle ddos.

teaearlgraycold
1 replies
10h45m

Yeah. Any host that won't infinitely scale out will solve this concern for you.

ehnto
0 replies
9h1m

I think most people pick Netlify for it's Infra as a Service offering, so it would be nice if they had a way to throttle and budget in that offering.

I would even imagine Netlify's target market is small to mid size businesses who really don't need ridiculous burstable scaling capacity at all. Seems like a bit of a trap door for that customer base.

I agree though, I wouldn't host on them as a small business due to that risk, but I am also happy running my own server so I might be an edge case.

cotillion
1 replies
10h35m

My experience is that customers don't really care that much about small amounts of downtime no matter what size you are, people mostly get that unexpected stuff happens as long as you don't get hacked or misplace their data. Customers might complain a bit but seldom leave because of a few hours downtime.

This seems to mostly hold true to developers also, GitHub manages to survive just fine after all.

chgs
0 replies
9h19m

Depends on your service. 20 second downtime on loading HN? Nobody cares. 20 second downtime on the last play of the Super Bowl - big problems.

For most internet consumers we’re accustomed to poor service so if a page doesn’t load we’ll assume it’s a local problem and try again 20 seconds later, same with buffering, it’s just something that happens occasionally. This is increasing the case for phone calls too. Legacy live tv and radio going silent though is still a major issue, especially on live events.

jefozabuss
3 replies
10h42m

If you want to sleep tight just get a dedicated server or VPS from something like Hetzner and/or combine with CDN providers like BunnyCDN - set up alerts just in case though. It takes more time and resources to manage it but you could save a lot on it in this case.

raxxorraxor
0 replies
9h3m

That is my setup after leaving AWS for some of my services (low user amount b2b).

I put in far less resources and maintenance after I had the system running. Especially if you need to manage the software running anyway.

mro_name
0 replies
9h54m

I'd even say build your system so as it can run on shared hosting. This way you even save the management.

herbst
0 replies
9h39m

This so much. My hetzner (best choice for a media server within Europe) has 0 downtime in 1.5 years. And exactly as you said I am using bunny as well, which costs me a few $ per year.

jart
3 replies
10h25m

Use a token bucket on your web server to catch abusive IPs and then blackhole them using `iptables -t raw -I PREROUTING -s ip -j DROP`. I know. I run https://ipv4.games/ which invites hackers to unleash their botnets, and the service runs on a small VM with only a few cores. It's been attacked by botnets with 49,131,669 IP addresses. There's no Cloudflare frontend or anything like that, because back when I used Cloudflare, the people who attacked the service would actually bring down the Cloudflare nodes before they brought down my web server. I doubt I've ever paid more than $100/month to operate the service. Please note that your service provider needs to have free ingress in order for this strategy to be effective.

wielebny
1 replies
9h30m

This strategy may work for a (D)DoS that is targeted to an application layer, but won't work if the attack is designed to exhaust your bandwidth.

Once you're receiving more traffic than you network cards can handle, it does not matter if you'll drop the packets with iptables or not.

I was the target of attacks that caused Hetzner to terminate my contract. I was leasing physical servers there, so I assume the attacks were overwhelming their infrastructure.

zettabomb
0 replies
8h51m

These days it seems that DDoS attacks are often not targeted at bandwidth either, but rather packets per second. It is (apparently) much easier to exhaust routing capacity with an inordinate number of tiny packets than with a still large number of large packets. Cloudflare has some fun ways to deal with this [0].

[0] https://blog.cloudflare.com/mitigating-a-754-million-pps-ddo...

bravetraveler
0 replies
1h35m

Eventually you're probably going to want an ipset, at least. Otherwise processing your chain will continuously cost more, and more, and more.

niceice
2 replies
10h37m

How do these hosting providers sleep?

chgs
0 replies
9h10m

In enormous mansions atop large piles of money.

MikeDelta
0 replies
8h49m

I guess on eiderdown pillows.

dstroot
1 replies
9h11m

Can anyone share an example of edge middleware that might protect you on Vercel?

codewithcheese
1 replies
10h22m

Vervel charges $400/TB for excess bandwidth, it's not even DDoS you should worry about, just moderate success.

chgs
0 replies
9h12m

That’s a crazy high bandwidth. Bandwidth isn’t free, but $400 will get you a month of 10gig in my local peering point, that’s 1TB in 15 minutes.

redbell
0 replies
8h34m

Someone that doesn't want me to do well can simply ddos me into bankruptcy out of nowhere.

An interesting story that expands on the above concept but a different vector entitled, "Illegal Life Pro Tip: Want to ruin your competitors business?" : https://news.ycombinator.com/item?id=36566634

op00to
0 replies
6h18m

“We leave your safe deposit box unlocked. You might want to forge your own lock and key. If we happen to notice someone stealing out of your box, we will let them grab as much as they can for one minute, then maybe install our own lock if our revenue is close to target.”

immibis
0 replies
9h34m

Those services exist, and you have the option to use them. Netlify is not one. Apparently, you chose that the un-insured solution was best for you.

Mandatum
0 replies
9h35m

Wait until you learn that Vercel only supports blocking IP CIDR ranges on the Enterprise plan.

CalRobert
0 replies
9h33m

Vercel seems to exist only to promote lock-in.

triyambakam
52 replies
11h43m

So if I want to migrate off of netlify to something better, where?

lopkeny12ko
13 replies
11h31m

For less than 1% of OP's monthly bill, you can build or obtain a more-than-enough server, drop your static files on it, and serve through nginx. And you get to keep it forever; there's no monthly subscription fee!

Seriously, maybe I'm just old, but I look at the pricing of these hip and modern SaaS products for dead simple software and I cannot believe my eyes. The "old fashioned way" works just fine (and has always worked just fine) and is orders of magnitude cheaper.

whatever1
5 replies
11h24m

But you need a static IP, stable internet, UPS, and permission from your wife to have a 24/7 powered noisy box in a room

mvdtnz
0 replies
10h17m

You don't need a static IP. You can use dynamic DNS or a free product such as Cloudflare Tunnel.

fuzzfactor
0 replies
3h12m

Dynamic type IP doesn't usually change for no reason, work around whatever instability you have, set BIOS so PC auto-restarts and OS launches apps when failed power returns, get a mini fanless PC which can be easily concealed, and you don't even need to tell anyone who doesn't have a need to know.

Whether that includes your wife or not is up to you ;)

fbdab103
0 replies
11h0m

I mean, the C10k problem was coined in 1999[0]. A 10W raspberry pi is going to be faster than any server of that day, so it depends on what exactly you are hosting if you need the noisy box.

[0] https://en.wikipedia.org/wiki/C10k_problem

Yodel0914
0 replies
11h14m

I pay less than $5/mth for a VPS with static IP and 1GB/mth transfer. If I get close to any of my CPU/Memory/Disk/Transfer capacity I get a friendly email letting me know that I might want to add more capacity.

LtWorf
0 replies
10h40m

How sensitive are you if you think a rpi is noisy?

satvikpendem
2 replies
11h28m

Hetzner or DigitalOcean with Coolify [0] works great, it's like an open source Heroku that runs on any host, you get git push to deploy, and a bunch of other features built in. It only works on one machine at a time though so it's not like a CDN but for small sites, it's great.

[0] https://coolify.io

tasuki
1 replies
11h3m

I use DigitalOcean to host some things (with my own setup for deployments with git push, because `curl | bash` is not a great way to install/maintain software).

How am I protected against extra charges for traffic?

welder
0 replies
10h3m

Same here, using DigitalOcean instead of Vercel to host my Next.js app. I have a billing alert to notify of unexpected bills, but I don't know what DO does if somehow TBs of traffic are sent to my apps or Cloudflare, because DO App Platform actually uses Cloudflare behind the scenes.

debok
1 replies
11h20m

Having built serverless apps and "old-fashioned" apps, I seriously believe the old fashioned way is better.

The best of both worlds is to host on AWS EC2 or a similar product from your web service provider of choice.

trog
0 replies
9h25m

EC2 is so much more expensive than a standard VPS from almost any other provider though. If you're not embedded heavily in other AWS products I don't think it's worth bothering with EC2 - LightSail is way more cost effective and gets you most of the features.

rtpg
0 replies
11h27m

I mean you gotta put your server somewhere (I guess hosting it on your connection?)

amluto
0 replies
11h26m

A lot less than 1%.

apsurd
8 replies
11h37m

https://render.com/ is free for static sites with custom domain support and SSL included. been happy so far!

used to use s3 for the longest time, but aside from costing a nominal fee, it's so unnecessarily complicated in this day and age.

boredtofears
3 replies
10h45m

used to use s3 for the longest time, but aside from costing a nominal fee, it's so unnecessarily complicated in this day and age

its like 10 minutes of setup tops to host on s3

apsurd
1 replies
10h33m

Not for me. It's always configuration voodoo. i've done it dozens of times and i used to think it's normal developer workflow. now i realize I was putting thumbtacks in my eyes for no good reason.

Creating an s3 bucket is easy enough. but you need to add the policy Json config to allow public access, and it has various versions across time and space. the one that works for me is like 15 years old iono. object resource "//*" something or other.

ok so now you have an s3-east-mybucket.com/index.html, ok custom domain that's route 53 yet more configuration vooodoo to point an s3 website enabled bucket blah blah.

wait! need SSL? oops actually that's cloudfront. need a cloudfront config voodo to point to an s3 config voodoo to your hopefully correctly configured route 53.

are you kidding me, 10 mins? You're a wizard. Now i do git push origin main and my sites up on render.com

boredtofears
0 replies
10h1m

yes, you need to spend the time to learn about the environment you are hosting your app in and then suddenly it won't seem like "voodoo" as much. you absolutely can set this up in 10 minutes (probably much quicker if you are adept with various infra tooling). theres even wizard dialogs for most of the things you've described right in the AWS UI nowadays.

Ringz
0 replies
9h52m

That depends on what level of knowledge you are based on. The UX of AWS offers even leading experienced admins and developers some surprising stumbling blocks.

jefozabuss
1 replies
11h23m

Just something to keep in mind: this also has about half of additional bandwidth costs (above 100GB) of the reddit post, so in your case you'd be billed for ~$57k or so with similar DDOS. At least they seem to provide monitoring/alerts based on their Security page.

apsurd
0 replies
11h18m

thanks for the heads up, need to do my due diligence here.

edit: refreshing that they highlight baked in ddos protection right up front on their marketing site: https://docs.render.com/ddos-protection

SXX
1 replies
11h15m

Keep in mind that according to their forum you'll be charged $30/100GB for bandwidth over free allowance of 100GB:

https://community.render.com/t/confused-about-the-free-tier/...

Exceeding allotted Bandwidth does result in automatic overage charges. $30 for additional 100 GB blocks.

So the same shady pracrice as on Netlify.

great_thanks
0 replies
8h56m

wow, thanks for mentioning that, though I'm wondering how this would work, since they didn't ask for any name/billing information at signup...

KingOfCoders
6 replies
11h36m

I use BunnyCDN to host several sites, they have a minimum cost of $1 per month and I usually pay $1 per month.

I run my sites [0] on Hugo and copy the generated sites (Makefile) to BunnyCDN with their command line tool.

It's a plain CDN, but does include DNS hosting for easy SSL certificates and has scriptable DNS [1] where you can run Javascript for dynamic DNS.

I went with them b/c they are in the EU, but I've stayed because I love them.

[0] e.g. https://www.amazingcto.com/

[1] https://bunny.net/dns/

thinkingemote
3 replies
10h22m

Looks nice but then so does Netlify at face level! Seems like it is pay as you go, and there is a settable max spend per month.

Does their ddos protection work differently than Netlify and could Bunny ever pull the same stunt with billing?

KingOfCoders
2 replies
10h1m

No they don't do Ddos protection AFAIK.

I think the main difference to me is

  Netlify  $550/TB [0]
  BunnyCDN  $10/TB [1]
You preload your account:

"In order to keep your service online, you are required to keep a positive account credit balance. Our system will automatically send multiple warning emails if your account balance drops beyond a certain point. If you fail to recharge your account, the system will automatically suspend your account"

[0] https://getdeploying.com/reference/data-egress

[1] https://bunny.net/pricing/

KingOfCoders
1 replies
4h34m

Just checked, you can also set a limit on monthly bandwidth.

"Monthly Bandwidth Limit (GB) - Limits the allowed bandwidth used in a month. If the limit is reached the zone will be disabled. Set to 0 for unlimited."

thinkingemote
0 replies
4h29m

Thanks!

mourner
1 replies
9h4m

Love Bunny too, wonderful service and great team. I wish there'd be an easy way to set up auto-deploy to Bunny Edge Storage on GitHub commit (to avoid doing so manually), but I guess it's not to hard to do through GitHub Actions.

KingOfCoders
0 replies
8h8m

Same.

toomuchtodo
4 replies
11h33m

Cloudflare. You will get a call if on a free or pro ($25/month) plan [1] if your bandwidth usage is so high it would warrant increasing your plan. Worst case, they turn you off. Preferred over denial of money attack based on your use case. Set and forget after pointing at your origin (time is money).

You'll get a call on any of their plans if your bandwidth usage exceeds certain thresholds, I am assuming your median usage is relatively tame.

Disclosure: Cloudflare enterprise customer, no other affiliation. I don't get anything for saying nice things.

[1] https://www.cloudflare.com/plans/

AtNightWeCode
3 replies
10h25m

You typically can't replace Netlify with Cloudflare. You need something like Github actions with some storage, S3 or something and then one can put Cloudflare in front of it all for caching, DDOS protection and so on.

yellow_lead
1 replies
10h20m

I believe you can replace most cases (static sites) with Cloudflare Workers [1].

[1] https://workers.cloudflare.com/

AtNightWeCode
0 replies
9h57m

Workers run before the cache so I would avoid this for static sites. One can use workers for dynamic routes on a static site though.

dombili
2 replies
11h34m

Github works well for me.

basil-rash
1 replies
11h30m

If it’s really pure static, github is great because it’s impossible for you to be billed. If you want some functions, Cloudflare free plan is nice because you can configure it to stop operating when the usage limit is reached, or pay $5 a month for more than you’ll likely ever need for a hobby project. Also bandwidth is free.

SXX
0 replies
10h38m

Do you need to configure Cloudflare to stop operating on limit?

I was thinking they never charge for anything unless you explicitly allowed it.

Matheus28
2 replies
11h30m

For static sites my choice is usually CloudFlare in front of a S3 bucket. It costs pennies. With cloudflare R2 it might be even easier.

Hamuko
1 replies
10h40m

Why not just use Cloudflare Pages?

Matheus28
0 replies
10h9m

It wasn’t an option when I setup my S3+cloudflare sites. It sounds like an excellent solution now.

zilti
0 replies
1h14m

Infomaniak

mirekrusin
0 replies
11h20m

Github static page.

maxboone
0 replies
11h41m

CloudFlare Pages/ Workers is pretty nice

lofties
0 replies
11h33m

I'm hosting a few large (multiple TB) projects on Cloudflare R2 with no issues for more than a year now. Super happy.

heavyset_go
0 replies
10h59m

I just migrated some projects to Cloudflare Pages for the time being.

dhaavi
0 replies
11h39m

Depending on your use case, Hetzner is as cheap as it gets while retaining high quality.

dalenw
0 replies
11h40m

I've always been a fan of cloudflare or firebase for static sites.

chronogram
0 replies
11h16m

For personal servers you can use the dynamic DNS feature on your modem in combination with Cloudflare if you like doing it at home, if it's for your business you can see if you can afford the €2 per month for Hetzner managed hosting or your local equivalent.

bsimpson
0 replies
10h52m

Firebase Hosting is rad.

INTPenis
0 replies
11h40m

I use AWS, S3, Cloudfront, ACM with a budget alert.

jzebedee
45 replies
11h38m

The most bizarre thing is that this is a known issue that folks have asked them for ways to mitigate, to no avail. The reddit thread even links to an extremely weird dialogue where Netlify's response boils down to, "if you're hosting a small site that gets DDoS'd, don't."

https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...

injuly
27 replies
11h7m

I'm hesitant to use "fancy" cloud service/hosting providers for reasons like this.

I don't understand why they won't just raise a 503 if the traffic exceeds the spend limit, or at the very least provide that as an option.

sshine
19 replies
11h3m

What's spend limit?

Autoscaling is a feature!

carlmr
18 replies
10h56m

I guess we need regulation for this.

throwawaaarrgh
8 replies
10h48m

Or, rather than creating more regulations, people could read the contracts they agree to when they get service, and use a competitor if they don't like it

baq
5 replies
10h33m

Yeah exactly why we need regulation - so contracts aren’t 100 pages long. DRY.

anomaly_
4 replies
9h49m

So, go to a different provider?

baq
3 replies
9h29m

More like don't start a business at all because of provider risk.

immibis
2 replies
9h3m

Or use a provider with a predictable cost structure. There are PLENTY. You didn't need to choose this exact one.

carlmr
1 replies
7h49m

The problem is that it isn't entirely clear which ones have predictable cost to the non-lawyers eye. I.e. they should have to have sane defaults, like reasonable spending limits and opt-out, by regulation, since the market is failing here.

immibis
0 replies
1h48m

It was clear though. Does Netlify not tell you their bandwidth pricing?

malka
0 replies
9h58m

I don't know many people knowledgeable enough to read and understand legalese. Except lawyers ofc.

bigfudge
0 replies
10m

There is such a thing as an unfair contract. Moreover, there are business practices that can become a local maxima in an industry and squeeze out competition which would actually be a net benefit to most consumers. Mobile roaming was the same before the EU intervened, and were now going back to the shitshow tjat preceded it in the uk.

arthur_sav
4 replies
9h57m

We don't need regulation for everything. Let customers vote with their wallet or even start a few lawsuits.

Adding more regulation makes the system slower.

xdennis
1 replies
9h2m

Microsoft is the only cloud provider with a spending limit. One choice is not competition. That's why regulation is needed.

rajamaka
0 replies
9h26m

Regulation is needed for lawsuits

immibis
0 replies
9h6m

I'd be in favour of a regulation to allow them to set a spend limit, opt-out.

I'm fine with their pricing structure right now, since you have PLENTY of providers to choose from - people can easily vote with their wallets and there's no problem that needs solving.

However, the unexpected spikes are a problem, and providers seemingly don't provide any way to solve them because they make more money by not solving them. A regulation to require all providers of post-billed services to provide spending limits would make a lot of sense.

Of course, customers should also have the option to opt out of the limit or set a very high limit.

This should apply to any service that's billed by usage calculated afterwards, not just web hosting, and not just technology.

jeroenhd
3 replies
10h28m

Why do we need regulation? "Keep the service up no matter what happens, no matter the cost" is a useful business model for companies that make the mistake of promising too many nines to their customers.

The issue at hand is that people put small websites on hosting providers designed for megacorporation wealth, like Netlify. I highly doubt the average blog needs more than a $10 VPS located in one single country without automatic fallback to another data centre. You can probably even go with a $5 VPS if you don't care about the first wave of HN front page bots not being able to reach your site.

Root_Denied
1 replies
9h49m

"Keep the service up no matter what happens, no matter the cost" is a useful business model

I mean, yeah - but that shouldn't be the default and it shouldn't be something that you can't opt out of if it is, which is what sounds like happened with Netlify.

jeroenhd
0 replies
9h35m

Why would Netlify offer the option to opt out when extreme availability is their core business? I'd argue that people are using the wrong service provider if they need to opt out in the first place.

Same goes for most of the other pay-as-you-go providers that turn HN into billing support every now and then; very rarely do I see "we suddenly got a $20k bill" posts about services that these extreme availability products make sense for.

jpc0
0 replies
9h37m

You can probably even go with a $5 VPS...

You wouldn't believe the amount of times I've said this and the response was "but it costs me nothing right now"...

jpalomaki
5 replies
10h18m

Playing ”devil’s” advocate: tracking spend in real-time is not trivial. It adds complexity to stack. Bugs in the feature can cause sites to go down (for long time) without a reason. Larger online businesses likely rather sort out the problems later than risk shutting down in the middle of unexpected success.

(But I also would like to see this feature)

INTPenis
2 replies
9h59m

Not really. AWS has budget alerts right? And I can read those budget alerts through their API.

So it would be trivial for me to poll their budget API for an alert, and immediatly trigger a shutdown of my Cloudfront service. Why can't they do that for me?

jpalomaki
1 replies
8h43m

"AWS Budgets information is updated up to three times a day. Updates typically occur 8–12 hours after the previous update" [1]

Something based on this could be definitely better than nothing, but might also give false impression of safety.

[1] https://docs.aws.amazon.com/cost-management/latest/userguide...

INTPenis
0 replies
8h40m

It's something. I started looking into the budget alert docs and it does use SNS so it should be easy to have something polling that queue and respond in any way necessary.

I'm imagining an alert to the on-call team, and a soft shutdown until the on-call team can figure out the next step.

If it can save a few thousand dollars, it's worth it. Each business must make their own estimate of course.

interstice
1 replies
9h59m

Vercel will happily tell you how much you are spending in pretty much realtime as it sails past your budget

leerob
0 replies
4h37m

OP is right though, realtime alerting is non-trivial to build. It looks us a lot of work at Vercel to get right. We also offer budgeting options where you can set spend limits now, too.

Saris
0 replies
33m

Yep, for a static site you can throw nginx on some VPS for $10 a year and it'll handle a decent amount of traffic.

Lammy
8 replies
11h20m

https://www.netlify.com/security/ sez “Active DDoS mitigation — Netlify monitors for traffic pattern anomalies and spikes, and effectively controls for them as needed” and now I'm curious about what that actually means.

serial_dev
3 replies
11h15m

It means that they will charge you 20k (a year's rent for me, no biggie) instead of 100k for your free website, or 5k if you got lucky.

DANmode
2 replies
11h6m

If you value uptime, even through being massively attacked, they can offer you that.

syrgian
0 replies
8h23m

I had the intuition that Netlify are extremely incompetent compared to Cloudflare, and this thread adds another data point. So no, if you value uptime you are not going to rely on them.

asmor
0 replies
10h44m

That is not unique to this price point and most of their competitors do not charge for unusual traffic spikes.

PaoloBarbolini
2 replies
11h17m

It means they protect themselves from layer 3 and 4 DDoS. For layer 7 you're mostly on your own. That's what most companies mean when they talk about DDoS anyway.

xyzzy123
0 replies
10h54m

Right and as a CDN they HAVE to handle layer 3 & 4 DDoS themselves so it's not like they're doing you any favours. The traffic is typically routed to the customer based on SNI.

Lammy
0 replies
11h4m

I found https://www.netlify.com/blog/2017/03/28/why-you-dont-need-cl... and it sounds like you're right.

“The cool thing is that we also provide a load balancer, and if our system has detected that our main load balancer is currently being hit by a large DDoS attack and is slow or unresponsive, we’ll simply route around that on the DNS level. Since we cache content at our edge nodes around the world, end users also experience extremely fast page load times because of this.”

weare138
0 replies
9h19m

They reroute the network traffic to ensure none of it gets dropped so they can accurately overcharge you for the the correct amount.

ehsankia
2 replies
11h28m

in other words, "if you're thinking of using netlify, don't".

geek_at
1 replies
11h10m

true. I have a 9€/mo vps at Contabo for my blog and once boasted on HN that my small VPS is able to handle reddit/hn hugs which one user seemed to take personally and they started a DDOS against my VPS.

I only realized this after Contabo contacted me and said the traffic is so high that other clients service is also degraded and they will have to take my VPS down if its much longer (which was understandable). Gladly the ddos stopped soon.

But never was there any talk about any cost, they were very supportive

jojobas
0 replies
10h24m

Even then Cloudflare forward proxy capable of real ddos handling wouldn't cost you $25 per month, some 0.5% of the 95% discounted bill.

But hey - just think about how much you saved on Netlify! Composable!

tsimionescu
1 replies
10h29m

To some extent, that answer is fair enough, assuming they make this clear up front. If their service is "we'll keep your site up no matter what, for a price" that's a fine service to offer. It's not what the vast majority of people want, of course.

If their advertising is targeted to small businesses and individuals who could never afford this type of service, they could be guilty of false advertising, at least morally guilty. I haven't seen their marketing so I wouldn't want to say.

bigfudge
0 replies
8m

Their marketing is very much like this. It’s completely misleading. They are definitively not selling “keep it up at all costs, money no object”

op00to
0 replies
6h12m

“Stop dressing so sexy if you don’t like the attention” is the vibe I got.

echelon
0 replies
10h36m

This rings so true!

I've dealt with Netlify's support [1], and one of their CS heads was incredibly rude to me and blamed me for the problem they created.

[1] https://news.ycombinator.com/item?id=35610956

OJFord
0 replies
1h17m

I don't fully understand Netlify, but it seems though it tries to be a one-stop solution for everything it doesn't have to be - you could put free Cloudflare in front of it and probably mitigate this kind of thing?

https://docs.netlify.com/domains-https/custom-domains/config...

dhaavi
38 replies
11h40m

And since my amount is too large, they offer to discount to 5%, which means I still need to pay 5 thousand dollars.

If they just reduce to 5% like that, it shows how disconnected this is from their real bandwidth cost. Really does feel like a scam.

rafram
25 replies
11h26m

Eh, I wouldn’t say that’s necessarily the case. AWS support, for example, tends to be really good about waiving charges for things that are clearly your mistake, like an unused instance that you forgot to turn off for a couple months. That’s not because hosting instances doesn’t actually cost Amazon anything! It’s because they want to keep you as a customer even if it loses them a bit of money right now.

In the Netlify case, though, insisting that this person still pay 5% is downright insulting. I’m sure they’re taking a hit already - just waive the whole thing.

Johnny555
13 replies
11h17m

That’s not because hosting instances doesn’t actually cost Amazon anything

Except it doesn't cost them anything. The marginal cost of keeping your single instance running is $0 (unless they were 100% out of capacity and they could have sold that instance to someone else either at full price or spot price)

johngladtj
10 replies
11h6m

Electricity costs money

littlestymaar
7 replies
10h43m

The electricity overhead of keeping an idle VM on an already running host is nearly zero though.

tsimionescu
6 replies
10h20m

Sure, but the electricity overhead of keeping one host running just to run an idle VM is not as trivial.

littlestymaar
5 replies
7h39m

But that's not what's happening: they aren't keeping a full host for you.

Your argument is like saying that a bus traveler costs the gas needed to power the bus, but it's never the case: the bus would be cruising no matter what. And symmetrically the VM host would be up no matter what you did with your instance.

tsimionescu
4 replies
4h26m

I'm not sure that's true. Shutting down unneeded hardware seems like a very simple but major optimization.

littlestymaar
3 replies
3h48m

You assume that the hardware would be unneeded, but that's a very strong assumption.

It would be very bad for any cloud provider to leave hosts with only one VM running on it, and you can be pretty sure only very small minority of their park that end up in that situation where shutting down a single VM would lead to a shut-down of the entire host, because it means that the host was vastly under-used in the first place.

tsimionescu
2 replies
2h22m

As far as I know, most cloud hosts don't actually support automatically moving live VMs, so I think it's fairly common for a host to be left running a single VM.

At least in AWS, they never supported this, and in fact may require you to reboot an instance occasionally in order for it to be moved to a new hardware host (typically when they are upgrading their hardware).

littlestymaar
1 replies
1h11m

But why are you talking about moving VMs?! Looks like you're adding tons of far-fetched speculations at every step of your reasoning.

The way you easily deal with this issue is very simple and does not require moving VMs: you just allocate newly spawned VMs to existing hosts with available room! When you do so (and they obviously all do!) you end up with little unused hardware…

tsimionescu
0 replies
14m

Say you have 3 hosts, each with a capacity of 10 VMs. At some point you have 28 running VMs - 10 on host1, 10 on host2, 8 on host3. Someone then closes down 2 of the VMs on host1, and 7 of the VMs on host3.

Now you have 19 VMs running, but need to keep all 3 hosts powered. If you don't have live VM moving, you are now forced to keep Host3 running only because 1 VM is running on it, even if that VM is idle. So, this one idle VM is responsible for all the energy consumption of host3, and will continue to be so until at least 3 more VMs get started (since you have room for 2 more VMs on host1).

If you did have live VM migration, or if the idle VM were powered down instead of running idle, you could close host3 completely, moving the VM to host 1, and only re-open host3 if needed for new VMs.

This is equivalent to the problem of memory fragmentation. Even though overall usage is low, if host usage is highly fragmented and you aren't allowed to move used memory around (compacting), you can end up consuming far more than actually needed.

JW_00000
1 replies
10h44m

But no electricity is used if your instance is up but idle.

tsimionescu
0 replies
10h26m

What does "idle" mean? Both a Linux or Windows OS not running any active software will still do computation and even network traffic (disk cache wrangling, indexing, checking for updates, NTP clock syncing etc), and requires electricity to do so.

It's very low cost, especially if its on a VM from a host that otherwise runs other VMs, but it's not 0. And if it happens to be the last VM preventing a hardware server from completely powering off, then it's actually quite far from 0.

pxeger1
0 replies
10h51m

“There is a good chance it costs them $0” = “in expectation it costs them >$0”

littlestymaar
0 replies
10h41m

This exactly.

toomuchtodo
7 replies
11h19m

Our AWS TAM says they don’t do this anymore, and we spend tens of millions of dollars with them annually. n=1, ymmv

hnlmorg
3 replies
11h13m

You presumably already have enterprise pricing discounts agreed though?

toomuchtodo
2 replies
11h12m

Yes, but nothing novel, and this is a recent development (within the last few weeks).

hnlmorg
0 replies
9h22m

I wonder if you’ve hit some kind of internal limit. I don’t know if such things exist but I’ve noticed a pattern around how discounts and credits are allocated.

I do feel your pain though. Managing AWS costs can be a full time job itself.

gcbirzan
0 replies
5h25m

Is it, though? We've been getting a lot of pushback for months, even for things that weren't really completely our fault (and were made worse by the horrible lag of cost explorer), or even for things that were aws bugs. Maybe now it's official policy, but definitely it's been hard to get refunds for a while now. They were throwing tens of thousands of dollars of credits at us to just play with new services a year and a half ago.

d1sxeyes
2 replies
8h23m

Nowadays for customers spending millions of dollars you'd expect (at least, Amazon would expect) that the customer has a FinOps department who are already working on getting the most 'bang for their buck' out of what they're paying for and minimising their spend, and they would jump to another platform in a heartbeat if they thought they could save money. It's not unreasonable to think that you don't need to do these customers any favours to keep their business, because those customers are big enough to look after themselves.

For smaller customers, the friendliness of customer support and the flexibility to help them if they make mistakes is much more likely to be a retention consideration. And who knows when a company spending 3 digits a month becomes a customer spending 6 digits a month? You want to be the provider of choice in case the company grows.

rafram
0 replies
2h55m

Yeah, exactly. I’m talking “I got billed $15 for an instance I haven’t used for the last few months. Can you refund me?”, not “You guys mind writing off a million or two?”

Repulsion9513
0 replies
3h17m

AWS will save us so much money! We don't have to pay for people to look after hardware! ... just pay for people to set up AWS, and maintain AWS, and make sure we're not paying thousands extra for AWS...

mewpmewp2
1 replies
11h9m

I would say it is a scam, because you can't set a budget limit.

szundi
0 replies
10h51m

Stunning

onion2k
0 replies
10h49m

AWS support, for example, tends to be really good about waiving charges for things that are clearly your mistake, like an unused instance that you forgot to turn off for a couple months.

This is an admission that their UX sucks and makes it hard to know what state your account is in and what you're paying for. They waive the fees because a few high profile cases of people paying thousands due to the AWS console being awful would drive a lot of customers away.

mrtksn
2 replies
10h41m

it shows how disconnected this is from their real bandwidth cost

It's a value added service, they don't trade bandwidth as a commodity. Therefore unfair characterisation.

Plus, if you dive deeper: Bandwidth doesn't cost anything because bandwidth is just about pulsing some light in some glass fiber and applying some minuscule voltage on some metal fiber.Okay, maybe it costs some amount of electricity but all this is just a business model for paying on capital expenditure through time share arrangements. People can have all kind of models for this, for example you can come together with others or pay it all by yourself to install the equipment and have free bandwidth for the lifetime of the equipment.

It's all just arrangements to cover the capital investment and earn something on top of it. That's not a scam. A scam would be if they didn't account correctly for the timeshare usage or induce usage to boost payments.

notpushkin
1 replies
7h56m

they don't trade bandwidth as a commodity

I really don't get your point. If you're a hosting provider, the very thing you're selling is bandwidth (and disk space). Everything else is a value added service.

mrtksn
0 replies
7h20m

I disagree, they are not a colocation service that happens to rent servers. They are opinionated platform for deploying web applications in a specific way. The bandwidth happens to be a necessity to do that and also a useful metric for billing by usage.

littlecranky67
2 replies
11h19m

Especially since they admit it was a DDoS attack. What I find outrageous is first that they charge for incomming traffic (which is often free with other providers), but also 55$ per 100GB. For comparisson, Hetzner charges you 1€ per 1TB of outgoing traffic while incoming is free.

klaustopher
1 replies
11h14m

If the attacker downloaded the 3,44mb audio file that OP mentioned, aren't we talking about outgoing traffic?

esperent
0 replies
9h35m

Even so, on Hetzner it would have cost them a total of $164, with no discount. On Netlify it's over 500 times higher, apparently.

Voultapher
2 replies
10h36m

They overcharge egress by about 500x https://getdeploying.com/reference/data-egress.

So even a reduction to 0.2% would habe been possible. Honestly don't understand why anyone feels comfortable overpaying so much. Especially when there is no configurable spending limit.

tombolino
0 replies
4h12m

no, they charge 6x according to that table. Because they are using aws

oefrha
0 replies
8h56m

It wouldn’t be possible for them. Netlify doesn’t own transit, so AWS needs to get their fat cut even if Netlify waives their fatter cut.

PUSH_AX
2 replies
9h27m

Some quick back of napkin math says 190TB would cost about $12k in AWS cloudfront costs.

boesboes
1 replies
6h52m

FYI: From my experience, if you do more the 20-25TB, you can get 75% off no questions asked.

dbish
0 replies
1h26m

If you’re regularly a large customer like that, seems like just moving to AWS directly would make the most sense right?

LiamPowell
21 replies
11h39m

I don't see why people are surprised by this or why people are calling it a scam. Netlify and others are extremely transparent about the fact that there are no limits. I completely understand not liking it and can see why the lack of limits would make it a bad option for plenty of people, but I don't see how it can possibly be called a scam.

hayst4ck
9 replies
11h20m

Because it's unbounded liability.

Not to mention the strong conflict of interest for netlify, who stands to gain from their customers being attacked. Netlify is getting paid for something criminal in nature having occurred.

It's like who is responsible for credit card fraud? If customers are responsible for credit card fraud, and it's their responsibility not to get scammed, then who implements fraud prevention measures and what effect would that have on the volume of fraud?

jeroenhd
7 replies
10h8m

Companies like these give out ridiculously huge free tiers in the hopes that very few users end up using the high free bandwidth limits. In most cases, they do. However, they do need to make their money back somehow.

I don't really get why people put their tiny static sites on hosts designed to never fall over no matter the traffic generated, no matter the situation. You're running a blog, not a government service. You don't need AWS or Netlify.

The ability to withstand almost any DDoS attack for a high price is a valuable service. It's not a scam. The people who get these huge bills just picked a hosting service that doesn't fit their requirements. I can promise you that the $3 shared hosting providers won't charge you $5k, five minutes after the DDoS starts your site just goes down.

chatmasta
2 replies
4h13m

they do need to make their money back somehow

You're assuming Netlify is paying for bandwidth in $/GB, when in reality they're probably paying $/gbps and thus have no costs to cover when a customer temporarily bursts their bandwidth.

jeroenhd
1 replies
3h11m

It doesn't really matter how Netlify ends up paying for their traffic, at the end of the day, there's a bill to be paid.

In your example, a DDoS sucking down bandwidth would cost more than a DDoS would had it been about total transfer volume. Their servers can only produce a set amount of network traffic at a time and on one single day, this one customer sucked up 5½gbps continuously, based on the 60TB figure provided in the reddit post.

This kind of extremely bursty traffic takes capacity that would otherwise be usable for tens or hundreds of customers, but to meet their guarantees, they must scale out massively to catch these bursts. I think it makes sense that making them dip into their bandwidth reserves should cost more than the average cost of a network transfer.

I don't know the actual costs Netlify has, and I'm sure the support rep saying they can drop this down to 20% or even 5% shows that there's a buffer here, but the 5 grand OP was asked to pay seems to come awful close to what you would pay on other high-reliability providers, such as Amazon. The max fee is probably to push their expensive customers into special deals (or to their competitors), but I find their 5% offer quite reasonable.

chatmasta
0 replies
2h46m

If they actually had to pay those costs, I promise you they wouldn't be letting their customers run up a bill without a credit check.

There's also the question of whether Netlify is even accurately tracking this bandwidth...

signaru
1 replies
9h16m

Any suggestions for hosts that will just make your site offline once it reaches its tier limit? Cloudflare and Netlify get suggested a lot and I was considering one of them before this.

jeroenhd
0 replies
7h9m

From what I can tell, OVH allows for unlimited traffic, unless you host in Sydney or Singapore.

Budget hosters will either cut you off completely (shut down your VPS) or throttle your network. For instance, Contabo doesn't charge extra, but it does reduce your network speed to 100mbps if you're exceeding an average connection speed of 100mbps over a timespan of 10 days. Leaseweb offers you the choice to power down a VPS when exceeding the bandwidth cap (though this is disabled by default).

If you need more bandwidth, Hetzner is popular, and charges around €1 per TB of bandwidth if you exceed their free bandwidth (+VAT, the $104k bill would be €40 under Hetzner, as 20TB is included for free) and provides configurable automated traffic email notifications before you hit that. Personally, I would add a warning after the very first terabyte, because I don't know what personal project even uses that much bandwidth.

Their dedicated servers don't seem to have a bandwidth limit, though there seems to be a fair use policy (there's this thread: https://lowendtalk.com/discussion/180504/hetzner-traffic-use... where a user complains about Hetzner threatening to end the contract after exceeding 250TB of traffic).

Many VPS providers and shared hosters won't send you these huge bills, but you should always read up on their policies when renting servers of any kind. These hosters don't come with free tiers (which I assume is the reason people consider services like Netlify in the first place) but they will usually tell you how they deal with bandwidth issues in their FAQs.

jskherman
1 replies
7h32m

Regardless, at the end of the day, budgets still need to be followed whether you're an individual or a business. It's simply insane in the first place that someone on the free tier would want absolutely no downtime regardless of how high the traffic is. For that, it would make sense for such an individual to be already on an Enterprise plan if they do expect it to likely happen and for which many do not.

jeroenhd
0 replies
6h58m

I think it depends on what you're using the free plan for. If you're kickstarting a business and manage to attract a wide audience by getting featured on HN/Reddit/the news, you may want to sacrifice a few thousand dollars for the user growth that all of this traffic provided. Paying enterprise pricing doesn't necessarily make sense if you're normally getting less than a few thousand visits per day. Same goes for all the hip and cool server solutions such as "serverless" cloud functions.

The core product is already enterprise-grade. Netlify's pricing page basically turns into a "contact sales" button when you select "enterprise", probably for businesses that did their math and are trying to get a discount. Everything about their website seems to target medium to large businesses or hopeful startups.

orlp
0 replies
8h52m

Not to mention the strong conflict of interest for netlify, who stands to gain from their customers being attacked. Netlify is getting paid for something criminal in nature having occurred.

I think you could argue that Netlify is guilty of racketeering in OP's case.

1. They admit illegal activity happened (a DDoS attack).

2. They demand money to be reimbursed for the illegal activity. However, the reimbursement they ask is several hundred times higher than the actual damages incurred.

romwell
4 replies
11h32m

Netlify and others are extremely transparent about the fact that there are no limits

Are they also transparent about the fact that they

1. Won't do anything about a DDoS, and

2. In case there's a DDoS (or some other unusual traffic spike), you'll only get notified waaaaay after the fact when you get the $100K bill, instead of getting a timely alert that would allow you to shut your site down to prevent getting extreme charges?

No and no.

It's a scam.

LiamPowell
3 replies
11h24m

The primary purpose of these services is to be able to scale up and continue working under heavy load, shutting the site down when this occurs would defeat the entire purpose of the service. I would say that they are transparent about both of the things you have listed by virtue of being one of those scaling serverless hosting services.

romwell
0 replies
10h10m

How about letting the user decide whether they want to scale beyond a certain point or incur huge charges?

Not too mention: if the primary purpose of these services is to allow a DDoS and then charge the user for it — then, yup, you're guessing it right: it's a scam.

When their business model makes DDoS attacks profitable for them... They're not in the hosting business, they're in DDoS/extortion business.

kobalsky
0 replies
10h54m

there is a middle ground between “I missed the spotlight because the service went down” and “this bill has ruined my life”.

they could ask the user for their budget when they are setting up their account as a basic guardrail, or they could give you a call

heisenbit
0 replies
9h54m

The primary service of these starter accounts for single users is way different.

emnudge
1 replies
11h25m

My previous understanding was that service would be stopped once you hit past the free tier.

Upon review, it does not look like this is the case. I have several very low traffic projects on which would have never been anywhere close to the free limit. However, if I get involved in a random spam attack, it seems I could be on the hook for several thousand dollars.

This is incredibly dangerous. Netlify is often used as a beginner friendly free tier for static hosting. Not as something that is cheap, but as something that is free. This is just an overall dangerous position to put people in.

LiamPowell
0 replies
11h21m

It does say it's pay-as-you-go on their pricing page. However they probably should have a giant warning page for new users who don't know that this is how this kind of service works if they want to target the beginner web-dev market. As far as I know, no other similar service has this though.

jug
0 replies
10h3m

I understand the lack of limits but I haven't accounted for DDoS attacks on Netlify infrastructure to impact me. I was assuming this only included real, "organic" traffic.

What I think Netlify needs on their Plans page is to include "DDoS attacks is included in your traffic" as well as their 20%/5% charge system.

and can see why the lack of limits would make it a bad option for plenty of people

Just out of curiosity, can you see any scenario where it WOULD be a decent option to use a free tier where you may be hit by a $20,000 or $5,000 bill out of the blue and outside of your control? You say "plenty" so I assume you consider this a reasonable system to some?

eviks
0 replies
10h23m

extremely transparent

they probably should have a giant warning page for new users who don't know that this is how this kind of service works

Pick one

diputsmonro
0 replies
11h23m

It smells like a scam, because they can suddenly bill any user they want for a scary number like $100,000, then when the user complains they "generously" reduce that to only 5%, or $5,000, hoping the user will just pay the massively reduced cost. This kind of thing - showing a huge number upfront then reducing it to a "small" number - is a classic scam.

Who controls the DDOS bots? Are they truly a separate entity? There is no direct evidence to link them together, but you would think that an honest company would be more proactive in preventing problems like this for their customers.

According to the linked reddit story, this is a known issue with Netlify and their response to past incidents is basically to pound sand. It all adds up to them purposefully trying to find ways to generate a high bill for their customers and hoping a small amount will pay for it.

batmansmk
0 replies
11h9m

I looked up the definition of scam and the goal is to make money out of the naivety of the victim. It involves a crisis, the illusion of shared exposure to a risk.

The price of cdn bandwidth is about 0.01/gb on low volume (cloudflare, aws, azure…) so op should be billed around $500 with 40TB. Netlify probably buys this for way less. He was presented a bill at $104k, « generously » reduced to $5k, still a x10 margin. Vercel and Netlify are outrageously expensive for what they do.

isoprophlex
17 replies
11h38m

There's no way in hell anyone should ever under any circumstance use a free service that might, for reasons entirely outside your control, suddenly bill you 5k, or 104k... or any non trivial amount really.

Just suspend service on excessive overages...

gullevek
9 replies
11h30m

Like in the past, when you went over your limit your page went offline. The good old Slashdot effect

iforgotpassword
8 replies
11h13m

Yeah Like wtf is wrong with that? Are people just to lazy to check what the conditions are when exceeding traffic? I'd never ever sign up for anything that just keeps charging...!?

JW_00000
2 replies
10h39m

But then where would you run a small hobby project that you'd like to run in the cloud? I have some small stuff I'm running on Google Cloud Platform, and honestly I'm scared of the same thing happening to me because there's no easy way to set a limit. But AWS and Azure have the same policy.

(In my case, I'm looking for somewhere I can easily deploy a set of ~5 Docker containers, they don't need to scale up, and it's a hobby project so I'd like to keep costs as low as possible.)

jeroenhd
0 replies
10h19m

The big, professional, business cloud providers aren't designed for small hobby projects with expenditure limits.

You should look into going old school and just renting a VPS with any VPS provider that's not AWS/GCP/Azure. I know the big three are super popular, as are all of these "serverless" cloud companies, but very few of them offer the most important service for a small project: shutting down before you owe them a fortune.

Depending on the guarantees you want, Oracle has a free tier with no time limit. It provides 4 ARM cores with something crazy like up to 6GB of RAM for free. You may have a few days if downtime during maintenance, but once you've allocated the resources, you'll eventually get your services back from what I can tell. Best part is, if you only use their free tier, you need to manually upgrade your account to even be able to buy anything extra. Just make sure you have backups in case Oracle pulls an Oracle.

Or you could get a VPS from a budget hoster like Contabo, which isn't free but will fit most hobby projects I know just fine. They may shut you down if they're suffering from a DDoS because of you, but you won't get a $100k bill.

iforgotpassword
0 replies
10h24m

self-Host with a provider that has clear TOS and not twenty pages of fine print. Something like the good old "if you exceed X TB in a month, you will be limited to 10mbit/s for the rest of that month". As long as you're running a hobby project or even some side project that starts making money but is still beta, this should be good enough. Once business gets serious, you will inevitably have to spend more money, and you might have to look at cloudflare et al if you're actually in a business where ddos happen.

Seriously, if you just want to run docker, maintaining a debian VPS for that is basically enabling unattended-upgrades and doing a dist-upgrade every two years. If you can't be arsed to do that then maybe you deserve the 100k bill...

twixfel
1 replies
8h47m

People don't check for every possible thing that can go wrong, otherwise we wouldn't have time to do anything. I remember when I got charged a "cancelling fee" for cancelling my Adobe subscription that accidentally went past its free trial. The situation was so ludicrous that I had never imagined that they would charge something like 6 months worth of subs me to cancel a monthly subscription. In the end I got out of it and paid nothing but I have absolutely hated Adobe ever since.

These things are scams because they prey on the fact that they're the only one shitty enough to do something so shitty and are counting on you not realising just how shitty they are.

iforgotpassword
0 replies
6h54m

Sorry, maybe this is a generational thing? I always look for a catch when something is "free". Your example as old as time, offer a free trial period but if you don't cancel X days before it ends, you automatically subscribe for another week/month/year. This has extended to getting a big discount for your first year after signup and then you pretty much have to cancel and go to a competitor, or wait until a couple days before the contract ends and some sales rep will call you and offer you another discount. It's scummy, it sucks, but it's been s reality for decades and is only getting worse.

With hosting - be it cloud or virtual or real hardware - the problem has always been that bandwidth use is completely outside your control. It was the first thing to check in the early 2000s and still is today, to an even greater extend.

So yes, sorry, as the other reply says, I might come across uncharitable or even condescending, but as tech people developing tech stuff how can one not be at least a little careful when there's a big "free candy" sign slapped onto something?

apsurd
1 replies
10h55m

too lazy is a bit uncharitable. These terms tend to be buried 8pt font disclaimer text and esoteric metering matrixes.

meanwhile in size 72 font on the marketing page it says FREE STATIC SITE HOSTING!

that's why this thread is more or less condemning scammy business practices.

[edit] check out this forum explanation from render.com billing:

Free Tier Services are suspended, no overage charges. Paid Tier Services are unaffected (Free Tier Services can be upgraded to a Paid Tier, this isn’t an overage because you are manually intervening.) Exceeding allotted Bandwidth does result in automatic overage charges. $30 for additional 100 GB blocks. Exceeding Pipeline Minutes results in deployments failing and no overage charges by default, you can configure whether you want to allow overage charges for additional blocks of Pipeline Minutes.

I still don't understand, free tiers are suspended so no overage charges, but then how can they exceed bandwidth of which we're liable? x_X

neocritter
0 replies
4h39m

Possibly making explicit that while it might go over before their suspension logic kicks in, you won't be charged for it on a free tier.

tombolino
0 replies
4h6m

first of all these cloud services are designed especially for that, for autoscaling. Second of all, if these are hobby projects we cannot look every hour the costs etc unlike people who's that their job. So stop calling people lazy and pretend like you are better than others because you are not.

aembleton
1 replies
9h4m

Use a virtual card with just a small amount of money on it to limit your liability. Won't work if you've entered a contract, but for a lot of these providers, including AWS it works.

op00to
0 replies
3h48m

This "one crazy trick" does nothing to limit your true liability.

If you go to a restaurant someone at your table orders 5,000 plates of mozzarella sticks, the fact that your credit card only covers $5 doesn't mean you are magically absolved from the rest of the bill.

For $100k, a debt collection firm would be more than happy to get a judgement against you. Credit card or no.

Nition
1 replies
10h49m

Yeah. Elsewhere in these comments there's a link to a support thread[0] where Netlify support essentially says you shouldn't ever need an option to suspend your site at a certain point because:

#1. If you think there's any chance of getting DDoSd, you should already be on a business plan instead of a starter tier.

#2. If you think there's any chance of your site going viral, you're going to want to pay the cost anyway to let all those people visit.

I agree that's ridiculous and that the lack of any option of capping costs would mean I'd never sign up for the service. But that's the official response, for what its worth.

[0] https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...

moritonal
0 replies
9h58m

So anyone whos site might be posted to HN should be on a business plan.

silent_cal
0 replies
2h18m

Totally agree, it's an unacceptable risk

bsimpson
0 replies
10h55m

I believe that's what Firebase Hosting does.

As I recall, you have to actively sign up for the paid plan (Blaze) to get pay-as-you-go billing. Otherwise, you get free quota, and if it's up, it's up.

I think it also integrates into all of Google Cloud's billing management stuff, but I've never had to bother with that.

SiVal
0 replies
11h2m

I understand that some businesses might want to take the hit from a cost surge because they get an even higher revenue surge. But a large fraction of sites aren't like that and would prefer a loss of service to a cost overrun. Service providers should always offer a "maximum out-of-pocket cost" service option. Those that don't aren't suitable vendors for most customers and customers should be warned about them.

fbdab103
12 replies
11h39m

Woof. I was considering kicking the tires on Netlify, but they are officially out of consideration now.

rKarpinski
8 replies
11h33m

Doesn't make me feel great about Vercel either (which I was looking at)

gempir
5 replies
10h43m

Why? If you use Vercel in the Free Tier you cannot automatically go to the Pro tier.

You need to manually upgrade.

Unless you need more than the Free-Tier Vercel should be fine.

auggierose
3 replies
10h18m

If you use Netlify in the free tier you cannot automatically go to pro tier either. Yet they will charge you for bandwidth > 100GB, and THERE IS NO WAY FOR YOU TO SWITCH THAT OFF, even in the free tier.

"Should be fine" is exactly what is not the case here. Better check your Vercel terms again.

aembleton
1 replies
8h41m

I've never given them any card details. Worst they can do is shutdown my website (or replace with goatse).

dbish
0 replies
1h21m

If you're in the US at least there are still bill collectors and the like which I wouldn’t want to deal with personally

gempir
0 replies
9h56m

Wow that's indeed very bad from Netlify.

From every Vercel document that I can read though is that when you exceed the limits of free tier they are just locked for 30 days unless you upgrade to Pro.

So unless I am mistaken this cannot happen on Vercel.

rKarpinski
0 replies
45m

Would be for a new business. Not a fan of introducing this sort of tail risk & whatever other risks I don't even know about for their managed service, when there are simpler ways to host I've done for years.

mdhb
0 replies
11h6m

Id recommend avoiding Vercel like the plague. Their entire business model is poorly aligned with your interests as a customer.

fxtentacle
0 replies
11h19m

That sounds like a company that is happy to ignore customer pain. Agree, default-on telemetry that cannot fully be disabled is a red flag that on their side, the people with development experience have no say.

MrBuddyCasino
0 replies
9h30m

This seems like a lot of brouhaha about nothing. They just recorded the fact that a user opted out of telemetry, they didn't stealthily send telemetry against user wishes.

They removed this call in the meantime [0].

[0] https://github.com/netlify/cli/pull/740/files

nindalf
10 replies
11h24m

Since the author has gone viral I expect some netlify exec is going to take over and write this bill off to $0. In the words of Kramer “these big companies, they just write it off!”

A moment of silence for the people who got DDoS-ed, didn’t go viral and still had to pay $5k.

factorialboy
5 replies
11h13m

You don't even know what a write-off is.

I-M-S
0 replies
10h49m

I believe the proper response should have been "Do you?"

nindalf
0 replies
8h33m

Do you?

brap
0 replies
10h42m

But they do. And they’re the ones writing it off.

_fizz_buzz_
0 replies
10h51m

Neither did Kramer :D

prmoustache
1 replies
10h21m

The very fact that you expect that making front page of hn will make them cancel that bill means that it will soon be over.

These kind of stories (alongside cancelled accounts) repeat over and over again and will soon become so not newsworthy that they will either not end up in front page, nor people will check on the eventual outcome which means these companies will get away with not moving a finger.

aurareturn
0 replies
8h6m

Nah. I think the dev community has long memory. Events like this is damaging for years. Whenever Netlify is mentioned, someone will inevitably point to this thread for a few years.

stef25
0 replies
9h31m

Thanks.

When I received a message from the bank saying my account was in the red I discovered that AWS had been billing me 1100 / month for 5 months before I even noticed. It was for something I'd set up one night while bored and then forgot about it. They drained my account :( Even had the nerve to say I had to pay for premium support only to get a "lol, pay" response.

MacNCheese23
0 replies
6h33m

didnt even take 2 hours till your predication came true, you are the messiah

mythz
6 replies
11h23m

More reasons why I avoid clouds with outrageous bandwidth fees, and prefer Hetzner's low cost fixed pricing with Cloudflare R2's 0 egress fees.

Even if the DDoS wasn't caught by Cloudflare, the total cost for 192TB bandwidth on Hetzner would be €172. Although even after 10 years on Hetzner I've never paid for any bandwidth, always well within their generous 20TB free bandwidth.

ashconnor
1 replies
10h55m

Hetzner will just null route your server if you’re DDoS’d.

account42
0 replies
6h19m

Which is vastly preferable to a thousand dollar bill (not to mention a hundred thousand dollar bill) for most websites.

SXX
1 replies
11h8m

First of all Hetzner would't let your server to be DDOSed for 192TB if it's not your normal usage. They'll likely just null route your IP if serious attack hit.

They also likely drop any charges if you escalate via support in case it was actually DDOS. E.g if you normally have 100GB / month and now you magically have 50TB / day.

What Netlify does is a scam.

ozfive
0 replies
10h56m

I feel a class action lawsuit is incoming. Potentially with FTC support...

Culonavirus
1 replies
10h20m

I really hate Clouflare and at the same time love them. Love them for their free, generous 500k req./day pages and workers, hate them for not having spending limits (or at least I can't find them). I get that they probably don't care about individuals and small businesses paying them peanuts, and corps can afford to pay extra if something blows up, but for me it just means I will only use a free account, and they get none of my money.

mythz
0 replies
10h7m

I also take advantage of their free services and only pay for R2 on Cloudflare since it's the best value managed provider I could find.

I prefer to keep my App's stateless and running in Docker containers which means storing all uploaded files and generated assets in R2 managed storage - which is also used for Litestream backups of our SQLite databases.

Blowouts are minimized when using low cost services, e.g. we had a rogue process that ended up causing 1.5M writes to R2, which only ended up costing us $4.50 in that month.

TheDataMaverick
6 replies
10h16m

I deleted all my Vercel project, moved to Cloudflare, send this support message to Vercel:

Vercel seems like the perfect solution, and I love how it supports the development community.

I am moving all my current and future hobby projects away from Vercel due to concerns raised in this discussion. https://news.ycombinator.com/item?id=39520776

Although I am a very small customer of Vercel, I have been advising larger organizations on IT and data infrastructure for the last decade or so.

I can say with very high certainty that spending limits are a critical discussion point in every large organization when making IT decisions. I've observed multiple instances where a potentially better solution was not selected due to the risk of overspending.

TheDataMaverick
4 replies
7h25m

Thanks @doque !!

Even though I've been actively looking for this, I couldn't find it.

However, this solution ONLY provides notifications; it doesn't address the underlying problem. I could be sick for five days and not check my phone.

What I need—and, in my experience, what all larger companies require—is a method to halt consumption entirely, similar to Snowflake."

doque
3 replies
7h21m

That method you're mentioning is provided by setting up a webhook that pauses a project, which can be fully automated.

TheDataMaverick
2 replies
7h14m

Thanks again @doque

For me it seems a little overkill to build an automated system that can handle webhook that can pauses a project

Hmm .. to maybe I'm overthinking it .. could you describe the architecture needed to setup a system like this?

KyleJune
1 replies
4h58m

It feels like they are intentionally making it complicated to figure out to avoid people actually setting hard limits. Vercel recently had a customer get charged 20k+ that posted on twitter. When Vercel employees mentioned this way of controlling it through a webhook, most people didn't even know about it. I feel like a hard limitbshould be easy to set in your projects settings and that it should have a default value.

TheDataMaverick
0 replies
1h24m

@KyleJune

I would consider extremely difficult to setup a reliable system that could listen to webhooks and pause projects.

My main concerns are .. how can you test it and how can you be sure that it is working in all future.

..

Are you able to find the twitter post you mentioned?

jamesdepp
5 replies
11h5m

I recently rewrote a website/small backend API for a non-profit organization. I could've gone with a serverless architecture for our forms handling API and reduced spending to nearly the free tier, but I had no good way to protect against a scenario like this. There was just not good enough documentation about how to completely cut off spending in the scenario of an attack, and I wasn't comfortable leaving the organization open to a cost attack like this.

So we're using Github Pages for static hosting and a $5 box from OVH now. Unmetered bandwidth, plenty resources for our purposes. Cheap enough, and we will never, EVER, have to worry about an attack like this. Well worth it imo.

Imo, serverless is great for internal jobs where you can control spending. For public facing things, you have to be a lot more careful.

rzmmm
0 replies
5h42m

It can but it won't result in surprise bill because the bandwidth is capped.

neurostimulant
0 replies
7h28m

GP probably refer to getting a huge surprise bill after a ddos attack, not github never got any ddos attack.

hilux
0 replies
9h56m

Worth reading - thanks for sharing.

Arguably a non-profit (unless it was selling stuff from the site, which is unlikely) would be exempt from their list of prohibitions.

gnyman
0 replies
9h16m

Yeah I'm doing the same, a combination of OVH, Scaleway and Hetzner.

Sure there is no such thing as "free unlimited" bandwidth but I much prefer unlimited with a fixed cost until they decide it's not worth it and shut me down vs unlimited risk with no ability to cap it.

The lack of cap is the worst part and it's 100% a business decision. Every provider who tracks bandwidth could add a cap but they just choose not to because it's too profitable and the risk is mostly* on the customer anyways.

*there is of course a tiny chance they the customer goes bankrupt and they get almost notning, but usually they just need to pretend to be nice and forgive all or most of it

stanac
4 replies
10h25m

I just checked pricing on hetzner (not affiliated with them). First 20 TB are included with VM price, after which it's 1.19 euros per TB. If this happened to someone hosting there it would cost them additional 50 euros. Can't believe the difference in traffic pricing. Netlify is over 1000x more expensive per TB.

hereonout2
1 replies
10h2m

People will throw up all sorts of excuses for not just renting a VM for a static site - scalability, ease of use, security.

But in the vast majority of cases you could just take a $5 VM, apt-get install nginx and be absolutely fine. A tiny bit more effort and you can make sure it's always up to date and very secure. Plus you get a VM you can use for other things when needed.

seszett
0 replies
9h43m

Just regular managed web hosting is enough for simple websites for people who don't actually want to manage a whole server. You don't have to manage anything with such a plan, just put your files where you're told and that's all. It's how simple websites have worked for decades.

Hetzner's plans start at 1,76€/month with a domain name included and unlimited traffic. OVH is slightly cheaper.

I'm probably too old (?) to understand the appeal of Netlify or other similar services, but I really don't understand why they get used.

By the way, Netlify says "100 GB bandwidth" is included with the free plan which I thought mirrored Hetzner's number of "30 TBit total bandwidth", but you have to click the details to see that it's 100 GB per month. So not bandwidth at all, but traffic.

severino
0 replies
9h24m

Problem is if you get 10x that traffic, which would already be 500€ in Hetzner, right? Of course its much cheaper and it's very unlikely that you'd get a huge bill but with them, but after all, it would be great if you could just say "Shut my machine down when my monthly spending gets to 100€"

hayst4ck
4 replies
11h17m

Defaulting to unbounded liability as the standard operating procedure for cloud infrastructure should be illegal.

notimetorelax
1 replies
10h48m

And it is, I remember cases where telcos were prevented from charging thousands for roaming expenses because they were not licensed to make large loans.

baq
0 replies
10h26m

Oh God roaming is one my biggest nightmares and it’s the worst UX to not get charged, too. ‘You want to go on vacation? Make sure to hunt down an option hidden 3 levels deep and enabled by default because we really like money and you didn’t read the fine print? Too bad, we really like money’

dkarras
0 replies
8h32m

looks like it is not just defaulting to unbounded liability. it is forcing unbounded liability as there is no way to turn it off and limit your exposure.

carlmr
0 replies
10h32m

Yep, this is a company that screams for more regulation.

ssijak
3 replies
11h28m

Put Cloudflare proxy in front of Netlify/Vercel deploys

tjosepo
2 replies
11h7m

Every Netlify project is assigned a Netlify subdomain (i.e. `example.netlify.app`) that cannot be removed or proxied.

If anyone figures out what your Netlify subdomain is, it's my understanding that they can DDoS you and there's nothing you can do about it.

spacemagic
1 replies
9h31m

That makes sense, but is the Netlify subdomain visible from your custom domain? How would they be able to figure it out, other than humans leaking it somehow?

tjosepo
0 replies
2h21m

It should not be visible, but security-by-obscurity is not something that makes me sleep well at night.

It's a design limitation of Netlify that might cost you $100,000 some day.

hiAndrewQuinn
3 replies
10h50m

This happened to me on a much smaller scale about a year back. I was never happier to have stuck to my guns, building my whole site with a single `hugo` command - it made it very easy to migrate off that platform for good.

If anyone has a solid bash one liner to stress test a website, so that I can test whether my cloud billing cap will work correctly if I accidentally try to egress 100 MB of data or something, I would seriously appreciate it. There was one on a blog post here like a year back, using apache iirc, but I forgot to bookmark it.

pushedx
2 replies
9h48m

apachebench (cli command ab), is an Apache licensed stress testing utility, not much relation to the web server, which can be easily used to stress test a webserver.

the binary should be easy to install with your package manager, you may have it installed already

example:

ab http://something.com

hiAndrewQuinn
0 replies
7h1m

Yes! That's the one, thank you.

gregorvand
3 replies
10h58m

There's a reason price plans are 'pay as you go' - so that they don't take on any risk for things like this. Moral of the story: never go for pay as you go by usage plans if this scenario is not mitigated through a third party. By no means am I trying to suggest a get out for Netlify here - more a rule of thumb for anyone worried the same could happen to them.

SXX
1 replies
10h42m

When you sign up for their "Free tier" there are no big red warning that they can charge you $100,000 out of the blue.

immibis
0 replies
1h22m

But they do tell you the pricing structure, right?

blibble
0 replies
7h17m

they are taking a risk because they haven't credit checked the user

most people wouldn't be able to pay a $100k bill

SoftTalker
3 replies
2h35m

I once got a $65,000 water bill from the city for one month. I laughed and called them and asked them to re-read the meter and correct it, and expected a quick resolution. But no, they insisted it was correct for some time and that I needed to pay it. They said I probably had a leaky faucet or running toilet.

There was no awareness on the part of the customer service people how ridiculous that was. It would be physically impossible for my service pipe to deliver that volume of water even if it had been running full open for the entire month. I kept escalating until I reached someone who agreed, and they sent someone out to re-read the meter. And my bill was reduced to about $35.00, the normal amount.

Front line customer support isn't always very in tune with what is sensible for a given customer's account.

0cf8612b2e1e
1 replies
2h25m

Water is a regulated utility. Anyone in a similar situation can contact the government authority who will gleefully tell the company to go to hell and possibly implement fines for inappropriate billing.

mr337
0 replies
1h51m

I wish that was so straight forward. You can google this incident where an empty lot got a 35K water bill and the water company said it was an error, then backtracked on that and still saying 35K is due.

Towards the end of 2023, the DWM seemingly corrected the issue. Revive received an email stating: “The prior balance on the account reflected water leakage that was the result of Department of Watershed actions. Once the leak was addressed and the account properly adjusted, the corrected balance for the property is $219.24.” However, DWM soon backtracked and claimed that the $219.24 quote was made in error and that the nearly $30,000 balance still applied.

https://lawblog.legalmatch.com/2024/02/26/empty-atlanta-lot-...

boringg
0 replies
2h33m

To be fair to front line support -- a lot of times it's just a warm body reading a script and it's paid accordingly.

Not always - but a lot of the times, especially for lower quality companies.

quickthrower2
2 replies
11h34m

Pulling my netlify hosted stuff today. F that. Sorry to hnbadges users!

therouwboat
1 replies
8h42m

Me too, it took like 30 mins to create cloudflare account and move my small static sites.

quickthrower2
0 replies
6h19m

Lol I just deleted the account (surprisingly easy). Since I didn’t have a custom domain, just let it die for now.

nanna
2 replies
10h15m

A few years ago I taught an introduction to website developmment module at a university where students built jekyll sites. I got them to to host on netlify. Now that I no longer teach at that uni and their email accounts will have expired I'm wondering if there's any way I can contact them to tell them to take their sites off netlify... Disaster if they get hit by charges like this.

kiprou
1 replies
9h8m

At the rate it's going this story will probably reach them without your help.

nanna
0 replies
5h16m

Not necessarily, the majority of them probably wouldn't have ended up in anything tech related. But they probably did end up back in China, so hopefully it would be hard to demand payment at least?

kaeruct
2 replies
10h18m

I have a bunch of pet projects on netlify free tier and I could never afford to pay this amount of money. What are some good alternatives that don't have this issue? I've already noticed cloudflare pages mentioned in these comments.

cowoder
1 replies
8h54m

My pipeline and hosting solutions are:

Static: Github Actions to build and deploy to BunnyCDN

Non-static: selfhosted Dokku on Hetzner

Neither is free, if you're looking for free, Github Pages or Cloudflare for static sites. Free non-static, I'm not sure there are solutions that don't have the same problem Netlify has.

Gys
0 replies
8h45m

Dokku is free (the Pro version is paid but that is really more an enterprise kind of thing).

viraptor
0 replies
10h15m

Have you tested it end to end? When was the last time? I'm slightly worried about solutions like that. I like when it's all host's responsibility. (Having your own system on top is nice though)

redman25
0 replies
9h33m

You could probably set up rate limiting with iptables/nftables as well if it’s a vps.

hardwaresofton
2 replies
10h18m

Why is there no self-hostable Netlify yet?

Is it just that no one would pay for it? I'm well aware of how terrible a customer developers make, but has to be nearly a non-issue with Hetzner in the USA now, with how much free traffic they give you (or any other provider, DO, etc). There's even Cloudflare R2 nowadays.

Your blog probably doesn't actually need sub 100ms serve times.

hardwaresofton
0 replies
9h18m

Yeah there are a bunch of selfhostable things:

Caprover (https://caprover.com/)

Dokku (https://github.com/dokku/dokku)

But people still choose Netlify and Vercel for ease of use I think.

Maybe we need something that's just Netlify. The closest I've seen to the "right" UX is Ness:

https://ness.sh

Though of course it's a tui so some people will get turned off of that (especially people who are willing to spend).

francisofascii
2 replies
2h4m

I guess I never want my personal blog/site to be successful. Can't trust that I will wake up to a huge bill from EC2 or my CDN provider. Feels like health insurance in that even when you have insurance, you are never 100% confident you will not get some huge hospital bill. The pessimistic part of me says this is all deliberate to prevent the "little guy" from competing these days.

dbish
1 replies
1h19m

If you run on a single EC2 instance and you aren’t running an auto scaling cluster or anything of the sort, it would be pretty hard to get a huge bill. I much prefer that and the chance that it goes down then autoscaling or severless. Most serverless solutions have also gotten so config heavy or complex to make changes that most projects feel much better to me on an instance I can ssh into and poke around without having to call up support.

francisofascii
0 replies
14m

Thanks, I use a "small" EC2 instance for my personal stuff (about $35 a month), with the Cloudflare free version, but I honestly don't know and what would happen if something went viral or a DOS. Would the bill be double? Triple? Would the site simply crash. What does Cloudflare do? I honestly have no idea.

RamblingCTO
2 replies
9h43m

Good reminder, just moved my blog to cloudflare pages!

spacemagic
1 replies
9h25m

I'm assuming they have DDoS protection by default, being a Cloudflare product? Or is the reasoning simply that they allow you to set a bandwidth/usage cap?

RamblingCTO
0 replies
4h18m

Yes, that was my understanding. Also unlimited bandwidth even on the free tier. So you get protection from DDoS and asshole business practices ;) Although there might be other caveats

ConorSheehan1
2 replies
10h37m

Does this issue only occur if you have billing info on file?

I'm using the free tier and have no billing info set. According to this https://github.com/netlify/ask-netlify/issues/6#issuecomment... > if you have an event that puts you over the free-tier limits, Netlify will ask you to update your billing information and add a CC

Although worryingly > We just had this happen and our site didn't stop working.

Is there any way to ensure if you hit the limit sites just stop working and you don't get billed?

turtles3
0 replies
9h52m

I'm also interested to know this. I have a couple of static sites running on the free tier for friends/family and now I'm planning on moving them all to a VPS as soon as I can.

It is beyond ridiculous that serverless providers don't offer a way to cap spending. The idea that it might cause your site to go offline is a complete non-argument. That what I _want_ to happen. I want to be able to say sure, I'm happy to sustain 10x traffic for a few hours, and maybe 3x sustained over days, but after that take it offline. I don't want infinitely scaling infra precisely because of the infinitely scaling costs.

breakingcups
0 replies
8h51m

No, and this is by design. If you go over the limits (can also happen if a build machine times out, ask me how I know), you will be billed without any recourse. If you have no billing information and refuse to set it, at the very least they'll permanently ban you from their platform.

Which, if it remains the only consequence, seems like a blessing now.

zsellera
1 replies
10h36m

After doing some math it doesn't feel like a ddos:

- $104k at ($55 / 100 GB) = 189 TB of traffic

- It means the popular ~3.5 MB media file was downloaded ~54M times

- Which sound like a lot, but if you get popular in a country with 1.4B people, it's not (~3% accessed).

What if it happens at AWS Cloudfront? At $.1/GB it sums up to ~$18k. In the light of these, Netlify's offer of ~$5k seems generous.

hayst4ck
0 replies
10h15m

The author claimed Netlify's own support agent said it was an L7 ddos and offered a 95% concession because it was a ddos.

There is no reason to question whether it was a ddos or not because, allegedly, both parties in this dispute already agree it was a ddos.

renonce
1 replies
9h2m

Here is my shiny new super business plan for a startup that will profit thousands from a supposedly non-paying customer:

1. Offer “free static website” with lots of templates and guides to help you build one

2. The first 100GB is free and beyond that it’s $0.01/MB. But no worries! Very few customers actually use up that free bandwidth and in case you need more you can purchase packages for $100/TB. Also we offer a free service that will help you get your site more visible by advertising it, it’s included by default.

3. After a month or so, randomly help a customer bump the website and make it popular by putting it in some list that is frequently crawled. Secretly hire someone else to crawl these websites and make lots of download requests

4. Once the customer suddenly gets 10TB of traffic, bill them for 9900GB which is $99000

5. As long as 1 out of 100 customers pay, you are profiting $990 per customer! For the rest of customers, offer a 5% discount so they only have to pay $1980. Threat taking them to collections if they refuse.

Become the next millionare by just selling free static websites to 1000 customers! Anyone join us?

dbish
0 replies
1h24m

Are you taking angel investors? Throw in that it comes with an AI chat assistant or a special vectordb so you can raise money and I’m in

pr337h4m
1 replies
10h51m

Any cloud service that offers prepaid billing on a credits model would get a lot of customers.

(Prepaid is also superior from a cash flow perspective too.)

kosolam
1 replies
11h29m

Any lawyer here can suggest if a class action suit is appropriate?

gnicholas
0 replies
11h6m

The test for federal class action lawsuit includes 4 prongs, all of which must be satisfied: numerosity, commonality, typicality, and adequacy. [1]

I would think (as a former lawyer with only passing familiarity with class actions) that 'typicality' would be the key question.

to determine typicality the courts consider to what extent plaintiffs’ claims are markedly different or are generally the same (for instance arising from the same event or pattern) as those of other class members with respect to the relevant legal theory and factual circumstances of the case. [1]

The defendant would probably claim that each plaintiff's issues are quite unique. However, this prong is apparently not based on the typicality of the specific facts giving rise to the lawsuit, but rather the typicality of the nature of the claim or defense. And it's apparently hard to 'win' (defeat a class action) via this prong. [2]

1: https://www.bonalaw.com/insights/legal-resources/what-are-th...

2: https://california-business-lawyer-corporate-lawyer.com/clas...

khaomungai
1 replies
9h18m

I am running my static sites on a VPS for only 1€ per month. It includes unlimited traffic, DDOS protection, and IDS/IPS at the service provider level.

There are no surprises! :)

undopamine
0 replies
6h18m

Provider name and specifications please?

kendriklampar
1 replies
11h41m

You just posted it on Reddit and I already see few comments of people say they will never use it. Stuff like this can cause PR damage worth millions to a company. I'm pretty sure that it will go viral and they won't charge you.

WyvernDrexx
0 replies
10h52m

Exactly. I think this also might be a reason why they didn't implement ddos protection on this level. So, they can grab as much as they can.

jiripospisil
1 replies
6h47m

Posted 4 hours ago with 958 points and it's already sitting at 68th place? Was the story too inconvenient for Y Combinator given that Netlify has bought 2? of YC backed companies?

brisvegas
0 replies
6h3m

very strange indeed, never seen anything like this on HN before...

gmihelac
1 replies
8h5m

This is going to be a dumb question. I am new to coding ( 3 months in ). I am building a simple static personal website with GitHub pages. I am worried since I had to input billing data to GitHub for my global campus application since I am a student.

Is this something I need to worry about? Does cloudflare provide a service that is cheap that can prevent something like this for my GitHub pages site?

jskherman
0 replies
7h44m

Getting DDoSed is rare but highly destructive when it does happen if you happen to use something like Netlify. GitHub Pages is under fair use and only for personal projects (no commercial uses). For commercial use, something like Cloudflare Pages would be better. Cloudflare can also help with their whole suite of tools with mitigating attacks and they will call you if you do exceed the free tier according to this comment: https://news.ycombinator.com/item?id=39520894.

davidguetta
1 replies
4h22m

Is the entire idea of paying for bandwith not absurd ? You can't really control it

I have some websites with milions + autogenerated webpages and it's flooded by bot activity that I don't particularly care about.

I've blocked some through cloudflare but some look exactly like he describes: old machines with old versions of some OS and browser scattered around the globe that seem to be scanning my entire website, maybe for AI training purpose ?

The point is, I can't block them at all.

see this thread here : https://www.reddit.com/r/webdev/comments/1azv0fs/is_this_tra...

immibis
0 replies
1h23m

There are providers that change you for connection speed and have no limits besides that. They're more expensive.

c4obi
1 replies
9h23m

good time to point out tiiny.host -> https://tiiny.host/

ixmerof
0 replies
8h54m

whoah, thats expensive. reminds me free services from early 2000's (lycos, 50web etc) with these file limits and banners

binsquare
1 replies
11h39m

That discount to 5% is incredibly odd and hand wavy, I would escalate to their manager.

They should definitely be able to accomodate and account for what should be a very common issue (does).

Reelix
0 replies
8h30m

They can discount to 5% since their fees are 10,000% higher than what it's costing them...

ytx
0 replies
28m

I'm curious what would've happened had the bill been for 10k or 1k? Would it still have been reduced, would OP have paid instead of posting on reddit/hn, would it have gotten as much attention?

yorwba
0 replies
11h32m

What was the website that got DDOSed? https://jyutping.org/en ? (Since it's behind Cloudflare now, I guess linking it here is fine.)

At first I thought it might've been https://hanhngiox.net/ , but that one appears to have simply expired.

xlbuttplug2
0 replies
2h13m

Cloudflare got a bunch of new "customers" judging by this thread.

Competitors could rinse and repeat this strategy to put netlify in the dirt for good.

xamde
0 replies
7h0m

I am a Netlify Pro customer. I was not aware of this unlimited spending concept. This is simply unacceptable from a risk-perspective.

uses
0 replies
1h15m

I can't get past why Netlify's infrastructure continued serving a 3MB file to a botnet 55,000,000 times? I would've assumed their system was smarter than the ubuntu/apache install in my basement.

tsp
0 replies
8h22m

I have been using Netlify for years, for my own projects, but also recommended it to all my freelance clients to host the projects that I was building for them. Going forward I will move all my static pages to other hosting providers.

The Netlify team must think: we waive the fees, because in this instance we noticed the negative press and want to avoid this from blowing up. When this happens to other users, we don’t care, as long as it does not go viral.

Such a pity, Netlify has great UX and I was so happy hosting static pages on their service. But without spending limits, this is not an option for me any more. I could not sleep well when there is a possibility of a $10.000 invoice reaching my inbox.

tommica
0 replies
9h33m

Yeah, not gonna use their services, don't want to deal with an issue like this.

throwaway598
0 replies
9h40m

I must add that at Netlify are consistent across disparate systems.

They consistently missed someone that doesn't get more than 10GB traffic a month now maxing out a 5Gbps line (60TB/day stated in TFA) for several days in a row.

And consistently missed someone with a tiny bill's now racking up 10s of thousands of dollars per day. Even if they do run a mainframe and batch process at the end of each day, that still went a few days. If extending lines of credit of tens of thousands of dollars is legal, that's very generous.

What further consistency is available at Netlify?

tacone
0 replies
10h50m

The pay as you go model is completely wrong.

Companies should be legally required to allow their customers to set a ceiling to their monthly spending.

sva_
0 replies
8h56m

I like companies like this, that give you such a straight-forward reason never to do business with them.

smerik
0 replies
10h45m

Thats criminal, hope you get things sorted. Amazon would wave it.

silasdavis
0 replies
10h51m

I was searching for 'cloudflare spending limits' based on comments here, at first glance they (like Vercel) seem to have a notify don't terminate policy.

What I didn't appreciate is static assets are completely free: https://developers.cloudflare.com/pages/functions/pricing/#s...

So therefore I assume static cloudflare pages are free.

Incidentally Vercel does seem to have an (annoyingly indirect) way of halting usage based on spending: https://vercel.com/docs/accounts/spend-management

Has anyone implemented this, are there any problems with it?

shrubble
0 replies
10h10m

If we take 60TB / 24 hours we get roughly 5.5Gbits/second of traffic.

You can get a 10gbps connection, per month, for under $4K a month to a datacenter. For full usage for one month, not one day. If buying at the scale of a much larger bandwidth user, the price is much less.

shantnutiwari
0 replies
1h9m

I love it that the reddit post asks him straight away to make a post on HN. Its like everyone knows HN is the default customer support for these 10x web 2.0 hyperscale companies...

sevg
0 replies
8h40m

I'm wondering whether this story has been manually downranked.

Posted only 3 hours ago and 800+ votes but it's suddenly dropped from top 3 (on page 1) to 38th (on page 2).

Is it worth noting that Netlify bought two Y Combinator startups or is that a crazy conspiracy theory?

On May 19, 2021, Netlify announced the acquisition of FeaturePeek, a Y Combinator and Matrix Partners backed startup that enables developer teams to preview frontend content.

On November 17, 2021, Netlify acquired Y Combinator and SignalFire-backed OneGraph to allow for the composition of apps with APIs and services using GraphQL.
schappim
0 replies
11h30m

This is such a shame, I found the user experience pretty good on Netlify and now I can’t risk staying with them.

rybosworld
0 replies
1h12m

This is true for all businesses but maybe more so for tech:

Don't have a business model that charges customers for your mistakes.

This customers bandwidth usage jumped from free tier to $100k in very short time. To be honest, this shouldn't even be possible. Any "free" tier that allows for a surprise $100k bill is not a free tier.

This bandwidth usage is the result of a mistake on Netlify's part. That much seems clear.

To go and suggest that the customer is responsible for any portion of the bill is where things really went sour imo. Don't do this. Ever. Unless you want your company to go viral for all the wrong reasons.

If you want another good example of how badly this can backfire, look at what happened when Unity announced their new pricing scheme. Unity's new pricing scheme also allowed for unbounded bills. At first they didn't even deny this. Later they said it was a customer misunderstanding. I.e., they blamed the customer for their mistake.

Thankfully, the CEO of Unity was fired.

The lessons are very straightforward:

1) Don't implement predatory pricing schemes (this can even be done unintentionally, but the intent doesn't matter).

2) If you do implement predatory pricing, the worst thing you can do is put on your surprised pikachu face when the customer asks why their bill is bigger than their annual income.

rtpg
0 replies
11h28m

Really uncomfortable with how many services like this are “we scale to your needs” and end up here. I guess capacity planning from oversubscribing bandwidth is its own can of worms but surely that is a bit… less surprise generating.

rsolva
0 replies
10h32m

Hosting personal sites on cheap hardware at home makes even more sense to me after reading this horror story.

Why do not more technically inclined people self-host? Is it force of habit from how things are done at work?

I would much rather run the "risk" of some occasional dowtime, than keeping the lights on at all costs under a DDoS-attack.

rijavecb
0 replies
7h0m

How come a post with so many votes and comments, submitted just 4 hours ago, is already on the 3rd page? Would have missed an interesting discussion haven't I seen a link to it on Reddit

raffraffraff
0 replies
9h36m

Wow.

I can't read it because I block reddit, but I assume it's a DDOS? With a bill that large, it would actually make business sense for them to team up with DDOSers.

pks016
0 replies
0m

Woah! I had two hobby sites from 2018. I thought I shut them down after the policy change, but I may have forgotten. Just deleted my account.

peter-m80
0 replies
10h31m

I ended up self hosting my own stuff behind cloudflare

notfed
0 replies
8h45m

There should be a "max $$ per day" set by the customer.

For example if my personal blog exceeds $1/day, I am ok if it goes down. Having no limit is insane.

nojvek
0 replies
6h54m

This is perhaps a good lesson for considering cloudflare.

I’m surprised Netlify has no ddos protection.

mrkramer
0 replies
8h6m

I will ask the same question as for BigQuery:

I thought you could cap(limit) your spending e.g. "if I reach $1000 in costs, abort/pause operations"[0].

[0] https://news.ycombinator.com/item?id=39472553

mmaunder
0 replies
10h51m

Netlify charge $55 per 100GB for overages on their starter plan. Keep in mind that they’re being billed by their provider at 95th percentile billing. That means they have ports running at say 100 gigabits per second and have a commit of say 50 gb/s and they’re billed a flat rate for that as long as they don’t go over. Because they’re billed 95th percentile they can spike their connection traffic massively for 5% of the time and not get billed more. So Netlify themselves have a safety net and don’t need real-time monitoring to immediately cut their usage. And of course any spikes from a single user are massively diluted among their entire user base. So, yeah, they’re gouging big time.

What they should have is monitoring per user and a default that 503s the site with no overages that has to be proactively disabled by the user. Instead they’re just letting it ride and trying their luck by negotiating down the overage charge to what they think the user can stomach.

mirekrusin
0 replies
11h22m

It's called web scale.

megous
0 replies
9h31m

Is this website a court or ombudsman or what? If a business suddenly charges you 10000x more than usual, just take them to court.

lettergram
0 replies
11h14m

If it’s a static website; use GitHub to host and put behind cloudflare.

If you need a nice front end, spin up a Wordpress instance with a provider like digital ocean or vultr or any other number of places. It’s like $5-10/ month and has terabyte bandwidth without issue typically.

Then put the site behind cloudflare or at least configure a plugin for ddos protection.

kyledrake
0 replies
2h28m

"190TB in 4 days"

That's approximately one month of transit through a gigabit connection, which indeed could be pretty expensive. Even at IP transit prices it would be something like $300/mo for a 1GB connection via HE but you would need something bigger to handle this traffic. I pay about $2k at bulk for that amount of bandwidth. Any way you cut it, you're getting some sort of bill (or kicked off) for that without some sort of ddos forgiveness.

That said, $104k feels excessive to me for static hosting ($0.57/gb, did I do that right?).

kloogans
0 replies
4h37m

Been using Netlify for 7 years now and I'll be moving all of my projects to cloudflare this week. That's a hell of a lot of risk to host a few hobby sites that virtually nobody visits. Cant imagine running a bootstrapped business and having this happen. Seems extremely predatory.

josebama
0 replies
9h56m

If they offer a platform as a service and they fail to protect that platform from ddos attacks, they should incur the cost and not the customer

jonplackett
0 replies
9h1m

We need a separate ‘shame’ tab at the top for crap like this. There must be so many more of these asshole bills being sent out that aren’t lucky enough to get voted up.

jm12345
0 replies
4h26m

Just deleted my netlify site and accounts after reading this. I had no idea what I agreed to and the potential consequences. Dodged a bullet!

indiantinker
0 replies
11h14m

They might not get Ddos protection because Shirky principal. Institutes further problems they are meant to solve

immibis
0 replies
9h35m

The lesson to be learned is to pay attention to pricing before signing up, and don't buy really expensive services. Assuming a gigabit connection, you have the potential to pay $13000 per day.

haunter
0 replies
10h28m

I hate that a lot of times the only solution is to post on HN/reddit/Twitter etc publicly. Just imagine all the other people who don’t use social media and ended up paying…

happytiger
0 replies
8h56m

I’m stunned there is no way to limit spend and have warning and a cap on a service like that. You should be able to specify billing thresholds and have sensible defaults when the downside is basically unlimited. Basic customer UX.

fullstackchris
0 replies
8h26m

I'm a huge Netlify fan, but stories like these are scary. Just spent the last hour building a custom serverless function that sends email / SMS at various bandwidth usages - potentially could be improved with an auto shutdown of the offending site beyond 75 GB:

https://gist.github.com/princefishthrower/4517ff44a9f4c5b2d2...

For now it's still a workaround... looks like I'll be migrating to Hetzner soon...

fevangelou
0 replies
9h42m

I hope you don't live in the US buddy... If you don't, you're probably fine. If you do and Netlify decides to go after you, it's probably gonna be tough and costly.

Which is crucial Netlify is called out for this. Hiding behind the "fine print" is pathetic, not a way to do business.

fabian2k
0 replies
10h55m

A successful distributed denial of money is likely much more devastating to anyone except really large companies than a successful DDoS. For a personal accounts it is entirely devastating with no upside, but even for a startup it would probably be better for your site to just go down instead of having huge bills generated.

eknkc
0 replies
11h26m

This and that similar story recently about Vercel makes me feel iffy about these services at this point. I guess I’ll either use Cloudflare pages or simple dedicated machines from Hetzner going forward.

dz0ny
0 replies
9h33m

Typically, it is not even a Distributed Denial of Service (DDOS) attack; instead, it is traffic from (SEO) bots that crawl through every possible combination of the page, such as query parameters for web shop filters. I hate those down to the guts, because you have to design a rate limiter that will limit them and let other traffic through.

darkotic
0 replies
1h25m

It feels even worse than that if you consider with this being widely known, competitors could make a point to attack Netlify customers knowing it may bankrupt them.

danpalmer
0 replies
2h35m

I was regularly at 80-90% of my free-tier Netlify bandwidth of 100GB.

The site was not live.

d1sxeyes
0 replies
7h41m

I had a couple of orphaned static sites on Netlify. Just deleted them, and won't put anything else on there now.

Now I get that this is their 'product'—selling hosting to high traffic customers, and I don't particularly begrudge them charging whatever margins they think are suitable for their product and let people make their decision.

But no mechanism to cap maximum spend is completely ridiculous. Even if you require a minimum cap of say $5/$10 dollars, as long as it's clear and transparent, I think that would be reasonable.

Anyway, this has scared me into never trusting Netlify with anything any more.

canucker2016
0 replies
8h24m

Netlify's 2017 blog post says you don't need Cloudflare.

from https://www.netlify.com/blog/2017/03/28/why-you-dont-need-cl...

"...Top to bottom, our infrastructure redundancies make sure we keep traffic flowing, so there's no need to add more redundancy with Cloudflare. ...

You don't need Cloudflare when you use Netlify

As you can see, we already offer what Cloudflare does, and more. If your site is not on Netlify, perhaps consider us for your one stop solution for hosting, SSL, DDoS protection, DNS load balancing, and continuous deployments. ..."

brettgo1
0 replies
11h35m

What the hell. Netlify has implemented all these customer-hostile billing changes. Literally the worst. I migrated my company's site off of Netlify when they tried charging for per user on one of my repos (a documentation site). Glad I left them and never looked back.

bradley13
0 replies
9h34m

Just to toss this out there: There are too many services that don't really let you cap your spend. AWS, for example: You can set alarms, but as far as I know, you cannot sent a spend limit.

I had a couple of small businesses hosted there, but this always worried me, so I moved them to a local provider, who provides what they need for a flat fee.

bilalq
0 replies
11h21m

A lot of today's managed service solutions turn scaling problems into billing problems. While that's a useful tradeoff for many business use-cases, there should still be a way to set limits so you don't wind up with insane cost overruns.

benterix
0 replies
55m

Just use Hetzner and never worry about getting into debt and having to beg some company's customer service to be so kind and reduce your imaginary "debt" with them.

azdanov
0 replies
6h27m

Scary. I deleted my netlify account. Just in case.

auggierose
0 replies
10h27m

Wow, insane. I would not have guessed that they simply charge you for additional bandwidth instead of just shutting your site down for the rest of the month.

This is not something you would assume from the pricing page. On the pricing page, they show you:

    Add-ons:
        Additional bandwidth
        Additional build minutes
        Additional teams
In my understanding, an Add-on is something I need to enable, not something THEY enable for me if they see fit. When I am logged into Netlify, they tell me:

    No currently enabled site add-ons with fees
Which apparently means nothing, as they will just automatically enable the add-on for me!!!

I have seven static sites on there, deleted 3 of them right now, will migrate the other 4 soon. Unbelievable bullshit.

arthur_sav
0 replies
9h54m

Oh wow, I'm already hosting a small website with them and was thinking of hosting more. Definitely out of the question now, will move to Cloudflare.

amoh14
0 replies
10h30m

Netlify doesn't offer alerts or budget-triggers? This is my nightmare every time I put up my card details with a cloud service provider.

The funny thing is - such platforms "scale" easily with the underlying assumption that scale equals profits, enough to justify increased cost. Needless to say, inaccurate assumption.

ada1981
0 replies
10h10m

It’s probably worth wrapping the entity that contracts with these type of services in a separate anonymous LLC structure so a DDoS attack doesn’t bankrupt you.

ZhadruOmjar
0 replies
9h41m

There is a lot to be said for just hosting your own VPS and learning how it works especially for smaller sites. The effort to host is small and you're not going to get smashed with a bill like this.

Zetobal
0 replies
10h31m

I mean I get it but you get notifications and emails every time you trigger extra traffic. Don't get me wrong it should be automatic or there should be a safety net but this is not only on netlify.

WyvernDrexx
0 replies
10h50m

Reading this article made me realise not to use these services for good. $5000 is a lot man.. It's not that we make money out of thin air.

Seb-C
0 replies
11h7m

That it why I always refuse to use hosting services with on-demand pricing, especially for personal projects: in the worst case scenario I'd rather see it down for a few days than be indebted for whatever amount.

Saris
0 replies
30m

I feel like unless a customer very explicitly opts in saying "I NEVER want my site to go down" then the host should just shut the site down if traffic spikes to high levels.

Since Netlify charges 55$/100GB for the exceeding bandwidth

Absolutely absurd fees, there is no basis in reality for that. Sounds like a very scummy company.

SXX
0 replies
11h13m

We really need to have list of these shady SaaS that will try to charge you money when you using their "Free" tier and never given them credit card.

This should not be tolerated. Full stop.

PaulKeeble
0 replies
3h49m

That is very concerning, I guess I am moving my site from Netlify urgently.

PaoloBarbolini
0 replies
11h23m

This is the reason I've never used all of these user-facing serverless services. The price depends on the usage, but if anything goes wrong they are the ones to decide what you pay. It's not comfortable thinking that you could screw up, or get DDoS'd, and the remedy is hoping they wave the bill.

PDSCodes
0 replies
9h53m

I am a business user with netlify. I have unlimited functions calls on that plan (fair use of course!) and use their JWT protection that redirects to a login site at the cdn level, so you can rate limit. Not a solution for public static sites though! You are metered on the starter and pro plans after the starting limits.

They seem to have dropped that plan now.

I was starting to move back to traditional hosting as these platforms are convenient, but you do lose control and get hammered for their addon services and simple things like static ips are beyond them, even if you offer to pay.

Also, if their cdn is naughty listed, corporate networks may block your site as you are sharing pro and business plans with free sites that maybe serving malware etc.

Hearing this story has pushed me to move.

I hope they sort that for you, they really should have the ability to protect a site and let you choose what to do if you are exceeding your limits.

Ninjinka
0 replies
7h35m

Terrifying. Just deleted all of my Netlify sites.

NicoJuicy
0 replies
11h21m

Cloudflare has free DDOS protection.

Additionally, they own ( or co-own) their DC, while Netlify and Vercel doesn't. So they can fix any billing issues at their end.

INTPenis
0 replies
11h39m

Doesn't Netlify have budget alerts?

I know during heavy DDoS attacks they might be too late, but also cache?

FlyingSnake
0 replies
10h30m

And that’s how you lose loyal customers who have evangelised your products for years. I’ve helped local mom-and-pop stores, bootstrapped startups etc setup on Netlify and now I fear that Netlify might send them a hefty bill.

I’ll remove all my websites from Netlify and moving to Cloudflare. Auf Wiedersehen.

6510
0 replies
9h44m

If architected properly having many people request the same file should make things work better and make it cheaper.

The internet and www are obviously marvelous technologies with many people to praise, webtorrent is nice but in a way a hack replicating things we already had.

We can do better