return to table of content

Cloud Egress Costs

PreInternet01
119 replies
21h45m

The cost of egress traffic is a very good reason for many organizations to not fully migrate to a cloud provider anytime soon. And since, unlike with storage costs, there doesn't seem to be an actual reason (other than: it makes migrating to competitors cost-prohibitive in a subset of cases), that seems kind of... weird?

Small example: an actual company I do some work for is in the business of delivering creative assets to distributors. This results in an egress of around 180TB per month, which is, on average just, around 500Mb/s.

So, this company currently operates 2 racks in commercial data centers, linked via 10Gb/s Ethernet-over-DWDM, with 2x512Mb/s and 1x1Gb/s Internet uplinks per DC. Each rack has 2 generic-OEM servers with ~64 AMD Zen cores, 1/2TB RAM, ~8TB NVMe and ~100TB SAS RAID6 storage per node.

Just the cost-savings over egress on AWS is enough to justify that setup, including the cost of an engineer to keep it all up and running (even though the effort required for that turns out to be minimal).

So, are cloud providers ignoring a significant market here, or is the markup on their current customers lucrative enough?

amluto
59 replies
20h42m

other than: it makes migrating to competitors cost-prohibitive in a subset of cases

My theory: it forces third party services into the same cloud.

Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples:

https://www.snowflake.com/en/data-cloud/pricing-options/

And look at this remarkable choice: you get to pick AWS, Azure, or GCP! Snowflake is paying a lot of money to host on those clouds, and they’re passing those costs on to customers.

Snowflake is big. They have lots of engineers. They are obviously cloud-agnostic: they already support three clouds. It would surely be much cheaper to operate a physical facility, and they could plausibly offer better performance (because NVMe is amazing), and they could split the cost savings with customers. But they don’t, and my theory is that egress from customers to Snowflake would negate any cost savings, and the variable nature of the costs would scare away customers.

So my theory is that the ways that customers avoid egress fees makes the major clouds a lot of money. IMO regulators should take a very careful look at this, but it’s an excellent business decision on the parts of the clouds.

nyc_data_geek
25 replies
19h12m

It's pretty simple. Excessive egress costs = vendor lock in, and yes, forcing third party services into the same cloud (the walled garden), and limiting customer choice.

Just another reason so many orgs are getting heartburn from going too deep too fast into the cloud.

echelon
24 replies
18h33m

Amazing:

  Cloudflare  --  Free for most services
  OVH Cloud  --  Free and unlimited
  Scaleway  --  Free for most services
Great:

  Hetzner  20-60 TB / mo per instance  $1.08
Not bad:

  Linode  1-20 TB / mo per instance  $5.00
  Oracle Cloud  10 TB / mo  $8.50
A bit much:

  Backblaze  3x the amount of data stored  $10.00
  Bunny CDN  --  $10.00
  DigitalOcean  100 GB - 10 TB / mo per instance  $10.00
  UpCloud  500 GB - 24 TB / mo per instance  $10.77
  Vultr  2 TB / mo for most services  $10.00
Uh...

  Fly.io  100 GB / mo  $20.00

Are you actually serious?

  Microsoft Azure  100 GB / mo  $78.30
  Amazon Web Services  100 GB / mo  $92.16
  Railway  --  $100.00
  Zeabur  10-100 GB, depends on plan  $100.00
  Google Cloud  Depends on service  $111.60

Screw you guys:

  Render  100 GB - 1 TB, depends on plan  $300.00
  Vercel  100 GB - 1 TB, depends on plan  $400.00
  Netlify  100 GB - 1 TB, depends on plan  $550.00

(We use Netlify and have well over 1TB of monthly traffic. They're insanely expensive for what they are. As soon as we have roadmap time to revisit it, we'll move away.)

I'm starting to think of cloud as less of an asset and as more of a liability. We can leverage them for temporary scale, but in no way will we tie ourselves to a particular vendor.

dijit
8 replies
16h49m

One thing we should all have a subtle understanding of: if you are a company selling infrastructure tooling as a service (Vercel, for example). You should probably try to avoid hyperscalers.

The adage is that you should outsource the things that are not your core competence. However if your product is infrastructure for developers then outsourcing that is going to be painful.

You are forced to pass down costs from your upstream supplier, and headroom is necessary (somewhat) because your provider can dictate terms essentially on a whim.

I feel like this is such an obvious statement, but we do seem to be in the gold-rush of avoiding responsibility so I'm not certain.

lumost
7 replies
11h38m

Practically, this means that if you are an infrastructure tolling company - then you must reinvent all pieces of the cloud supply chain before building your product.

That’s a very expensive proposition. While the paper cost of a few racks in a colo is low - the people time to get where you want to be is high. If you mess up in this evolution - there is a risk that others outcompete you on something that isn’t your core product.

vidarh
6 replies
6h14m

The people time of setup in a colo or managed provider is trivial if you hire someone who actually know what they're doing, and usually lower than a cloud setup. It's stupidly easy to mess up a cloud setup in ways that are hard with a rack.

I find this is mostly an issue with a lot of shops simply having nobody with actual ops experience.

lumost
5 replies
5h2m

It’s not so much the setup time for the hardware/networking. This takes some time, but can reasonably be done in a few months even for complex setups.

The problem comes when you then wish to have things like CI/CD, distributed databases etc.

vidarh
4 replies
4h15m

The vast majority of companies never reach a scale where wiring up the infra needs to take more than days.

And CI/CD and distributed databases are equally not an issue of you actually have someone experienced do it.

Done this many times.

candiodari
3 replies
2h11m

1) It's not like cloud databases are problem free. You can have very non-trivial issues, and they especially have nonstandard (and very nontrivial) transaction semantics. I'm not saying this is necessarily a problem, but unless you're on top of it. You need someone with real database administration experience.

You especially CANNOT "START TRANSACTION; SELECT * FROM products; SELECT * FROM five_other_tables; UPDATE sales ...; COMMIT TRANSACTION" (so all product prices and ... are included in the transaction "and nothing goes wrong"). It does not scale, it does not scale on postgres, it does not scale on Aurora, it does not scale on Spanner, it does not scale anywhere. It will scale on MySQL and MariaDB because it will just ignore your SELECT statements, which will cause a disaster at some point.

2) The big problem with cloud databases is the same as with normal databases: let bad developers just do what they want, and you'll have 40 queries per pageload, each returning 20-30 Mb of data. You NEED to optimize as soon as you hit even medium scale. And, sorry, "just slap a cache in front of it" is very likely to make the problem worse.

Cloud Hyperscalers salivate seeing this, have simply said: "don't pay an annoying database/system admin to tell you to FIX IT, simply pay us 10x what you pay that guy".

There's 2 kinds of people. There's people who look at the $10 pageload as both a technical embarassement and a potential business disaster. And there's business people who build hyperscalers, dine the CTO and hold a competition for a $100 pageload, while ordering their next Ferrari.

3) There's nothing automatic about automatic sharding. And if you "just shard UUID", your query performance will be limited. It will not scale. It just won't, you need to THINK about WHAT you shard, and HOW. No exceptions.

4) you'd be surprised how far ansible + k8s + postgres operator goes (for both CI/CD and distributed databases)

lumost
1 replies
1h38m

Once in my career, I did a high 7 figure migration from cloud to colo.

Our startups lead investor, was invested in a few firms that aimed to make a cloud experience on-prem. So we enjoyed strong board support, and had a strong team. On paper, the migration saved 10-20x its TCO over 3 years. We weren’t flying by night, and used high quality colo facilities near the cloud.

Everything went smooth enough at first, the hardware was late by 12 weeks - and there were some minor hiccups in the build. Time to turn-off of the original infra was maybe 2 months longer than planned. All in all 3 months of net delay where we were running both stacks and the financial clock was ticking. We even had industry press and tech talks on the success of the whole effort.

Now here is where problems actually started, 1 year post-build - folks wanted SSDs. While I fought to get SSDs in the initial build, I was overruled by a consultant for the main db vendor we had at the time. It’s tough to ask for 500k+ in SSDs with 1/4 the storage volume of spinners if everyone else thinks the drives are a waste.

2 years in, the main cloud vendor dropped prices by north of 50% for the hardware we had bought. To make matters worse, the service team wasn’t using the hardware we had to full effect - usage was around 25%.

By year four contracts weren’t renewed and the workload was migrated to cloud again. In total, we had probably saved 10-25% vs. aggressive negotiations with the cloud vendor. We could probably have beaten the Colo price if we had aggressively tuned the workload on the cloud. However there were political reasons not to do this. If you add in opportunity cost and the use of VC funds… we were probably negative ROI.

Given this backdrop, I never worked on colos again (this all went down 10+ years ago).

vidarh
0 replies
27m

So you had a badly planned transition, did it in a single big switch (why?), overprovisioned when already having cloud experience instead makes you perfectly placed for under provisioning, and it sounds like you paid up front for no good reason?

Sounds like lots of valuable lessons there, but assuming from this that cloud was better for you seems flawed.

1. Don't do migrations at once. In a large deployment you likely have multiple locations anyway, so do one at a time, and/or rack by rack.

2. Verify usage levels, and take advantage of your ability to scale into cloud instances to ensure you only provision tiny headroom on your non-cloud environment. Doing so makes the cost gap to a cloud environment much bigger.

3. The advantage is being able to tune servers to your workload. That you had to fight for SSDs is already a big red flag - if you don't have a culture of testing and picking based on what actually work, yeah, you may actually be best off in the cloud because you'll overpay by a massive factor no matter what you choose and in a cloud you can at least rapidly rectify the repeated major mistakes that will be made.

4. Don't pay upfront unless you're at a scale where it brings you discounts at a return that makes it worth it. Rent or lease to own. For smaller environments, rent managed servers. The cloud provider dropping prices 2 years in shouldn't matter because by year 3 you should have paid off the hardware at an average cost well below anyway, and start cycling out hardware as your load requires and as/when hardware fails and/or it's not cost effective to have it take up space in your racks any more. Depending on your setup, that might be soon, or you might find yourself still running 6+ year old hardware you paid off 3 years ago for many tasks.

If you can't beat cloud on TCO, you're doing something very wrong. I use cloud for lots of things that aren't cost sensitive, but there are extremely few scenarios where it is the cost effective option.

vidarh
0 replies
24m

I wrote an orchestrator from scratch in the pre-K8S days, and even factoring in the opportunity cost of that colo and eventually Hetzner was far cheaper than cloud, to the point where - and we costed this out yearly - the company wouldn't have survived a year if we had to pay AWS level prices for infrastructure.

Cloud is great for many things, but not cutting cost.

ehPReth
3 replies
14h26m

there's no way it actually costs them that much right?

kyledrake
2 replies
13h13m

Not even close. I think a 1gbps ip transit hookup at a data center goes for $200-400 a month right now. And you can push 192TB a month with that. That's not the bulk cost, they buy larger amounts in volume and probably get a huge discount on it. In fact given their size, they might not pay for it at all (peering agreements), so it's just operating costs.

nixgeek
1 replies
12h51m

A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards.

Add on the usual and egregious $150-400/mo for a SMF/OS1 crossconnect in many datacenters.

Now, I'll grant you, "not even close" and handsome margins likely still applies but "I'm going to hook a server directly into a 1Gbps IP transit port for $200" vs. all the complexity of a hyperscaler network is not apples to apples.

kyledrake
0 replies
4h31m

The important takeaway message I'm trying to deliver is that AWS is comically overpriced regardless of what they are trying to run. Direct connections aside, reasonable cloud competitors with similar networks are charging $0-1000 for transit and the same amount would cost $8500 on AWS. Cloudflare in particular has demonstrated that you can run these systems so efficiently that they can charge nothing for egress and still make a healthy profit selling related cloud services.

supriyo-biswas
2 replies
13h7m

In the case of fly.io it’s a little different because they have a anycast network with traffic being accepted at the closest region and being proxied to your running VM. This is not unlike a CDN, and if you look at the North America/Europe pricing of most CDNs, fly.io is quite similar.

wolfendin
1 replies
11h3m

Anycast does not really make the pricing that different.

Especially since one does not have to backhaul the traffic to one specific PoP.

Cloudflare manages to run their Anycast CDN as a loss leader just fine.

supriyo-biswas
0 replies
10h41m

I believe Cloudflare is in the long-haul of commoditizing their competitors and making their profit through other auxiliary services.

Also, on the plan routing is atrocious, with all requests from Africa and Asia being directed into Europe, which helps keep costs down, but it cannot then be compared with fly.io.

avereveard
2 replies
16h18m

Aws transfer costs seems off by a factor of ten sticker price is $0.09-0.14 per gb depending on region how did you get 100gb for 98$?

cchance
0 replies
13h48m

100gb is the free 1tb is for the 98$ as the pricing based on 1tb chunks in the article

angoragoats
0 replies
15h18m

If I'm not mistaken, this is copied from the OP, which is pointing out that you get 100GB/mo free (that's the "100GB/mo" part). The $92.16 cost is for 1TB of egress. It's clearer in the original article.

phonon
1 replies
17h15m

AWS Lightsail is actually pretty good, comparable to Linode (except it maxes out at 7 TB/month).

https://aws.amazon.com/lightsail/pricing/

hughesjj
0 replies
13h25m

It's also in the SLA to not use lightsail as a proxy

tomcam
0 replies
17h47m

I’m guessing you can negotiate for sharply reduced rates. No one pays list if they can help it.

quickthrower2
0 replies
14h45m

With Netlify it costs more than hiring a courier to ship an SSD that you paid for and turn up in person and copy it and deliver it next day.

blackoil
0 replies
16h6m

It looks like Netlify $550 also includes hosting and CDN so not apple to apple.

quickthrower2
19 replies
14h51m

So an anticompetitive practice.

zer00eyz
18 replies
9h55m

No.

Everyone knew what the costs were going in.

AWS, cloud was NEVER the cheaper solution. It was never faster, it was never better.

It was easier, and easier came with a price. Like lambs to the slaughter lots of people embraced it. Amazon is a profitable company because of it.

DeathArrow
5 replies
9h43m

It was easier, and easier came with a price.

I kind of doubt it's easier. If you use your own servers you need some stuff to manage them. If you use cloud, you need some cloud engineers to manage your cloud infrastructure. On top of that, if your developers use cloud APIs and frameworks, they have to learn that.

whizzter
1 replies
8h37m

As someone working at a dev shop/full-service-agency mostly using cloud(Azure) and having also managed server on my own in the past it's an easy breakdown.

1: We have a dozen or so regular clients and being in a relatively high salary cost country, while our cloud bill isn't insignificant it's still far less than one full time employee, even less so than having 24/7 on-call handled (we have large enough 24/7 critical clients).

2: The cloud provides broken down billing, it's easy to pass it to the customer and just point to it, having our own employee we would need to structure up the billing process somehow. Our customers are outside of the IT field and doesn't care as long as it works and doesn't cost too much (it's a mostly fixed expenditure and easily separable from our development hours).

3: We have one client that does on-prem (partly out of habit) that's actually looking at parts to the cloud, partly because they've expanded internationally but I wouldn't be surprised if it's a bit of testing the waters since they have to spend a fair bit of their time on security and stability (security doesn't go away entirely of course but I'd be more sure of my backup processes on a large cloud compared to just locally).

4: I'm a tad on the fence of cloud-api's:

On the positive side: No need to implement a bunch of systems(badly?) and no need to build/install monitoring tools for those (and the extra administration burden that comes with updating those tools for X number of clients).

On the negative: There's a tad of churn on some cloud-api's, luckily Azure is mostly stable with plenty of migration time until you come around to a customer again (reading about Goog's churn feels like horror stories), and yes there is a bit of vendor lock-in if you go all-out (luckily we've not seen any reasons to change over).

vidarh
0 replies
6h20m

If you don't still have that on call ops engineer, you're tricking yourself into thinking you have cover you simply don't. The low level hardware and networking issues that a cloud provider would abstract from you in a well set up system are a rounding error of the set of ops issues.

I used to provide services like that, on call, and the number of ops related interventions for my cloud customers were consistently higher than for those on colo setups.

karolist
1 replies
9h20m

It's easier but only at the beginning, when the needs are simple and you dream of unpredictable QPS spikes that cloud will magically handle for you, summed up pretty well in https://world.hey.com/dhh/we-have-left-the-cloud-251760fb

whizzter
0 replies
8h22m

I think the QPS dream is something that was teased at to show feasibility (and scaling up services on the cloud IS simple when needed).

I think the bread and butter for the big clouds however are customers like the shop I'm at (see sibling comment), we can do development work and put it on our clients (mostly non-IT that has no interesting in hiring their own department) and if we for some reason need to part ways it's easy to "hand-over-the-keys" since subscriptions can be handed over between organizations.

But getting customers back is usually equally easy since there was never any fuzz when leaving so when they've become dissatisfied with the "cheap" options they don't remember leaving being a problem, regardless the winner is the cloud provider (but the costs are still small enough for each client that nobody is unhappy).

willtemperley
0 replies
7h19m

Yes, AWS-style cloud is just software-defined infrastructure.

It's like virtual lego - the bricks fit the same way as physical lego but the difference is you can rent as many virtual bricks as you like, almost instantly with little up-front cost.

You still need to know how to build the lego pirate ship either way.

quickthrower2
3 replies
8h40m

Simply being expensive isn’t anti competitive. I wasn’t suggesting tbat.

zer00eyz
2 replies
6h34m

I got that, but anticompetitive means something.

Cloud wasn't predatory in this... the price has always been there. If you lease a car and they mandate you get your oil changed with them for a price, and the price sucks, that's YOUR fault.

No one got locked into the cloud and THEN the egress prices went up... everyone went in knowing this.

The industry needs to man up and own shitty decisions rather than double down on them forever.

Wytwwww
1 replies
3h19m

Cloud wasn't predatory in this.

I'd argue it's still quite predatory. Basically, they reel in users with other services/products and then charge extortionate prices for egress specifically with massive margins just because they can.

In certain use cases it becomes all or nothing, either you host everything on the cloud or nothing at all which is certainly abusive and highly anti-competitive.

zer00eyz
0 replies
2h53m

Sign up now for free (and we will auto charge you next month)... The nice razor handle is cheap, but the blades are expensive.

No one hid the prices from any one... it was all very up front and out in the open. No more data centers no more systems admins, no more capacity planing just scale on a dime... Here is the price chart....

And everyone didn't want to do that HARD work any more. They could just grow and grow and grow and it would save them money.

You know what happens when your not gonna run out of storage or bandwidth or servers... Everything gets bloated.

SO sure, we can say abusive... Amazon abused all the poor stupid vp's who took the lazy way out and let their systems get so fat and bloated on button mashing "l" shaped engineers. Crying about the lock in, about the pricing after you signed up for the service is your own fault. Take the hit and move on!

morningsam
3 replies
6h46m

There have been other instances in which exit fees, which is what this amounts to, were considered anticompetitive, e.g. [1] (although this was settled so there is no ruling). Google itself has started to waive egress costs for GCP customers leaving the platform last month, which, according to some sources, is simply a direct consequence of new EU legislation (Data Act) [2], but according to others is in anticipation of wider-reaching EU antitrust investigations [3].

[1]: https://www.agriculturedive.com/news/koch-foods-antitrust-ch...

[2]: https://www.computerweekly.com/news/366566360/Googles-data-e...

[3]: https://blog.min.io/googles-new-egress-policy/

growse
2 replies
5h45m

I'm a little ignorant of the upcoming regs, but are they aiming to basically say "if a customer wants to leave a service, then the provider must provide them with all their data for free"?

I've not thought about the unintended consequences of this, but it feels like a reasonable regulation to have.

close04
1 replies
4h23m

Ingress and egress should be treated the same. Without any anti-competitive reasoning in mind a provider can’t claim that egress in particular should be more expensive than ingress. Even more so when ingress is many times completely free.

The asymmetry is obviously meant to trap customers, which is anti-competitive.

rstupek
0 replies
59m

I'm pretty sure ingress started off costing less due to the nature of usage. If you were to see a graph of the usage you'd see ingress is used less. One reason is an http request takes less bytes than the response to the request. So the disparity in cost can't simple be attributed to a desire to trap customers

ctrw
0 replies
7h6m

That is not how being anti-competitive works.

creativeSlumber
0 replies
1h8m

Everyone knew what the costs were going in.

Knowing the cost beforehand and anti-competitive are two different things.

amluto
0 replies
5h11m

Everyone knew what the costs were going in.

I disagree. The company that put their infrastructure on AWS may well have made an informed choice. But the third party service hosted in AWS doesn’t have a choice, because AWS would punish their mutual customer with egress fees if the service moved out of AWS.

Sytten
0 replies
3h57m

But it was heavily marketed that way so at the very least you have to acknowledge that there was a fair bit of false advertisement...

I don't think making an informed decision has anything to do with competitiveness in any case. Example I might make an informed decision to publish my app on the app store, but this doesn't mean its practices are not anti-competition.

Eridrus
6 replies
20h20m

Snowflake's margins are like 90%+ on top of the compute they sell, and they pass on all these costs including egress directly to customers.

amluto
3 replies
19h51m

Their list price for storage capacity is only on the order of 2x what S3 charges, and Snowflake and S3 likely both offer discounts. Comparing compute costs is harder.

If I were running a service like Snowflake, I would certainly appreciate the effortless scaling that the major clouds offer. But I also know what I pay for actual servers and roughly what I would pay in a major cloud, and I much prefer the former.

dalyons
1 replies
19h46m

tbf, snowflakes whole thing & what makes them "next gen" vs a traditional datawarehouse is that you dont may much for data at rest. You pay as little as possible for the "warehousing" part, and pay mostly/only for compute/querying.

pas
0 replies
19h16m

wait, you still pay your S3 storage bills, right?

just with DataBricks. you don't pay when you don't use their software, cool. but then you pay for per core per second for the license in addition to running the VMs, right?

sure S3-like storage is 'cheap', but if you want up-to-date data pipelines you'll end up running these things for many hours each day. and plus you pay for each bucket operation, and so on.

Eridrus
0 replies
19h46m

Yeah, storage is much more competitive, but there have been leaks on what hardware they run on, their margins are incredible, which makes sense since they are selling software, not hardware!

Anyway, there is an explicit egress charge: https://docs.snowflake.com/en/user-guide/cost-understanding-...

infogulch
0 replies
19h3m

That's not the point. The point is that if the customer and SAAS provider "happen to" be in the same cloud then nobody pays any egress costs, but if the provider operated out of their own datacenter then the customer would pay additional egress costs to AWS to export their data to the SAAS provider. Snowflake is a good example because by the nature of the business customers send them a lot of data. This not only makes the customer pay more, but it also makes their billing and cost accounting much more complicated and unpredictable.

api
0 replies
19h30m

“Let’s get rid of IT and stop using on prem software. Outsource it all to the cloud and SaaS. We’ll save so much money!” … said the entire world pretty much …

This is hilarious. Now that nobody makes on prem software anymore and nobody knows how to do IT, it’s time to turn the screws. The prices are only going up. You can’t move your data because it’s all locked into proprietary silos and only readable by one app running in one place.

I told you so. So did loads of others. Oh well.

snapplebobapple
0 replies
16h14m

Its this plus locking customers to one cloud, because egress kills intercloud syncing on any moderately large data set. Any smart customer would have a duplicated setup across clouds if egress cost what it actually cost instead of 100x plus what it actually costs

rfoo
0 replies
5h40m

While I agree with what you said in general, Snowflake is a poor example. Data warehouses like Snowflake really can use the Tbps+ aggregated bandwidth between S3 and EC2 in the same region. There is no way for this to work over the Internet.

ren_engineer
0 replies
2h3m

Snowflake and most other cloud services offer on-prem for these use cases, so it really doesn't make sense for them to roll their own data centers because it would be pretty niche. Cloud works for startups because they are there already, on-prem for enterprise customers with their own hardware

jupp0r
0 replies
17h16m

It's an interesting point, but I doubt it's the lions share of egress that's going to other data centers vs to customers. Fan out is where it gets expensive.

ctrw
0 replies
7h7m

It's not a theory it's literally why two projects I was on used snowflake in aws.

Spooky23
0 replies
11h19m

That’s definitely a factor. You can even pay AWS for the infrastructure used by SaaS apps to aggregate more volume.

toast0
11 replies
21h17m

My feeling is that egress is easily measured, so it's where costs that are hard to assess get moved to.

It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers.

I think there is also some geo-specific pricing that gets hidden in a global price; bandwidth can be a lot more expensive in some locations than others and if you are charged 5x for egress in south america, nobody will use the south america locations and that's not good for business.

amluto
7 replies
20h26m

I don’t believe this. Operating an internal cloud network is expensive, but it’s expensive because of internal traffic, and they don’t charge for that internal traffic. Egress is just like traffic to any other system, and AWS doesn’t charge for that.

Also:

It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers.

It’s quite a bit worse than 10x

zokier
6 replies
19h24m

AWS charges for internal traffic too, both cross-az and cross-region.

sgarland
5 replies
19h6m

The inter-AZ charges kill me. They’re charging you for traffic over what is almost certainly private fiber, of a distance no more than 50 miles or so. Just absurd.

Then, they have the audacity to recommend you make everything Multi-AZ. “Don’t worry, RDS-RDS traffic isn’t billed!” Yes, but the apps running in us-east-1a that are hitting -1b and -1c due to load balancers most assuredly do bill for traffic.

jiggawatts
1 replies
13h13m

What’s extra special is that the cloud providers are now disincentivised from making those load balancers zone topology aware. If they made them efficiently pick the same zone whenever possible, they would be missing out on cross-AZ traffic billing, which is basically a money printer.

zokier
0 replies
10h31m

AWS ELB certainly is zone aware, iirc cross-zone load-balancing is opt-in

aflukasz
1 replies
9h10m

BTW, if nothing changed the last year or two, then even if you want to run in a single AZ, but also to have (automatically) replicated RDS instance, then I believe the replica MUST be in another AZ. Meaning that any potential failover puts you in exactly that position you have mentioned - apps running in one AZ, database in the other. Suddenly you pay 0.01USD/GB for transfer between those, billed in each direction separately.

sgarland
0 replies
2h4m

Correct. The same thing also happens if you use the Blue/Green functionality to do an upgrade – unless you specify (and even then, I'm not sure if you can do so), the Blue instance will be in a different AZ than Green.

wmf
0 replies
18h39m

Not billing for S3 and RDS replication traffic is also an unfair subsidy for those services.

raid2000
2 replies
20h47m

Right. Egress is an imperfect, but reasonable metric for overall utilization. If they started charging for CPU hertz above a certain threshold, that'd be a harder sell.

zokier
0 replies
19h23m

If they started charging for CPU hertz above a certain threshold, that'd be a harder sell.

You mean like EC2 T series burstable CPU credits?

felbane
0 replies
19h9m

I mean, they kinda already do depending on your plan (see: AWS EC2 spot instances)

ozr
11 replies
21h7m

Definitely lucrative enough. The use case you've described isn't particularly uncommon, but lots of companies just pay for the egress.

The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.

api
10 replies
20h34m

I’ve had people refuse to believe that bandwidth is actually very cheap and cloud markup is insane (hundreds or even thousands of times cost).

I show them bare metal providers and colo that bills by size of pipe rather than transfer. They refuse to believe it or assume there must be a catch. There usually isn’t, though sometimes the very cheapest skimp on things like rich peering and can be slightly slower or less reliable. But still cheapest is relative here. Expensive bare metal or colo bandwidth is still usually hundreds of times less than big three cloud egress.

It’s just nuts.

It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…

cvwright
4 replies
18h28m

Equinix bare metal bandwidth is pretty crazy expensive too.

5c/GB is less than the 9c that Amazon charges, but still, ouch.

api
1 replies
17h56m

They are an outlier… an extreme one.

jpgvm
0 replies
9h54m

Also very few people buy Equinix transit except maybe as a backup.

You buy space in Equinix DCs because they are network neutral and have a high density of ISP POPs, i.e you can light up fibre between your cage and HE, Level3, whatever other transits you want to use + peer with everyone on the local peering fabric super easily.

I consider Equinix transit to be an option only when you really can't be assed to do anything else and you don't intend to hammer the links anyway, otherwise yeah you should be doing all the things above because otherwise you could have rented colo in some shitty backwater DC instead.

tonyhb
0 replies
18h9m

This is my biggest gripe against Equinix. Other than that, they're 10/10.

rsync
0 replies
15h18m

You pay equinix for the space.

You pay hurricane for the bandwidth.

kkielhofner
2 replies
18h55m

Cloud providers have done an incredible job creating an entire generation of people who have no knowledge of such things.

$0.09/GB? I guess that’s just what it costs. Expected when you’ve never looked at, considered, or even heard of things like peering and buying transit or even co-location. Enter 95th percentile billing on a gig port for $500/mo or whatever…

Same goes for hardware. Want to watch something glorious? Take someone who has only ever used VMs, etc in big cloud and give them even a moderate spec new bare metal server. The performance melts their brains.

Then they realize the entire thing - colo, servers, bandwidth, even staff is a fraction of the cost.

api
1 replies
18h51m

A big part of the problem is that software development has wrapped itself around cloud native. Sure that metal server has brain melting performance for nothing, but try provisioning it or running it. Nobody knows how and we are dependent on stacks like K8S that require an ops team to run.

The industry really found a way to stop the downward cost spiral of personal computing and bring everyone back into mainframe. Moore’s Law is not dead but you no longer see the benefits. The difference is just pocketed by cloud providers who will use the gains to pack more low performing VMs onto less hardware.

sleepydog
0 replies
14h25m

Maybe oxide can pull the needle back in the other direction -- cloud should not be the default choice for every company

re-thc
0 replies
12h12m

It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…

No 1 believes they can do anything anymore. It's not just hosting and hardware either. Everything single thing becomes a SAAS that gets purchased and they all refuse to believe it can be done in-house.

It’s just nuts.

It's exactly that. I've seen upper management in many companies value external solutions as more cost effective because if they had to do it internally it'd cost 2x as much.

hsbauauvhabzb
0 replies
17h2m

Any recommended reading on how this stuff truly works? I feel like I understand the core protocols (tcp, ip, bgp, etc) not not how the internet is ‘built’

thimp
6 replies
21h16m

I saw a hilarious fuck up a few months ago. Company sets up an AWS hosted always on VPN solution. Connects 1000 staff through it. Celebrates how they saved $50k on the VPN solution. Gets $25k AWS bill for the just the first month of egress traffic. Turns out the data was leaving AWS egress three separate times.

ailurooo
3 replies
20h3m

how were they spending that 50k previously? a bespoke saas thing or self hosting?

dijit
2 replies
18h54m

Likely they are mixing timelines (one time cost or yearly cost with AWS’s monthly charges).

Cisco anyconnect VPN capable appliances (that can do 10GBE) are very expensive, and licenses are per user- so if an appliance needed an upgrade it is conceivable that it could cost $50k in the first year.

thimp
0 replies
8h31m

This. It wasn’t Cisco. Far crappier vendor! I don’t want to name names but they had a few high profile incidents that suggested their software was written by idiots. There was a panic move by the security people to switch away and still tick the box but they didn’t really understand the billing and architectural models of AWS.

Pricing was around 75k for 1000 seats for a year. They thought it was going to be $25k a year but it turned out to be that a month.

CaliforniaKarl
0 replies
18h50m

I agree. You're probably looking at:

• The cost of two hardware gateways (the depreciation cost, that is)

• The cost of 24x7xNBD (next business day) hardware support

• The cost of the user/session licenses (which might be depreciated also)

• The cost of software support/upgrades

constrain5795
1 replies
18h27m

AWS VPN base cost is $72/mo + $36/u/mo. So they were spending $80k/mo before?

thimp
0 replies
8h34m

It was not an AWS product before or after. It was a SaaS VPN before and a vendor product running on EC2 after. The TCO for a year was $50k before (licenses mostly) and it was replaced with a total cost per month of $25k (licenses, EC2 instances and egress bandwidth) because of misunderstanding of how it worked. As the concentrator and egress gateway were separate components they deployed them in the home region for the org. The end was an egress damage multiplier because local traffic was routed globally to a different AWS region over public internet and then went egress.

There was also a global policy rolled out which pushed the VPN client to various infra machines in branch offices. Some of them were running GitHub runners and data processing software which had terabytes of egress per month.

The whole thing was a disaster and they rolled it back within a week.

asmor
6 replies
21h18m

It really depends on the quality of the peering you expect. It doesn't matter, until it does. Consumer ISPs sometimes do their utmost to not peer with open exchanges, and the entire thing gets even more complex when you go to places where bandwidth is more expensive (i.e. Oceania).

There's a reason the favorite chart to exemplify value Cloudflare reps like to show is Argo Smart Routing, and why it costs about $100 per TB just like AWS and GCP.

Fripplebubby
4 replies
19h42m

I agree, and I would also put forward that most people don't understand what peering is or how it works. When people (usually developers who are not network engineers and have not worked at that level of the stack) talk about "egress", they mean delivering bits from your network (cloud or otherwise) to any other network on the internet. How can you put just one price on delivering a bit either to a host within the same datacenter or one on the opposite side of the planet? Physics still mean that one is more expensive than the other.

The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".

quickthrower2
1 replies
14h40m

Like to see the data on that! Would it really be cheaper if they charged granular by cost like old telephone plans where they list each country and c/min + connection charge?

asmor
0 replies
8h39m

The big clouds already do this by continent.

wmf
0 replies
18h36m

Transit is also the same price regardless of distance.

pests
0 replies
16h20m

USPS manages to do it.

naiv
0 replies
20h49m

We are using Argo for our xhr search traffic as it makes more sense than setting up different servers/vms in parts of the world. Each request is only 1kb max.

But I would not use it for static assets. For this we use Bunny edge storage to provide faster response times at very reasonable prices.

hatsix
3 replies
14h56m

Our 20tb/m costs us $414... if you're saying that you can amortize those servers, rent the space and pay for upkeep for less than $4k a month?

We have another service that has 20x the bandwidth, but it's a legacy GAE that has super cheap premium egress... But I'm told that AWS says that their pre-pay discount will be competitive in about a year, the rate that we're growing.

The negotiated egress prices are much lower, so long as you are buying future egress. If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

shiftpgdn
0 replies
12h14m

I could buy a whole off lease fully deprcriated server and a year of colo with a gig link for under $4,000, yes. I would almost certainly see better performance than any cloud provider can give me too.

re-thc
0 replies
12h22m

If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

Shouldn't it be the other way around? Why are they giving you are discount if you're not likely to jump?

pinkgolem
0 replies
13h28m

If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

Aka as long as you are not using the cloud as cloud the cloud can be cheap?

rent the space and pay for upkeep for less than $4k a month?

I mean, you can just rent servers somewhere else.. or do a colocation rack. Also I would strongly suggest going with the first one

electroly
3 replies
19h18m

If you still want to use some AWS services, you can get an AWS Direct Connect fiber cross connect from your data center rack to AWS, just like you do with your Internet connections. They operate Direct Connect in lots of third party carrier-neutral data centers. AWS egress over Direct Connect is $0.02/GB instead of $0.09/GB over the public Internet. You can serve customers through your unmetered Internet connections while accessing S3 (or whatever) via Direct Connect on the backend.

ricktdotorg
1 replies
15h13m

i'm a customer of the GCP equivalent: partner interconnect. our DC is in an equinix facility, they wire up drops for us that layer 3 straight into GCP. unmetered 1Gbps for about 250 bucks a month per (paid to EQX not GCP). are AWS really charging you per Gb for data egress from AWS into your own DC over an AWS direct connect??

electroly
0 replies
15h9m

I'm not familiar with GCP but this page below seems basically identical to AWS Direct Connect's pricing structure, including $0.02/GB for egress. Are you sure nobody is paying egress on your behalf?

https://cloud.google.com/network-connectivity/docs/interconn...

Ctrl+F "Data transfer from a VPC network through a Cloud Interconnect connection"

amluto
0 replies
18h39m

I can pay overpriced cross-connect rates in giant name brand datacenters, with or without terminating one end at Direct Connect. (AFAICT the $1600/mo or so for 10Gbps doesn’t actually cover the cost of the cross-connect.)

But that extra $65k/mo to fully utilize the link is utterly and completely nuts. My mind boggles when someone calls that a good deal. I can buy and fully depreciate the equipment needed to utilize that in a couple of days. (Well, I can’t actually buy equipment on the AWS end, but I can _rent_ it, from AWS, for a lot less money than $64k/mo.)

And I don’t believe at all that it costs AWS anything close to this much to operate the thing. I can, after all, send 10Gbps between two EC2 instances (or between S3 and EC2) for a lot less money.

That $65k is simply a punitive charge IMO. AWS wants me to avoid paying it by not doing this or, in cases where I can’t avoid it (e.g. the thing is actually a link to my office or a factory or whatever) to collect as much money as they can without driving me off AWS entirely.

xyst
1 replies
19h31m

just 1 engineer? lol. Let's hope it's not business critical for the servers to be up and running at all times if you only need 1 engineer.

pinkgolem
0 replies
10h50m

you need an oncall team be it cloud or not, taking one fte position and dedicating it to managing 2 servers(documentation/updates/backup & restore procedures testing) seems rather.. high

jonatron
1 replies
21h35m

The AWS Enterprise Discount Program apparently requires $1M per year spend. 180TB is about $13k on AWS so presumably not enough to be interesting to them. Hopefully someone who works at AWS can share some info.

tky
0 replies
20h32m

EDP can be great if you can meet their required year over year growth requirement and if your spend is high enough to make the discounts (which are variable and negotiated often at a product offering level, not blanket) offset the required top-shelf support contract. For smaller orgs even at the $1-2M/mo level, it can often be a risk not worth taking vs other cost savings mechanisms and locating high-egress use cases elsewhere.

Egress bandwidth pricing has been the sacred cow money firehose forever, despite transit costs continuing to shrink.

belter
1 replies
18h22m

Context matters here. How critical is that workload? What the economic and reputational impact for the company, if one of the physical connections or some technical problem with the data center causes a downtime of hours or days?

pinkgolem
0 replies
10h54m

looking at the uptime from aws and from most big outtage notices i have read in the last few years, there does not seem to be a benefit in regards to reliability when using cloud.

see reddit, see amazon/aws outages taking with them netflix/disney plus etc

honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region

also: store your backups somewhere else, not on your primary cloud provider, not connected to your auth system, not overwritable from anywhere

vidarh
0 replies
6h27m

Yeah, I've built several setups where total cost of ownership was lower than egress cost alone would have been on AWS. Both w/physical hardware in colos and with rented managed servers at providers like OVH and Hetzner.

rsync
0 replies
15h25m

Not every cloud provider has egress charges…

mgaunard
0 replies
7h45m

There is no scenario in which cloud is cheaper than bare metal to operate.

What's cheaper with cloud is that you don't need upfront costs, but here, you already have the whole infrastructure already there, up and running.

Why would you even consider getting rid of it and replacing it with AWS, makes no sense.

kansi
0 replies
20h27m

Are you able to share which provider do they use for their rack setup?

intelVISA
0 replies
3h32m

Between the vendor lock-in and egregious costs it's a sad state for so many in our profession to call this 'the future'.

cocothem
0 replies
4h43m

Are you a Google Cloud customer looking to exit Google Cloud? If so, you are eligible for free data transfer when you migrate all of your Google Cloud workloads and data from Google Cloud to another cloud provider or an on-premises data center.

https://cloud.google.com/exit-cloud

andix
0 replies
18h5m

180 TB is still a small customer in the scale of big cloud providers, so they probably just don't care. If the customers are willing to pay the price they are taking their money, if they go to one of the smaller providers they are fine with it too.

Also it could be possible to set up some hybrid solution and offload the egress heavy assets serving to another provider and only run the "brains" inside AWS/etc.

Scubabear68
15 replies
19h34m

The article says “ Cloud providers charge for egress because it costs them money to send data out of their network. They have to pay for the infrastructure and bandwidth required to send data to users”.

The charge is not based on cost in the case of the big names. They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free.

Hence the high costs to deter this behavior.

ehhthing
13 replies
18h33m

They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free.

This isn't remotely true.

The bandwidth alliance exists, and a lot of cloud companies are on the list: https://www.cloudflare.com/en-gb/bandwidth-alliance/

The actual answer is much more complicated. For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive.

Google cloud's "premium" bandwidth is much akin to AWS Global Accelerator since it utilizes Google's own backbone network for as long as possible before exiting at the nearest peering point between Google and whatever ISP your end user is at. AWS Global Accelerator has some other options available, that make it fundamentally a different product, but the routing characteristics are much more similar to GCP Premium bandwidth than anything else AWS offers.

Scubabear68
6 replies
16h55m

That’s an impressive word salad, but sorry, no. Egress costs are high strictly to keep you there.

This is why egress is dirt cheap on other platforms outside the big 3 cloud providers.

This is also why ingress is free.

ehhthing
5 replies
10h2m

Ingress is free because it helps them balance their pipes, and it would be really shitty to charge for DDoS attacks. As far as I can tell, with the exception of some really expensive network environments (e.g. China), nobody has ever charged for ingress.

With an exception to OVH, none of the cheap providers the article has listed have any kind of backbone network. They all rely fully on transit providers. Turns out backbone networks are expensive to operate!

OVH is the sole example of a provider that has a backbone network, and admittedly, it's pretty good. However, nowhere near expansive as the big three, and it falls flat in Asia (which is the hardest to route traffic in). Also OVH has to build datacenters so cheaply that one of them burnt to the ground in recent years...

(Cloudflare has a backbone too, but you have to pay a lot extra to use it. Linode uses the Akamai backbone now but that's a very recent acquisition and it's expected that Akamai will eventually raise costs significantly)

Yes, bandwidth is way too expensive on cloud providers. AWS Lightsail is proof of that. However, I see no reason to believe that this is purely for vendor locking, and nobody has been able to give any evidence of causation between the two beyond "well it's so expensive!!!"

Scubabear68
2 replies
2h38m

Again with the word salad.

It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform.

They therefore charge high fees to move your data off the platform to discourage this behavior. Meaning you now need to use cloud provider X’s services to do anything with the data.

Attempts at vendor lock-in have been core to software service companies since they were born.

ehhthing
1 replies
2h20m

It is very simple. If you move your data off cloud provider X, cloud provider X is losing revenue because you are doing things with your data off their platform.

Right but if this were the case then why does the Bandwidth Alliance allow you to move data at a much lower cost for 2/3 of the major cloud providers? If they _really_ cared so much about not allowing you to do processing with a third party, the Bandwidth Alliance wouldn't exist!

AWS is the sole hold-out here, and I think the way that Cloudflare worded this makes it pretty clear that the Bandwidth Alliance is basically a middle finger to AWS than anything else, but it also seems clear that the cloud companies aren't actively trying to make it costly to do data processing on a third party.

In fact if you want to move off GCP right now, Google will waive all egress fees to do so: https://cloud.google.com/blog/products/networking/eliminatin...

Scubabear68
0 replies
16m

Bandwidth alliance looks to be a political tool for cloud providers to save face. Not dissimilar to public companies paying token tribute to ESG which is all the rage these days.

Nevermark
1 replies
4h49m

Ingress is free because it helps them balance their pipes

That isn't how business works. Companies maximize their profits and "balance" isn't a profit center. If it didn't benefit them in some customer leveraging way, they would charge for ingress.

You pay for everything. Either directly or indirectly. Indirectly often turns out to be much more expensive.

ehhthing
0 replies
2h29m

That isn't how business works. Companies maximize their profits and "balance" isn't a profit center. If it didn't benefit them in some customer leveraging way, they would charge for ingress.

What I'm referring to is the practice of balancing peering ratios. That is, when you make transit/peering arrangements with other ISPs, some ISPs will charge more if the amount of data you're sending to them vs the amount of data you're receiving from them is not balanced. It is in Google's best financial interest to at least try to balance their pipes in this way.

acdha
2 replies
16h56m

The bandwidth alliance exists to try to cut into AWS’ business. They could always have unilaterally cut rates closer to their cost but that margin was appealing, until they realized that they were never going to catch up with AWS without being cheaper.

ehhthing
1 replies
9h50m

This is also a fair take, but not a very compelling reason why bandwidth costs are vendor lock in...

acdha
0 replies
5h28m

High egress makes it expense both to leave and to use other services: if you use S3, you’re probably putting processing and analysis in AWS because using someone else’s service would incur hefty egress charges.

foofie
1 replies
10h53m

This isn't remotely true.

Nothing in your comment rejects or disproves the claim that egress costs are vendor lockdown.

Your link to the bandwidth alliance explicitly states that their justification for network costs is unloading infrastructure costs onto end users as data fees. That's their only and best justification. This is clearly a business decision that has no bearing in operational costs.

Some cloud providers charge nothing, others only start charging after hitting a high threshold from a single instance. Do they not operate infrastructure?

It's their business, it's their business model. Some restaurants charge you for a glass of tap water too. Let's not pretend they do it because of infrastructure costs.

ehhthing
0 replies
9h57m

Some cloud providers charge nothing, others only start charging after hitting a high threshold from a single instance. Do they not operate infrastructure?

Yes you do pay for the rest of their infrastructure when you rent servers from them...

I'm not saying that the fees aren't extremely overpriced. I know what a gigabit port costs. But saying it's to keep vendor locking is just not true, and nobody has suggested any actual proof of it being true.

teraflop
0 replies
17h4m

Google's "standard" bandwidth pricing is about 15%-45% cheaper than "premium", which is admittedly a significant discount, but it's still an order of magnitude more expensive than some of the other options on the list.

dilyevsky
0 replies
19h12m

It also forces downstream vendors that ingest a lot of data (e.g datadog, snowflake) to maintain cloud presence because their customers will not to want to pay egress fees

geor9e
14 replies
21h28m

I had no idea it was this crazy expensive. I can host a 1 TB folder of movies on my Google Fiber internet Synology NAS website, and let my friend download it for free. But if I hosted my website on Google Cloud, they bill me $111.60? How are these cloud services getting away with this pricing?

jeffbee
7 replies
20h19m

Your little NAS is not directly connected to hundreds of metro areas around the globe. Even on Google Fiber, which is an excellent network, you only have a few dozen points of exchange, all in America. As a Google Cloud customer your egress traffic will be on Google's network all the way to the very edges of the world, at hundreds of interconnect points and exchanges. Your traffic will be on private networks until it is with a few miles of your end user, in all likelihood. This is a comparison between apples and diamonds.

veeti
6 replies
19h54m

Except they will charge you the same extortionate egress fees for outbound traffic from a single compute instance ("an apple"), no global content delivery network involved ("a diamond").

wrs
5 replies
19h28m

It doesn’t matter where your instance is, the point is the traffic will flow on Google’s worldwide private backbone until it gets very close to where it’s going.

Aeolun
4 replies
17h49m

You say that like its a service Google is providing, but it’s really just cheaper for them.

jeffbee
3 replies
16h34m

Explain that. Google egress is expensive because it is capable. You could operate a product like Zoom from it.

pests
2 replies
16h0m

This is similar, with different tradeoffs, to what Riot did for their gaming backbone. They wrote a blog about this but I cant't find it.

Instead of relying on consumer ISP's to route gaming packets for speed, they would instead route them for costs. Taking extra hops and extra latency before reaching the target server.

Riot built an overlay network using their own fiber and negotiated BGP deals with consumer ISP's to get gaming traffic off the ISP's internal network on onto Riot's ASAP so they could route the packets directly to their gaming servers.

billywhizz
1 replies
14h57m
pests
0 replies
13h5m

Thanks. I think those cover details of what I'm talking about but not the blog post I remember.

A key issue discussed was some setting relating to BGP (or similar) that consumer ISPs wouldn't initially implement or wouldnt configure for Riot. Something related to weights or something that affected the cost analysis for routing decisions that prevented packets from wanting to take the Riot path.

edit:

Just fully read that QZ article. I think it does a good job of describing what they did with useful inforgraphics (esp the "Actual Data Route" graphic) for a layman audience. I think many of us here can extrapolate what kind of effort they had to go through to get that done.

tootie
2 replies
14h2m

Residential usually has some kind of hard data cap. Like if you transfer >10TB they'll just cut you off. You're also generally barred from running any kind of commercial application.

metadat
0 replies
13h44m

The data cap thing is generally only with comcast and other tacky low-quality hall-of-mirrors bargain-bin types of cable ISPs that don't sell proper Internet connections (maybe Cocks Cable, too?).

For example, AT&T fiber has no bandwidth caps and offers symmetric 1-10 gigabit service in an ever expanding number of areas throughout the US. I pay $55/mo for 1gbps service, and have done more than 100TB/mo for many months and it's fine.

alberto180
0 replies
7h50m

Maybe in the US. Not here in Europe

redox99
0 replies
7h47m

They figure most people are hosting websites which are only a couple of MB, so they need to charge this much to be able to extract egress money from them.

hacker_newz
0 replies
18h33m

Your Google Fiber IP is part of a residential network with a completely different service agreement and larger limitations than Google Cloud.

dehrmann
0 replies
19h58m

let my friend download it for free

Except you paid the cost for the fiber connection. You also get one more 9 of reliability from their DC.

teamspirit
11 replies
21h38m

Isn’t this what makes cloudflare’s R2 a great proposition? No egress costs with very competitive storage and operation pricing.

It’s not as mature as the others, I don’t believe you can use your own encryption key for example, but why would anyone starting out not choose R2? Hopefully they push the market in their direction.

rezonant
8 replies
20h55m

R2 is an insane value proposition. We calculated the cost of moving HLS video delivery to it and unless you are delivering a huge amount of traffic, the costs are far lower. Highly recommend you consider it, assuming you are willing/able to handle appropriate transcoding yourself (if not, Cloudflare has it's Stream product)

deskamess
7 replies
20h2m

What is 'appropriate transcoding'?

0x6c6f6c
4 replies
17h44m

Assumedly, implementing the logic yourself for transcoding source content into various encodings on demand.

If you're storing 1080p h265 MP4s in R2 and the client can only support 480p h264 it'll request that but you need to "appropriately transcode" for them.

The alternative is a service that does this for you, like Cloudflare Stream. Upload source video, service handles transcoding and integrates easily with standard video players with HLS or Dash.

Just in case: I am not OP

ksarw
2 replies
10h55m

I don't think Cloudflare's free egress covers video storage and retrieval

https://community.cloudflare.com/t/can-we-serve-video-with-r...

Please correct me if I am mistaken; also R2 is not a CDN, and more like s3 in terms of delivering from the edge

No issue with your comment specifically, just wondering if you know. currently using s3+cloudfront for mp4 storage+delivery, and would like to move to something better if possible.

rezonant
0 replies
5h54m

Please correct me if I am mistaken; also R2 is not a CDN, and more like s3 in terms of delivering from the edge

Yes, it uses Cloudflare's edge network and caching infrastructure when you use a custom domain (really the only option because usage of the default domain is very limited). So yes, it's a CDN :-)

https://community.cloudflare.com/t/cdn-support-for-r2/514573

currently using s3+cloudfront for mp4 storage+delivery, and would like to move to something better if possible.

What I have found is that (A) the egress costs are extreme compared to R2 (remember, R2 does charge per read/write though) but (B) storage is cheaper on S3. You should do a study on your storage and egress, but unless you are storing way more than your egress TB, it's probably a good deal.

Cloudflare also has a mechanism for transparently pulling content from an upstream S3 compatible store into R2. I think it's called Super Slurper iirc

goestoo
0 replies
9h15m

Their terms say https://www.cloudflare.com/service-specific-terms-applicatio...

  Unless you are an Enterprise customer, Cloudflare offers specific Paid Services (e.g., the Developer Platform, Images, and Stream) that you must use in order to serve video and other large files via the CDN
https://www.cloudflare.com/service-specific-terms-developer-...

  The Cloudflare Developer Platform consists of the following Services: (i) Cloudflare Workers, a Service that permits developers to deploy and run encapsulated versions of their proprietary software source code (each a “Workers Script”) on Cloudflare’s edge servers; (ii) Cloudflare Pages, a JAMstack platform for frontend developers to collaborate and deploy websites; (iii) Cloudflare Queues, a managed message queuing service; (iv) Workers KV, D1, Durable Objects, Vectorize, Hyperdrive, and R2, storage offerings used to serve HTML and non-HTML content; and (v) Workers AI, a Service that allows customers to use Cloudflare’s inference infrastructure to invoke select third-party machine learning models (subject to applicable open source licenses or third-party terms of use for such models).
Based on their terms, serving video in R2 should be fine.

rezonant
0 replies
13h44m

This exactly, although it's exceedingly rare to do it on demand. Typically you do it when the video is first uploaded.

That, and it's usually HLS ABR, more about that on my parent comment.

rezonant
1 replies
13h43m

Video transcoding: Taking the raw uploaded video and re-encoding it so it is appropriate for being sent to video players over the web.

Typically this takes the form of creating "renditions" of the source video at lower resolutions, but you'll have to decode and re-encode the video to do that, hence transcoding.

Pretty much all video you watch online is delivered using an adaptive bitrate technology like HLS or DASH. That's what allows the video to automatically reduce its resolution if your network can't handle streaming the video without buffering.

deskamess
0 replies
6h25m

Thanks. Our use case is simpler so we only have one variant that gets uploaded and served. However we do not have any associated .m3u8 files, so that was my concern.

shdh
0 replies
21h35m

R2 works great from my experience, no egress costs are a huge sell.

l5870uoo9y
0 replies
21h28m

I offloaded all heavy files (videos and GIFs) to R2 and serve them on my web app. I reckon I save at least $240/yearly (12*$20) compared to serving them through Vercel. I didn't expect mere static files being such a cost factor.

njitbew
9 replies
19h38m

Oracle Cloud only charges a fraction of want Google, Microsoft, and Amazon charge. Any idea how Oracle is able to keep the cost so low? Or are the others just inflating the price so customers don’t move to the competitor? In that case Oracle deserves a shout out for not applying these vendor lock-in practices.

pier25
1 replies
18h1m

Egress is an artificial cost. Cloud providers don't pay for it.

Culonavirus
0 replies
4h31m

Well I'm sure there are some costs, but Google charging you an arm and a leg for traffic when they literally own multiple sea cables going around the world and a bazillion datacenters seems a bit sus...

wongarsu
0 replies
19h12m

Hetzner charges $1/TB. Oracle charges $7/TB, AWS $90/TB.

Oracle has probably really good margins on egress costs. With AWS/GCP/Azure the costs are absurd because for a lot of their customers it's not a big cost during operation, but makes moving data off cost prohibitive. It's simply a vendor lock-in mechanism for them.

vlovich123
0 replies
19h29m

Egress prices have insane levels of markup.

suyash
0 replies
19h26m

Other are just charging what they are because people are too ignorant to complain/object and will just pay.

shell_game
0 replies
6h9m

Oracle has other lock in practices lol… they don’t need to jack up egress fees.

hacker_newz
0 replies
18h39m

Oracle does not deserve a shout out for charging market rate for their service.

dilyevsky
0 replies
19h10m

The margin on egress is insane. Oracle got into cloud late in the game and burned a lot of goodwill downmarket so they have to sacrifice that to play catchup

andersa
0 replies
19h32m

The large providers are overcharging between 10 and 100x on egress. Cloudflare has a blog on it somewhere.

naiv
8 replies
21h52m

I think the overview is missing a column of what data is actually allowed for 'free' egress.

Iirc eg Cloudflare does not allow videos etc. as part of their free tier.

edit: confused ingress and egress

kkielhofner
3 replies
21h37m

We’re talking about egress - traffic leaving the network. Due to traffic patterns and other reasons (Hotel California) ingress is always free, egress is charged.

There is some debate about the status of hosting video when using Cloudflare as a CDN.

These comparisons are for hosts/object store.

Cloudflare R2 has no restrictions on the type of content while not charging for egress.

avereveard
1 replies
16h4m

R2 has some reasonable bandwidth usage in their terms of services and I don't see very large enterprises willing to play suspension roulette against unknown limits

weird-eye-issue
0 replies
9h27m

I think you're confused

naiv
0 replies
21h7m

sorry , I typed too fast and meant egress

asmor
2 replies
21h35m

Hetzner has seen two known cases of people being evicted for bandwidth use, and they were saturating the link 24/7 for several months by syncing an endless amount of footage to archive.

Aeolun
1 replies
17h43m

Do you have a source for this? Would like to read about that.

stephenr
0 replies
3h14m

Another factor when it's a CDN is what they will/won't allow to be cached (and thus moving the point of mass-egress from your origin to their network).

Cloudflare has limits on the size of objects that can be cached, making it relatively impractical to use for a download-focused site with resources more than a few hundred MB.

zerr198
6 replies
19h10m

We moved to AWS a few years ago, after self-hosting our own storage for years. What a nightmare.. The egress fees were killing us, so we had to find something else. Being weary of these large tech companies, and being that these days we're more privacy conscious as well. We found a new startup that outsources storage nodes to individuals / businesses. You have to install what they call a "relayer" on prem. The relayer is the S3 endpoint. It breaks up the data and encrypts it before sending each piece to a different node on the network. No centralized behemoth holding us hostage over our data and no egress fees. We're a smaller company, so I was able to talk them into trying it out. I doubt you'd be able to talk an enterprise into using a lesser-known technology though.

narimoney
1 replies
17h11m

what is the name of this startup? So i can read about it?

zerr198
0 replies
17h1m
chrismartin
1 replies
18h50m

So, you went with Pied Piper? :)

zerr198
0 replies
17h47m

hahah, yes, in a sense :)

Beefin
1 replies
5h52m

is AWS Outpost not an option?

zerr198
0 replies
3h2m

Was not familiar with AWS Outpost and honestly just want to get away from the monopolistic tech companies. Though I'm curious. Isn't this product just a way to interface with the storage as you would if it were stored on AWS servers? We would be storing the data locally?

mholt
6 replies
21h46m

This website is missing Wasabi [0], which has free egress. (Not necessarily endorsing, just noticed.)

[0]: https://wasabi.com

naiv
2 replies
21h5m

They have free egress up to your data storage usage.

So if you use 1TB storage but 2TB egress , it will not be free anymore.

mholt
1 replies
21h3m

Right; Backblaze has similar pricing as you can see on the table.

dessimus
0 replies
15h47m

Backblaze let's you choose to pay more to exceed said cap, Wasabi's Pricing FAQ says customers that exceed the amount they are storing "are not a good fit" and that they reserve the right to rate-limit or cancel the account entirely.

asmor
2 replies
21h30m

heavily recommending against wasabi, they increased prices recently and announced this via their newsletter, the one with an unsubscribe button, and then blamed me for unsubscribing (and never refunded the charge). highly unprofessional and a lack of understanding of the GDPR for someone who is supposed to hold all your data.

ijhuygft776
1 replies
19h55m

Do a credit card charge back and unsubscribe from the service (if they let you unsubscribe, lol)... If they don't let you unsubscribe, report your card stolen (I do that once in a while).

asmor
0 replies
11h17m

It wasn't worth fighting them over. And posting once on HN about the experience does a lot more to incentivize them to be better anyway.

mocamoca
4 replies
21h40m

OVHCloud is free https://us.ovhcloud.com/public-cloud/faq/

As a data company we really benefit from Scaleway, Hetzner and Ovh.

However, Scale way has no plans to add an US infra

And I don't know if Hetzner has US présence?

raid2000
2 replies
20h45m

European providers benefit from lower cross-connect fees in datacenters and more internet exchanges for easy peering. It's not surprising they offer more bandwidth at the same cost.

toast0
1 replies
17h27m

The US has plenty of internet exchanges. And plenty of smaller hosters with cheap bandwidth. Usually located near an internet exchange.

It's not like AWS or GCP have locations that are terribly far from major exchanges either.

raid2000
0 replies
8h56m

You still have to pay for cross-connects to connect to exchanges in datacenters.

In North America, the median cross-connect price is $300, more than five times higher than $58, which is the median price in Europe.[1]

Of course, that difference means nothing to a hyperscaler, but it's led to a difference in pricing culture between the smaller providers.

1: https://www.fiercetelecom.com/telecom/study-u-s-fiber-cross-...

reaperman
0 replies
21h39m

Hetzner has a limited US presence for VM products but not baremetal/dedicated solutions.

zokier
3 replies
19h28m

I wonder what would be the implications of AWS egress traffic increasing significantly (like order of magnitude). It is easy to rant about costs when you do not need to consider the consequences. Obviously AWS would need to build up their own network capacity, that's the easy part. More importantly they would need build up interchange infra too. This is the point I struggle to even imagine what the scale is like. Is it just plopping in couple of new 100G ports, or would that be just a drop in the ocean and in reality its more like couple of racks of network gear. Then there is the question how the ISPs/peers AWS is connected to are able to handle the traffic; AWS is not able to just unilaterally increase their egress capacity, the traffic needs to go somewhere. Besides technical questions, such increase might also cause renegotiations for peering agreements especially if the traffic balance shifts dramatically. And so on.

This got me also thinking, would operating some sort of spot market for traffic make sense to AWS? I imagine for them the main expence is maintaining capacity and that to me implies that optimizing off-peak use could be beneficial.

wmf
1 replies
18h35m

If AWS has massive diseconomies of scale that calls into question the entire model of public cloud.

jayveeone
0 replies
11h8m

What public cloud?

toast0
0 replies
16h51m

This got me also thinking, would operating some sort of spot market for traffic make sense to AWS?

Probably not, except in exceptional cases. I don't know how AWS manages their networks, but chances are they have a lot of spare capacity most of the time. And likely some major customers with location flexibility.

When I was a #1 customer of a different cloud, they had a fiber cut between one of their buildings and their IX PoP, and lost a sizable amount of capacity until the fiber was repaired. They asked us to move a ton of traffic, and we asked 'why us?' and the answer was basically if we moved that much traffic, none of their other customers needed to move anything, and we had the ability to move traffic quickly (and we had our lower bandwidth core service in the affected buildings that really couldn't move). I don't know how much Amazon retail and Amazon prime streaming use, but it could be significant and is almost certainly easy to move.

thepaulmcbride
3 replies
21h15m

Offt, not a good look for Railway, Render or Vercel!

paulgb
1 replies
19h42m

I’m a fan of all three, but my mindset is that I’m not just paying for raw infrastructure, I’m paying for the engineering work I don’t have to do when I use them.

thepaulmcbride
0 replies
19h13m

I’m a fan of them too. Their pricing gives me a “we will punish your success” vibe. I suppose that’s preferable to paying through the nose for not scaling.

mrkurt
0 replies
15h41m

They all need to make margins on top of aws/gcp expenses. Its just a knock on effect from such expensive public cloud egress.

jupp0r
3 replies
21h47m

I'm confused by that website using $/mo as the unit to describe the cost of 1TB egress. How does the time component play into this? I would have expected the cost to be described in $/TB.

amzans
1 replies
21h0m

Thanks, that makes sense. I'll update this so it's clearer.

jupp0r
0 replies
18h8m

Awesome, thanks and sorry for the pedantism.

mocamoca
0 replies
21h35m

I got confused too at first.

But it makes sense because the counters get reset each month.

So if you consume "101TB" over a year, distributed as 90TB in January and then 1TB per month... You could get hit pretty hard in January and have free bandwidth for the rest of the year.

cj
3 replies
21h26m

Cloudflare -- Free

Perhaps on self-serve plans with insignificant traffic. On enterprise plans they do set bandwidth caps. Our company exceeded our bandwidth cap and our account rep reached out to reprice our contract.

I asked him if turning off Argo (a service billed by bandwidth) would remove the bandwidth overage and they said no.

TlDR: Cloudflare bandwidth isn’t free at the enterprise tier.

tootie
0 replies
20h41m

Their cap is very high though. 1TB/mo would cover a lot of small to mid-size businesses depending on their use case.

rezonant
0 replies
20h52m

Cloudflare's enterprise offerings are bizarre. We went through the process to get it priced out and it made no sense for us. I could see how some of the features would be worth it for some businesses, but not for us.

naiv
0 replies
20h51m

I assume they internally have different quality levels of traffic so Enterprise would get better (and more expensive) routing?

8organicbits
3 replies
11h24m

One trick to watch out for: a cloud provider offering inexpensive egress can still make high-egress expensive by controlling the network speed of their instances. Several vendors require you to scale your whole instance if you saturate any of the components (like needing to upgrade from the 2 CPU to the 4 CPU instance to increase the network throughput, even though you don't need more CPU). On paper egress looks cheap and the instances look well priced for the hardware you get, but if you workload doesn't need balanced a balanced instance (1 CPU / 1 GB RAM / etc.) the pricing can get wonky.

foofie
1 replies
11h18m

Several vendors require you to scale your whole instance if you saturate any of the components (...)

Interesting. Which cloud providers do you have in mind?

PuffinBlue
0 replies
5h31m

The GP might be referring to how most of the non-hyperscalers like Linode or DigitalOcean or Vultr tie total monthly egress to instance size. IIRC they do allow overage charges at $x per GB but it quickly becomes more expensive to do it that way.

Mortiffer
0 replies
9h29m

Like scaleway

zmmmmm
2 replies
15h56m

I'm curious what people would estimate is an actual reasonable cost for egress? As in, what it costs to provide that service plus a sensible margin?

It seems to me that eventually it will come to pass that competition regulators start to look at this. For now there's enough of an appearance of healthy competition that they probably aren't interested. But below the surface what they should be understanding is that these are highly anticompetitive mechanisms being deployed and the complexity of unwinding it later on may be such that intervention is justified much sooner than would ordinarily be the case.

spacesloth7
0 replies
11h55m

fly.io has an article in which they (very transparently) say that they mark it up 3x, which means that Vultr/DigitalOcean’s 1c/gb is likely very close to the true cost

https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/

sammy2255
0 replies
15h23m
yakobwood
2 replies
18h30m

Google announced Jan 11th they’re eliminating data transfer fees when migrating from Google Cloud: https://cloud.google.com/blog/products/networking/eliminatin...

lolinder
0 replies
18h7m

This article isn't really about migration costs, it's about the cost of just sending data out into the wider internet (outside your cloud provider's data center). GCP is still very much charging for egress, they just now allow you to apply to get those fees waived if you are migrating off of GCP.

dns_snek
0 replies
6h35m

They're not eliminating data transfer fees, they've added a process through which you can apply to have your egress fees eliminated, subject to their approval. That's an important distinction when it comes to big tech.

redox99
2 replies
7h49m

I use Oracle Cloud only because of their cheap egress + lots of regions. Hetzner and similar are cheap but don't have enough regions.

otmahthe
1 replies
4h46m

What do you think about their services overall ? Do they have serverless computin too ?

redox99
0 replies
2h55m

Worse than AWS for sure. I only use compute and block storage.

Absurdly good free tier[1], much better pricing, and I like the customizable "flex" VMs instead of hardcoded VM sizes.

[1] Both the true free tier in which you won't get charged anything if you mess up, and the "always free" stuff in the normal pay as you go scheme.

rambambram
2 replies
7h32m

Let's say you pay a lot for some cloud provider and you find out that your app is best and cheapest served by a good ol' regular webhosting provider. Please, in that case, make sure you leave your money and data at independent webhosts. There's a trend going on - at least in Europe - of consolidating smaller webhosts into big players, under the guidance of some big groups (private equity?). They quickly raise their prices by 100's of percents.

I experienced this a couple of times with different webhosts (in the Netherlands) over the last three to four years. Very recently my monthly bill at one of them went up from around 3,- to around 18,-.

Whenever I read about 'egress costs' I usually laugh about that because I don't even understand it; my data is my data, why pay to get it off your hard drive!? But for some time I'm also seeing dark clouds for my own use cases. I don't find it hard to believe anymore that webhost after webhost is consolidating, trying to play AWS, upping the prices a lot, and finally... also establishing egress costs!??

skrebbel
0 replies
7h18m

Yeah wow huh. I had my wife’s food blog (wordpress) hosted at a Dutch provider called Neostrada. I think it was something like 50 euros per year at the start. They got bought, sent around emails with sleazy wording like “we’re upgrading your plan to fit our new pricing scheme” and then within ~2 years I ended up owing them 450 euros a year. I’ve never cancelled anything this quickly before.

A 9x price hike in just a few years has got to be the least scrupulous PE move I’ve ever witnessed in NL. I agree with your prediction that it’s just a matter of time before they tack on exorbitant egress fees.

Fripplebubby
0 replies
2h28m

If you believe that egress prices are fake and pure profit, I encourage you (or someone else reading this with the same belief) to try running your own independent web hosting service ;)

I think what you will discover is that network bandwidth is a finite resource you have to pay for and ration out to your customers, and that the easiest way to finitize and ration something your customers expect (errantly) to be infinite is to charge for it, and charge exactly that price that will cap demand!

jintrosartro
2 replies
20h13m
lolinder
0 replies
19h58m

That seems to be specifically about the transfer fees that kick in when you're leaving GCP. It looks like there's an application process to get the free egress for moving off the cloud, so they haven't just done away with egress costs altogether.

This article is about egress costs for just operating a service that talks to the outside world, not just when migrating data to a new service.

clhodapp
0 replies
19h59m

That's probably how they wanted you to read it, but in fact it's only free if you are transferring it out and shutting down your account

bcaxis
2 replies
21h48m

Nice chart

Some vendors let you pool your egress that comes with the compute you buy. Others do not. That can impact costs quite a bit.

rezonant
1 replies
20h51m

Right, Linode works this way for one.

geraldcombs
0 replies
18h9m

So does DigitalOcean. It'd be nice if the page specified which vendors do and don't.

alanfranz
2 replies
11h25m

We should also remember that, for google cloud, if you’d like to migrate away you actually can without paying egress costs:

https://cloud.google.com/blog/products/networking/eliminatin...

This doesn’t change business as usual egress fees, it’s likely a stopgap to prevent being accused of anticompetitive practices.

lannisterstark
0 replies
11h16m

Doesn't backblaze also just send you a hard drive with your data on it as long as you "pre-pay" for it and then refunds you when you return it?

dns_snek
0 replies
6h38m

Correction, you can apply to exit Google cloud without paying egress costs, subject to their approval.

speedgoose
1 replies
12h23m

I work in academia and Microsoft Azure doesn’t bill us egress up to 15% of the total bill [0].

Well, I still use Hetzner for the simpler needs because it’s much cheaper.

I wonder if the two other big cloud providers could also significantly reduce the egress fees when asked nicely.

[0]: https://azure.microsoft.com/en-us/blog/azure-egress-fee-waiv...

yread
0 replies
10h29m

In EU academia has some kind of deal that they almost never pay for egress. Some use ExpressRoute where egress is free, too

shdh
1 replies
21h38m

CloudFlare is great

ijhuygft776
0 replies
21h34m

not in this universe

richardw
1 replies
16h8m

Interesting that Google started on a system cobbled together from cheap servers, creating reliability out of unreliable components and against the Sun’s of the time. Now all the cloud vendors are effectively Sun.

quickthrower2
0 replies
14h32m

Google was/is an infrastructure company (though one lens, of course it is also an ad company). Just like Amazon is a warehouse company.

pxeger1
1 replies
10h3m

It’s worth noting that, while the big 3 providers do gouge for egress, they typically also have superior network performance and reliability. This is especially true when comparing against budget options like Hetzner.

DeathArrow
0 replies
9h36m

Does Google offer 100x the network performance of Hetzner? Because egress price is 100x more.

msravi
1 replies
11h26m

Hetzner with its ARM offering of 2xvCPU, 4GB RAM, 40GB Storage, 20TB egress at EUR 3.79/mo is unbeatable value for a small-medium website.

severino
0 replies
3h30m

The only downside is that Hetzner does not allow you to set a limit to control the egress traffic expenses, does it? So that you can be protected against the so called "denial-of-wallet" attacks.

marban
1 replies
10h30m

It says free for OVH but on their website for S3 I see: Outgoing public traffic $0.011 /GB

What am I missing?

Yujf
0 replies
9h39m

With a VPS you get free egress

breck-fresen
1 replies
17h33m

One thing all the posts like this miss is you don't pay sticker price at modest scale. And if you're a whale who can move your workload around to generate competitive bids, you can negotiate prices down to cost + some fixed margin.

The big guys also forecast their usage and make long-term commitments, which will yield savings beyond the advertised (e.g.) 3 year reserved instance pricing. Forecasting demand is a foreign concept for most cloud users, and elasticity has real cost in the form of reduced utilization.

That said, as a small fish, I find the egress costs obscene.

Source: I used to work on AWS

chatmasta
0 replies
16h33m

What is "at cost?" The cost to AWS is $/gbps (size of pipe), but they charge customers $/gb (stuff through the pipe). So if they're still charging you on the dimension of $/gb, you're probably getting ripped off no matter what they say about their "cost."

web3-is-a-scam
0 replies
20h1m

AWS egress is one of if not the biggest costs on our bill, it’s downright criminal.

up2isomorphism
0 replies
16h17m

I use to this kind of comparison to show the cost impact on public cloud, but the fact is that when business planning guy see this it becomes an evidence for cost saving to them: since these numbers, while looks expensive to engineers but they are nothing to them, and they can hire less people. I think it is helpful for typical engineering people to know more about corporate IT cost structure.

tomcam
0 replies
17h49m

Question from ignorance, not expertise: many many sites are database-driven, so why not just just back up each insert locally?

tbragin
0 replies
3h41m

Cloud network egress costs is one of the reasons why increasingly infrastructure vendors likely to incur high data transfer costs (database / data warehouse services, message buses, machine learning, etc..) are moving to introduce BYOC or "Bring Your Own Cloud" deployment model (Data Plane runs in the customer VPC - original Databricks deployment model), instead of only pure SaaS (Data Plane runs in the vendor VPC - e.g. Snowflake) in their cloud offerings.

Databricks now has fully hosted too, but Snowflake still did not go the other way to introduce BYOC. Newer entrants, however - RedPanda, AnyScale, ClickHouse... - are going the way of providing both BYOC and pure SaaS options in their cloud. There are many other reasons to prefer BYOC, legal, security, data privacy, but network egress costs is one of them.

stephenr
0 replies
6h34m

This has brought BunnyCDN to my attention as a possible replacement for Cloudflare on a download-heavy site.

Can anyone with experience/knowledge comment on the finer details of Bunny in terms of what can be cached (not worried about default rules more about hard limits) eg Cloudflare has size limits we regularly exceed, meaning the largest downloads all end up hitting the origin anyway.

I've looked at pricing for a bunch of other CDNs but they'd be hundreds to thousands of dollars per month for the amount of bandwidth we need (2-4TB/month), but it seems like Bunny might actually be cost effective, if they allow us to cache larger objects, even if it's for a shorter period.

sosodev
0 replies
12h40m

Worth noting that bunny CDN's volume pricing is $5/TB with further decreases as the volume increases.

sneak
0 replies
12h45m

Related: is there something like https://diskprices.com for VPSes?

I have a storage-heavy egress-heavy app that I need to run that I am trying to find a cost effective VPS for (ideally in a specific region). Most VPSes have little storage and egress caps.

ksec
0 replies
16h38m

Bandwidth Costs Around the World ( 2016 ). [1], I am wondering Cloudflare would get another update on it. How Bandwidth cost has come down over the past 8 years. And compare to Cloud Egress, which obviously haven't moved much at all. And the cost per port per Gbps over time.

[1] https://blog.cloudflare.com/bandwidth-costs-around-the-world...

jksmith
0 replies
12h2m

So let's assume also that their costs are dropping as well due to the deflationary nature of hardware costs. Seems like a pretty damn good business model to me. All they have to do now is get you hooked so you can't leave.

jedberg
0 replies
19h22m

There are many reasons the Netflix uses AWS for everything except serving the actual video bits, but this is the main one. Even before they built their own CDN they used other providers with much cheaper egress.

hiAndrewQuinn
0 replies
10h26m

Making it cheap for productive people to enter, and costly to exit, is a time honored strategy by countries and businesses alike. There is nothing particularly surprising going on here.

godzillabrennus
0 replies
11h2m

Netlify is trying hard to make their platform unaffordable for everyone.

dijit
0 replies
21h48m

For the google cloud one, is that the premium networking option or the standard networking?

Those are very different, the standard network is more akin to AWSs egress as GCPs Premium option (the default) uses local PoPs and dark fibre over Googles backbone to origin.

cheema33
0 replies
19h26m

We have used OVH for years. We were on MS Azure before that. Getting nickled and dimed for every little thing drove us away. Zero regrets.

charcircuit
0 replies
21h12m

AWS's egress needs to decrease by 100x before it will make sense for me to use them.

brycelarkin
0 replies
11h59m

The recommended way to do egress from AWS is through Cloudfront, which provides 1TB of free egress instead of 100GB as stated in the chart.

Regardless, I feel like AWS is price gouging for egress.

andix
0 replies
18h12m

I would be interested how much discount on the egress fees is possible for big customers. Especially if you're running a service that is heavy on egress, for example something with video processing that also exports/streams videos. Or are those companies just doomed not to use the big three cloud providers?

Those egress prices are not at all about the cost of the traffic, it's more like an additional hidden fee. And a vendor lock in not to use multiple cloud providers.

amenghra
0 replies
21h42m

some cloud providers offer a certain amount of free egress each month, while others charge a flat fee for egress over a certain amount I parse this sentance as saying the same thing twice.

PaywallBuster
0 replies
13h15m

Bunny have the "Volume" package: starts at 5$/TB, could go down to 2$ depending on volume

Mortiffer
0 replies
9h30m

Pls also consider the egress network speed. I got burned by scaleway when they changed their blob store pricing. Trying to my grade out to Hetzer would have taken months at the network speeds they were providing

DeathArrow
0 replies
9h47m

Once you buy in a cloud, both you and your data become hostages. And you have to pay ransom monthly.

DeathArrow
0 replies
9h31m

"Relax, " said the night man "We are programmed to receive You can check out any time you like But you can never leave!"