Ask HN: Video streaming is expensive yet YouTube "seems" to do it for free. How?

Disclaimer: I used to work at a live video streaming company as a financial analyst so quite familiar with this

The biggest cost is as you imagine the streaming - getting the video to the viewer. It was a large part of our variable cost and we had a (literal) mad genius dev ops person holed up in his own office cave that managed the whole operation.

Ive long forgotten the special optimizations he did but he would keep finding ways to improve margin / efficiency.

Encoding is a cost but I don’t recall it being significant

Storage isnt generally expensive. Think about how cheap you as a consumer can go get 2 TB of storage, and extrapolate.

The other big expense - people! All those engineers to build back and front end systems. That’s what ruined us - too many people were needed and not enough money coming in so we were burning cash.

For the streaming factor specifically, I can at least offer something resembling an answer: Google. In the early 2000s they bought up a bunch of dark fiber and peered with all the major US ISPs, and they were able to do this because no ISP wants to be the one that blocks or degrades Google. As a result they were able to host video streaming on their network without immediately being shut down by Comcast and co. Instead they had to go after Netflix.

Google has a lot of custom encoding silicon, too, AFAIK.

Storage is the biggest question of the three. Linus Sebastian specifically called this out when YouTube started really pushing to make the non-Premium experience dreadful. There isn't really some secret special sauce you can buy or make for storage. Literally everything is being stored with the same hard drives, SSDs, discs, or tapes you can just go out and buy. The only specialization you can do is build or buy equipment to handle extreme numbers of them. Google does buy these in bulk, so they probably get a discount on storage, but it's not something that would make storage costs just go away.

YouTube generated $31.5 billion in advertising revenue in 2023.

https://www.businessofapps.com/data/youtube-statistics/#:~:t....

That's... a lot! Plenty of historical precedent for fully advertising-supported media with high expenses, from OTA network television and radio to free weekly newspapers... or inexpensive subscriptions to daily newspapers subsidized by advertising. Advertising has been paying the bills for electronic media for a century now.

For streaming, google have caches in like every ISP network. Also majority of the people watches the latest and same type of content that is mainly served from the homepage, which is easier to cache and serve.

If you have the ipvfoo extension, you can see it in action. (its easier to see with IPv6)

https://github.com/pmarks-net/ipvfoo

Disclaimer: I worked at Google and occasionally did things for YT, but it was 2015 and before. I did look at their P&L, somewhat.

egress costs were enormous and YT was not profitable. I don't know if it is now, but I wouldn't be surprised to find it is. They sure have enough ads.

As several people say below, caching content around the world is key, so that not all requests are serviced in NoCal.

Is there any way that I can pay YouTube so that the videos I post do not have ads when people view them for free? I think there is a clear answer to this question for video, and it’s ~$60/month.

It's purely a scale problem.

On the "Revenue" side you will quite probably need to have enough eyeballs that advertisers come to you directly to display ads and do so in volume.

On the "Costs" side you'd want to be big enough that you can just store your content in ~3 of your own datacentres, cache the "hot content" in a site or two per country then give away caches to ISPs (who will gladly host them in their own network for free).

Biggest cost will be bandwidth/streaming servers. Encoding/storage is comparatively cheap. If you were small you would likely start to do this from a few 100Gbps dedicated servers per continent. https://www.fdcservers.net/configurator?fixedFilter=15&fixed... If we set an average of 3-4Mbps per stream you're looking at each server handling 20,000 videos served and the hourly cost of the server would be around say $4/hour so you're looking at around $0.20 per 1000 video hours in theory, in practice it will be higher. Worst case closer to $0.50 per 1000 video hours due to utilisation rates.

It's fairly straightforward -- use your fountain of money from search ads plus a huge infrastructure to support it to pay the costs rather than pass them on to users.

Operate at a loss to drive out all competition and prevent new competition from arising. Increase ad obtrusiveness to drive up revenue, and every once in a while increase creator payouts to keep creators on your platform, until hopefully the lines cross and you start to make money. Maybe. Every once in a while a competitor might make a go for it, and you'll have to reduce ads or offer more incentives to creators to drive them out. Sometimes you may have to lobby the government to help you out on this.

One bonus is you can use the goodwill from your video site to drive traffic to your search engine.

If you don't have a fountain of infinite money from your search engine (or if you don't currently operate a gigantic search engine), then you might not be able to pull this off.

For most players streaming are not profitable. Youtube is likely profitable, but since Google doesn’t report separate income, we don’t know how much.

Netflix is.

Disney, peacock, Paramount, Max , etc are not profitable with the hope they can capture future monopoly standing.

Prime Video is likely also not profitable or break even given their studio investments (e.g. MGM, first party content).

They improve their stack on multiple fronts

1 - They use their own custom hardware encoders now optimised for their scale (previously they’d use a ton of intel cpu for encoding) This is often not possible for smaller startups to have their own custom hardware, but one could setup their own Xilinx FPGA based hardware accelerated encoding servers, Xillinx Alveo Boards are popular for media acceleration, they support both ffmpeg and gstreamer for acceleration needs. Nvidia Nvenc encoding is decent too but mostly for the most popular encoding types, and honestly an fpga based accelerator is probably better after cpu unless you have gpus lying around.

2 - Much lower bandwith costs, a lot of startups build their streaming applications on top of AWS, but that’s insanely expensive due to the insanely expensive price of AWS itself

As a streaming service you could immediately cut down costs by shifting to ovhcloud, hetzner etc, they often have dedicated gameserver like offerings which are beneficial for streaming usecases too. That itself should allow you to bring down costs by 10x or even more (you have to choose the plan correctly on other tho, ovh, hetzner etc say you have unlimited bandwith on some packages but there is always a catch or they throttle you later, need to be careful about the offering you choose but otherwise a lot better)

3 - Having colocation hosts with your own servers setup and each video encoded at different sizes for different media devices also help only sending the near-right amount of content quality based video size based on device screensize and resolution. This gives a lot of additional cost advantage

4 - Running your own CDN relay service, I’ve run my own cdn service before to cut down on costs aws cdn is quite expensive, but even cheaper cdn services like bunnycdn, etc come at a decent markup, cloudflare cdn can be nearly free yes, but they’ll kick you out the moment you start consuming too much bandwith. So I hosted my own anycast network with geodns and controlled multiple servers across the globe acting as cdn with ansible playbooks for maintenance and control, and monitored them with my own monitoring stack to keep the cdn service alive and kicking, it work decently nice honestly, yes it required me to learn additional stuff, but it made me even more happier, price for sufficient level of bandwith it essentially allowed me to get very cheap rates for cdn service, along with additional cost optimisations at the edge by pre-allocating more cdn servers at peak usage to relay the stream once to the edge server and then broadcasting it from there to all local devices snappily and then bringing down the number of servers, once a major stream is over.

That takes care of cost reduction in encoding, bandwith and services like cdn.

There are a 110 otherways to reduce costs, but those are the primary ones you want to invest in, I excluded storage because you can already get cheap enough storage from a lot of managed providers like wasabi, backblaze, etc moreover streaming typically is less focused on long term storage so short term storage cost optimisation should be last resort, using reliable storage is a better idea.

Here’s an additional talk by mike solomon (youtube engineering) on scalability from 12 yrs ago [5] also the written blog version of it is here [6], a lot of it is still relevant, especially for young startups because bigtech youtube scale decisions are often not rational for young startups who have 1-2 years runway and need to make the most out of vc money or heck even bootstrapped haha (I was bootstrapped so price was a big deal for me, hence had to do a lot of stuff custom).

While making decisions always make sure to ask yourself at each step, “is this move relevant for my business” at youtube scale where their bandwith costs are in billions, hiring a $30 million hardware acceleration team makes sense to them, it does not to your startup, always take tips and hints from large behemoths like youtube, instagram, etc who’s devs love showing how they do a lot of things in talks available across internet but never forget to ask yourself the critical question of if its relevant to me.

A lot of the optimisations i did like custom cdn, managing my own storage, algorithms to allocate edge servers to reduce central load while boosting reliability, low latency and also reducing costs, does require very enthusiastic developers who love to learn things and are decently competent about it too. Each optimisation might give you 5-7% boost, some big ones will give you 200-500% boosts, but even those small changes add up to a big cost reduction overtime, if you get a big bill initially for your streaming service, dont get disheartened, allocate a team that specifically focuses on this and has deep tech knowledge doing 7%-10% optimisation each month across parts of the stack, and compound interest will help you move mountains.

Good luck to everyone here who are working on their streaming startups !

[1](https://www.semianalysis.com/p/google-new-custom-silicon-rep...) [5](https://www.youtube.com/watch?v=G-lGCC4KKok) [6](https://highscalability.com/7-years-of-youtube-scalability-l...)

The economics fundamentally changed a couple of years ago when cloudflare released R2. I don't know if anyone has built a streaming client for it yet, but R2 takes the largest expense (outgoing bandwidth) and zeroes it out.

Yor business would be wholly dependent on cloudflare. But if you don't have to pay for bandwidth, the economics aren't that bad.

(I ran a large porn site a lifetime ago, long before cdns were ubiquitous. If I was in the business today, I would absolutely put everything on R2 and make it work no matter how much client development it took)

youtube is the modern nanny and that is the most valuable ad space on the planet

It probably costs them a ton of money, but they probably make a ton more so that’s OK.

Also: bandwidth gets a lot cheaper when you own the pipes.

And one that is less obvious: despite having hundreds of millions of videos available, a large contingent of people are watching the same ultra-popular ones. There are some economies of scale to be had there.

Google has huge interconnect and exchange deals and dark fiber all over the place. It’s hard to explain just how much free transit and connect Google has to ISPs, and how preferable their deals are for delivery. They are a MASSIVE pipe of extremely low cost Internet.

Since the main cost of video is streaming and the pipe required to deliver that streaming, they have an insane moat built because of it.

That is why. There are a lot of other explanations, but that is really why.

Bandwidth/Transport costs for live streaming, especially in group conference call scenarios (where N streams needs to be broadcasted to N-1 participants) become prohibitively expensive after about 8 or so participants unless you can offload those bandwidth requirements to other places (e.g., like in a peer-to-peer architecture).

How Zoom manages to do this in a client-server fashion and is still financially solvent is also a question I've had for a while, but like others say, discounts on the transport and peering arrangements will be a key part in making those economics work, as compression and storage are relatively solved problems here.

Video streaming is expensive for you. Google is incredible at making things cheap that we all thought were expensive.

Fingers crossed you are starting a competitor to youtube.

Storage, transcoding/encoding, and any other compute operations (rendering, etc) are small compared to data transfer costs.

At the scale of the largest streaming apps (Disney, Netflix, YouTube, etc) you are moving petabytes of data PER DAY. At that size, you have access to significant savings on CDNs, backbone providers, etc. in many cases the discounts will be 90% - I have seen as high as 99% - or higher off the “list” price (which are usually never paid by anyone anyway).

You also tend to own your own backbone and can link in whichever ISP wherever you want for the “final mile.”

Final note, when you have been doing this long enough, you can start shaping the traffic based off previous patterns. I remember an eBay listing years ago for a Netflix local storage device that was meant to store shows at an ISP’s data center.

The economics are simple:

I know advertisements are a thing for YT, but is it enough?

Yes, it is -- virtually certainly. We can assume YouTube is profitable. It's not broken out directly in quarterly reports, but it doesn't make any sense that Google would still be running it after all these years (almost 20) if it weren't.

But obviously YouTube didn't start out as profitable. You need scale, which provides two things:

1) Marginal storage and streaming costs go down (Google is big enough to save huge amounts of money by running its own data centers, peering agreements, caching near customers, etc.)

2) More advertisers running more ads that can be targeted to more users whose preferences you know more about

So no, you can't run it profitably.

This is a classic example of a business that is only profitable at scale, that needs to lose a lot of money at first as it grows until it achieves scale. And it's not just scale on the traditional tech/users side, it's scale on the advertising side as well -- advertisers aren't going to bother running ads on your platform until you have enough users for them to care.

It's also pretty strongly a "winner-takes-all" network effects situation, where video publishers want to put up their videos where the viewers are, and viewers want to visit the site where all the content is. So if you wanted to create a YT competitor, I don't know how you'd convince content creators to post their videos to your site in addition to YT, or how to convince consumers to watch said videos on your site instead of YT.

The value of YouTube isn't purely monetary. Controlling YouTube is extremely advantageous for Google, even if it isn't particularly profitable.

Besides the general advantage of having control over such a massive platform, which definitely plays an important part in the lives of hundreds of millions of people, Google likely views YouTube as important to control. If YouTube were a separate entity, it could e.g. freely choose their ad providers or even provides ads themselves, essentially creating competition for Google. Google also has trivial access to the data there and therefore the easiest access if they want to train AI on that data. Last but not least I think Google sees YouTube as vital for their corporate image and their social mission presented in Google Jigsaw.

Encoding is largely super-linear for a single stream, so you just need enough cores for the intake * formats. Streaming is mostly chunking and a smart player that loads the right chunks at the right time. Storage is bottom dollar, use whatever the cheapest disks you've got that you can attach to fiber, then cache the hell out of everything.

So in short, the only "on-demand" component is encoding, and if you don't have an 'available in an instant' promise, you can do it on spot instances on the cheapest cloud you can find; The rest is just storage and distribution - if you own a world-wide network of datacenters for your successful advertising service, that's kinda an already solved problem for you - just allocate a few racks to a new service.

I of course downplay everything and simplify massively - but at a high level, it's just a lot of ffmpeg -> S3 -> html5 player. The harder problems are in the long tail - high latency, content licensing & geo fencing, etc.

Source: used to SRE for a video streaming provider (not YT), also former GG

Streaming is expensive, but less than you think. If you run your own servers and don't use the cloud, you can serve a million user a month for a few thousand dollars.

Great question! YouTube's scale definitely plays a big role. With the massive user base, ad revenue can cover the costs. Plus, Google’s infrastructure helps reduce expenses. For a new platform, the economies of scale and established ad networks make a huge difference.

https://peering.google.com/

Economies of scale. The variable price for a new start up to stream a TB of video is orders of magnitudes higher than for Google at this point and the ad revenue is massive now too. They're 11% of all video watched on a TV, that's ONLY counting TVS, and advertisers know this now and they are treated seriously.

YouTube is vertically integrated.

They're not paying a margin to advertising companies because they are the advertising company, they're not paying a margin to datacenters because they are the datacenter.

The data gleaned from YT views helps them to run search and vice versa.

Mullvad give you practically unlimited bandwidth around the world for €5/month.

And Hetzner will sell you 1TB of hot storage on a 1Gbps connection for €3.49/month.

Taking these consumer level price points, you just need to make more than €4/user month to cover your marginal costs. Quite a demanding, intensive user too.

I remember a lot of chatter around Google buying up fiber after many companies had aggressively built out networks in the "broadband" scramble, which then lay unused. So-called "dark fiber". 2005 article:

https://www.cnet.com/tech/tech-industry/google-wants-dark-fi...

Google bought Youtube the next year, 2006. This must have been massively useful for moving video around at lower cost rather than the public networks, which were probably built more for normal web traffic. Then peer with local companies who have the clients that are watching videos.

especially at scale

The marginal costs go down a bit if your scale is truly immense. Google can afford to design/manufacture/deploy hyper-efficient custom silicon ASICs for encoding. Also because their critical mass of users provide valuable network effects, they can get away with particularly poor quality encoding (IMHO) and the vast majority of users still won't switch to other platforms with higher visual quality - but other (non-pornographic) video platforms generally don't have that luxury.

It is expensive, and YouTube also makes a lot of money via ads, subscriptions, partnerships. Whether it is ultimately profitable or not is anyone's guess, since they don't report the numbers publicly.

YouTube made $30B in revenue last year. 10% of Alphabet’s revenue.

Google services margin is around 30%. Even if YT is burning money they are likely making $5B in profit. They don’t report profit by income streams.

Google was one of the first internet scale companies with likely 10s of millions of servers and fiber that they own around the world.

They are also doing quite well in Cloud. Not as well as Azure and AWS but that division is growing.

At this point, something like Tiktok with a better sticky algorithm is the way to beat Google.

One of my mistakes was thinking Facebook will be crushed by Google.

Plenty of blind spots when you’re a big ship like Google.

This seems like a valid and legitimate question, but I can't help but think that this question was posted, in a veiled manner, by Google/Youtube/Alphabet as propaganda.

Video streaming is getting cheaper all the time. Bandwidth costs are dropping every year (substantially in most cases) and bitrates aren't keeping up (the few exceptions probably being AppleTV+ and BBC iPlayer which do 30-50mbit/sec 4K HDR streams).

You can do this for so much cheaper than AWS etc price for bandwidth. You can get 100gigE transit from he.net for list price $4500/month. Add probably the same again for colo + hardware (don't need much hardware these days to saturate 100gigE) and you can probably stream videos to 20,000 concurrent users at 5mbit/sec for ~$10k/month.

Another way would be to use someone like OVH who offers dedicated servers with 10gigE (supposedly 'guaranteed') for about $800/month each list price, without having to bother with colo and ip transit setup.

Obviously this is highly simplified as you will require encoding resource and storage, but again with someone like OVH you'd be able to spin up a lot of cheap boxes to do this. How much this will cost will depend on how many videos you get and how many views per video etc.

So IMO the actual bandwidth is a bit of a non issue. The far bigger issue is getting users to use your platform (marketing is MUCH more expensive than IP transit) and then having advertisers on your site. This is a much harder problem to solve and where the real barrier to entry is.

They have what's called Google cache in every country. In some of these countries, and in many places, ISPs are allowed to connect to the cache for free, including at IXs, which are basically internet exchange hubs around the world. Google has its own fiber, so they do not pay anyone for transit in many places. Additionally, because so many people watch YouTube, ISPs are motivated to connect or peer with Google's caches on their own. This arrangement means they also don't need to pay for traffic. So streaming is extremely cost-effective for YouTube, making it hard for others to compete with them on streaming costs.

So bandwidth incurs the high costs. Are we in need of another year 2000 event, where the dot com bubble burst, sending bandwidth costs into he cellar?

You can make an MVP:

It doesn't support up or down scaling or recoding. Force the user to only upload supported file formats and codex.

Minimal hosting by forcing viewers to also upload: the WebTorrent protocol is ideal for this.

Why ads at all? Those already exist. It's an ideal now platform to try out micropayments. Look at http://value4value.io for example.

If this starts to roll, only then you scale.

they lose on the unit item, but make it up on the volume!

Youtube makes more than $ 30B in advertising alone, and that ignores Youtube premium and on-demand content.

Quite sure they are making lots of money.

DIY (instead of the cloud) is the answer. If you're pushing terabytes+ from day 1 on a shoestring, you're going to want your own CDN. If you can manage a queueing system and the occasional wait, run your own (or rented physical) hardware for transcoding at as high a utilisation as you can.

Build vs buy pushes you to "build" early on when your margins are slim and your volume is huge.

I know a popular website rolled out their own nginx proxy servers to various low cost providers so that they could run their own cdn. It saved them tons of money.

As someone who works in a large cloud company, there is a lot going on to create a data center. By data center I mean compute and storage. These large companies have perfected the economics and engineering needed to create a data center. They spend billions on R&D. They basically own everything in their supply chain. Smaller companies can't compete.

Additionally, they don't pay the same electricity and water bill that others pay for their data center. They get a discount because they are creating jobs.

Getting streaming to be cost effective starts from decades of R&D investment + getting low cost electricity and water + owning supply chain.

I worked on YouTube transcoding about 12 years ago. First, the scale is mostly reused - what’s doing transcoding now is doing a different compute job later. Transcoding was also done for most videos only on idle compute. Second, Google had 300k+ caches around the world, in many surprising places (buses, cruise ships) as well as many thousands of other larger but not full data center locations; get the content as close as possible to the user. (I imagine now all transcoding from the mezzanine format is done in real time on an edge GPU for all but the most popular platforms and content). Tl;dr: build out a huge amount of infrastructure to serve ads very quickly and you can piggyback video serving on that at little marginal cost.

It's not free. They didn't have a business model for most of their existence.

Google, famously shitty at branding (and that's being kind), paid an obscene $1.6B for YouTube because "Google Video" was a monumental failure. Of course it was: Everyone thought it was just a search engine for video, not something that you would contribute to... any more than an individual "contributed" to Google's search results.

So they rewarded an enterprise that had no business plan.

It sure doesn’t feel free. YouTube has cranked the ads up so frickin’ high that I swear they’re quietly in panic mode about profitability.

Every month I notice the temperature of the pot is up a few degrees. This month it’s unskippable 15 second ads before most videos. Last month it was the first search result now being an ad. Before that it was how 5 second ads are now 7 seconds.

If I thought to write them all down I’d have a dozen more steps to share.

My kids now call it “the Bad YouTube” vs. YouTube Kids because the former is flooded with ads.

Streaming, encoding, and storage demands enormous costs -- especially at scale

When you look at costs per unit, then it gets cheaper at scale, not more expensive.

For streaming, at scale you can afford to do peering yourself, instead of buying bandwidth.

For encoding, at scale you can afford special purpose encoding hardware, instead of using general purpose hardware.

For storage, at scale you can get cheap bulk deals with drive manufacturers.

Google has hosts in just about every ISP data center. They sent out their hardware to spam IT departments for decades.

Other CDN like akamai are available, but the cloud business has eaten most colo providers lunches (a "gray" fiber network can be expensive to hold).

Since Google has other services likely reusing the same internal CDN resources, their stream-traffic routing costs could be an order of magnitude lower than traditional providers.

Just a guess, but live feeds tend to be high-latency for a reason... =)

"If something is free, you are the product."

Can't someone use CDNs to deliver the bulk of the video files?

I know advertisements are a thing for YT, but is it enough?

Google uses search revenues to subsidize the cost of YT. It's anti-trust, but the government hasn't had the will to prosecute as of yet.

I think YT is finally self sufficient, but probably still relies on google's existing infrastructure to a large degree to keep costs down.