This is such a clever way of sampling, kudos to the authors. Back when I was at Pew we tried to map YouTube using random walks through the API's "related videos" endpoint and it seemed like we hit a saturation point after a year, but the magnitude described here suggests there's a quite a long tail that flies under the radar. Google started locking down the API almost immediately after we published our study, I'm glad to see folks still pursuing research with good old-fashioned scraping. Our analysis was at the channel level and focused only on popular ones but it's interesting how some of the figures on TubeStats are pretty close to what we found (e.g. language distribution): https://www.pewresearch.org/internet/2019/07/25/a-week-in-th...
This technique isn't new. Biologists use it to count the number of fish in a lake. (Catch 100 fish, tag them, wait a week, catch 100 fish again, count the number of tagged fishes in this batch)
Do you get the same 100 dumb fish?
Why are they dumb? Free tag.
Imagine being the only fish without a tag. Everyone at school will know how lame you are.
This comment. Please see here.
It would be illegal not to have a tag. If the fish has nothing to hide, it shouldn't worry about being tagged.
And, also, the fish gets tagged for its own good.
They'll even call you tinfoil fish.
Catching fish is theoretically not perfectly random (risk-averse fish are less likely to get selected/caught) but that's the best method in those circumstances and it's reasonable to argue that the effect is insignificant.
You make a very weak argument, and are simply assuming the conclusion.
What makes it the "best method"? Would it be better to use a seine, or a trap, or hook-and-line? How would we know if there are subpopulations that have different likelihood of capture by different methods?
To say it's "reasonable to argue that the effect is insignificant" is purely assertion. Why is it unreasonable to argue that a fish could learn from the first experience and be less likely to be captured a second time?
If what you mean is that it's better than a completely blind guess, then I'd agree. But it's not clearly the best method nor is it clearly unbiased.
Fair points. But, mark-recapture is about practicality. It's not perfect, but it's a solid compromise between accuracy and feasibility (so I mean best in these regards, to be 100% clear). Sure, different methods might skew results, but this technique is about getting a reliable estimate, not pinpoint accuracy. As for learning behavior in fish, that's considered in many studies (and many other things, like listed here: https://fishbio.com/fate-chance-encounters-mark-recapture-st... ), but overall, it doesn't hugely skew the population estimates. So, again, it's about what works best in the field, not in theory.
only if you're within a 100 mile radius of me the ultimate dumb fish
Wouldn't a previously caught fish be less likely to fall for the same trick a second time?
That's typically the Lincoln-Petersen Estimator. You can use this type of approach to estimate the number of bugs in your code too! If reviewer A catches 4 bugs, and reviewer B catches 5 bugs, with 2 being the same, then you can estimate there are 10 total bugs in the code (7 caught, 3 uncaught) based on the Lincoln-Petersen Estimator.
But this implies that all bugs are of equal likelihood of being found which I would highly doubt, no?
Yes, it's obviously not a perfect estimate, but can be directionally helpful.
You could bucket bugs into categories by severity or type and that might improve the estimate, as well.
Oh this is a really interesting concept.
I guess it underestimates the number hard to find bugs though since it assumes same likelyhood to be found.
A similar approach is “bebugging” or fault seeding: purposely adding bugs to measure the effectiveness of your testing and to estimate how many real bugs remain. (Just don’t forget to remove the seeded bugs!)
https://en.m.wikipedia.org/wiki/Bebugging
That's not actually the technique the authors are using. Catching 100 fish would be analogous to "sample 100 YouTube videos at random", but they don't have a direct method of doing so. Instead, they're guessing possible YouTube video links at random and seeing how many resolve to videos.
In the "100 fish" example, the formula for approximating the total number of fish is:
In their YouTube sampling method, the formula for approximating the total number of videos is: Notice that this is flipped: in the fish example the main measurement is "tagged" (the number of fish that were tagged the second time you caught them), which is in the denominator. But when counting YouTube videos, the main measurement is "valid" (the number of urls that resolved to videos), which is in the numerator.Did you understand where the 2^64 came from in their explanation btw? I would have thought it would be (64^10)*16 according to their description of the string.
Edit: Oh because 64^10 * 16 = (2^6)^10 * (2^4)
The YouTube identifiers are actually 64 bit integers encoded using url-safe base64 encoding. Hence the limited number of possible characters for the 11th position.
It’s not even new in the YouTube space as they acknowledge from 2011
https://dl.acm.org/doi/10.1145/2068816.2068851
That's only vaguely the same. It would be much closer if they divided the lake into a 3D grid and sampled random cubes from it.
Also related is the unseen species problem (if you sample N things, and get Y repeats, what's the estimated total population size?).
https://en.wikipedia.org/wiki/Unseen_species_problem http://www.stat.yale.edu/~yw562/reprints/species-si.pdf
Isn’t this just a variation of the Monte Carlo method?
I made the same connection but it’s still the first time I’ve seen it used for reverse looking up IDs.
I think YouTube locked down their APIs after the Cambridge Analytica scandal.
in the end, that scandal was the open web's official death sentence :(
The issue wasn't the analytics either. The issue was the engagement algorithms and lack of accountability. Those problems still exist today.
And what kind of accountability is that? An engagement algorithm is a simple thing that gives people more of what they want. It just turns out that what we want is a lot more negative than most people are willing to admit to themselves.
Engagement can be quite unrelated to what people like. A well crafted troll comment will draw tons of engagement, not because people like it.
If people didn't like engaging with troll comments, they wouldn't do it. It's not required, and they aren't getting paid.
This comment has a remarkable lack of nuance in it. That isn't even remotely close to how how human motivation works. We do all kinds of things motivated by emotions that have nothing to do with "liking" it.
I don't think people "like" it as much as hate elicits a response from your brain, like it or not.
If people had perfect self-control, they wouldn't do it. IMO it's somewhat irresponsible for the algorithm makers to profit from that - it's basically selling an unregulated, heavily optimized drug. They downrank scammy content for instance, which limits its reach - why not also downrank trolling? (obviously bc the former directly impacts profits, but not the latter, but still)
I would rephrase that to 'what we predictably respond to'.
You can legitimately claim that people respond in a very striking and predictable way to being set on fire, and even find ways to exploit this behavior for your benefit somehow, and it still doesn't make setting people on fire a net benefit or a service to them in any way.
Just because you can condition an intelligent organism in a certain way doesn't make that become a desirable outcome. Maybe you're identifying a doomsday switch, an exploit in the code that resists patching and bricks the machine. If you successfully do that, it's very much on you whether you make the logical leap to 'therefore we must apply this as hard as possible!'
So as usual, the exploitative agents get to destroy the commons and come out on top.
We need to figure out how to target the malicious individuals and groups instead of getting creeped out by them to the point of destroying most of the so praised democratizing of computing. Between this and locking down the local desktop and mobile software and hardware, we've never got to having the promised "bicycle for the mind".
Subscription VPNs show that there is potential for private security online.
Could a company be a bug you put a bounty on?
no one promised you anything
In which ways were the Cambridge Analytica thing and the openness of Youtube APIs (or other web APIs) related? I just don't see the connection
The original open API from the Facebook was open for the benefit of the good actors to use their data. You can disagree with how it's used, but u can't disagree with the intention.
With the CA scandal, now all the big companies would lock down their app data and sell ads strictly through their limited API only, so the ads buyer would have much less control before.
It's basically saying: u cant behave with the open data. Then we will do business only
CA was about 3rd parties scraping private user data.
Companies are locking down access to public posts. This has nothing to do with CA, just with companies moving away from the open web towards vertical integration.
Companies requiring users to login to view public posts (Twitter, Instagram, Facebook, Reddit) has nothing to do with protecting user data. It's just that tech companies now want to be in control of who can view their public posts.
I'm a bit hazy on the details of the event but the spirit still applies: there were more access to the data that were not 100% profit driven. Now the it's locked down as the companies want to cover their asses and do not want another CA
Wasn't the "open" data policy used to create Clearview AI to create a profile and provide it to US govt departments?
They actually held out for a couple of years after Facebook and didn't start forcing audits and cutting quotas until 2019/2020
Isn't this ironic, given how google bots scour the web relentlessly and hammer sites almost to death?
I have been hosting sites and online services for a long time now and never had this problem, or heard of this issue ever before.
If your site can't even handle a crawler, you need to seriously question your hosting provider, or your architecture.
Perhaps stop and reconsider such a dismissive opinion given that "you've never had this issue before" then? Or go read up a bit more on how crawlers work in 2023.
If your site is very popular and the content changes frequently, you can find yourself getting crawled a higher frequency than you might want, particularly since Google can crawl your site at a high rate of concurrency, hitting many pages at once, which might not be great for your backend services if you're not used to that level of simultaneous traffic.
"Hammered to death" is probably hyperbole but I have worked with several clients who had to use Google's Search Console tooling[0] to rate-limit how often Googlebot crawled their site because it was indeed too much.
0: https://developers.google.com/search/docs/crawling-indexing/...
if your site is popular and you have a problem with crawlers use robots.txt (in particular the Crawl-delay stanza)
also for less friendly crawlers a rate limiter is needed anyway :(
(of course the existence of such tools doesn't give carte blanche to any crawler to overload sites ... but let's say they implement some sensing, based on response times, that means a significant load is probably needed to increase response times, which definitely can raise some eyebrows, and with autoscaling can cost a lot of money to site operators)
I have a website thats get crawled at least 50 times per second. Is that a real deal? No not really. The site is probably doing 10.000 requests per second. I mean a popular site is indexed a lot. Your webserver should be designed for it. What tech are you using if I may ask?
I worked at a company back in 2005-2010 where we had a massive problem with Googlebot crawlers hammering our servers, stuff like 10-100x the organic traffic.
That's pre-cloud ubiquity so scaling up meant buying servers, installing them on a data center, and paying rent for the racks. It was a fucking nightmare to deal with.
This is one of the most important parts of the EUs upcoming digital services act in my opinion. Platforms have to share data with (vetted) researchers, public interest groups and journalists.
Vetted always means people with the time, resources and desire to navigate through the vetting process, which makes them biased.
I would argue it's better than nothing, and what are they going to be biased towards?
For aggregated data and stats like this I think it could be fully publicly available.
"Rules for thee, but not for me"
This would find things like unlisted videos which don’t have links to them from recommendations.
That’s a really good point. I wonder if they have an estimate of the percentage of YouTube videos that are unlisted.