Looking at the Downdetector home page [1], it looks like many more services are having outages, not just the ones owned by Meta, including:
- YouTube
- Google Play
- T-Mobile
- X (Twitter)
- Discord
- TikTok
- Pokemon Go
- Snapchat
It looks like they all have the same failure point.
It logged me out and told me that my credentials were incorrect; I thought my credentials had been stolen, so I'm kinda personally glad that it seems to be happening to a lot of other people too. I know that's a bit selfish, but :shrug:
A much better UX would be clear error messaging informing users that the service is down and there is no problem with their individual account.
This would prevent people from panicking they've been hacked and/or unnecessarily resetting their password.
You are absolutely correct. That would be a much better experience.
That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearly a daily event.
The obvious way to avoid those issues is a manual process. Problem there tends to be that the same service disruptions also tend to disrupt manual processes.
So you're right, but also I strongly suspect it's a much more difficult problem than it sounds like on the surface.
But there's something off here. I wouldn't expecting to be shown as logged out when the services are down. I'd expect calls to fail with something aka 500 and an error showing "something happen edited on our side". Not all the apps going haywire.
At the scale of Meta, "down" is a nuanced concept. You are very unlikely to get every piece of functionality seizing up at once. What you are likely to get is some services ceasing to function and other services doing error-handling.
For example, if the service that authenticates a user stops working but the service that shows the login form works, then you get a complex interaction. The resulting messaging - and thus user experience - depend entirely on how the login page service was coded to handle whatever failure the authentication service offered up. If that happens to be indistinguishable from a failure to authenticate due to incorrect credentials from the perspective of the login form service, well, here we are.
At Meta's scale, there's likely quite a few underlying services. Which means we could be getting something a dozen or more complex interactions away from wherever the failures are happening.
Slightly unrelated question, but just how "Big" is Meta? I know it's vast, but as an outsider I have trouble grokking the scale of it.
When most people talk about serving thousands and maybe millions of requests per second, Meta talks about billions of requests per second.
https://read.engineerscodex.com/p/how-facebook-scaled-memcac...
If that happens to be indistinguishable from a failure to authenticate due to incorrect credentials from the perspective of the login form service, well, here we are.
If you can't distinguish those, then that is bad software design.
Isn't this just the standard problem of reporting useful error messages? Like, yes, there are academic situations where you can't distinguish between two possible error sources, but the vast majority of insufficiently informative error messages in the real world arise because low effort was applied to doing so.
Come on use a little imagination. DNS lookup for the db holding the shard with the user credentials disappears. Code isn’t expecting this, throws a generic 4xx because security instead of a generic 5xx (plenty of people writing auth code will take the stance all failures are presented the same as a bad password or non-existing username); caller interprets this a login failure.
Same auth system system used to validate logins to the bastions that have access to DNS. Voilá.
plenty of people writing auth code will take the stance all failures are presented the same as a bad password or non-existing username
Those people would be wrong. You can take all unexpected errors and stick them behind a generic error message like "something went wrong" but you should not lie to your users with your error message.
That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearly a daily event.
Well, in principle, the frontend just has to distinguish between HTTP status 500 (something broken in the backend, not the fault of the user) and some HTTP status code 4xx (the user did something wrong).
There are quite some harsh comments here below. You can't plan for every possible failure point, who knows what part of a system/infra out of everything that they have went down and triggered this behaviour. Some things you just can't catch/predict. Especially in huge systems like theirs. I would expect people here to understand things like these and not just call people names for something like this, we all know things seem simple/clear from the outside, but the job of debugging and fixing something like this take quite some effort.
You can't plan for every contigency, but you can reserve potentially scary message for situations where you know they are correct. An unpected error state should NOT result in a "invalid credentialiald error".
This is the nature of credentials errors. The more information you give, the more you're telling an untrusted and therefore assumed-hostile agent.
I hate it because it's bad UX, but that's the thinking behind it.
Pushing people to unnecessarily reset credentials increases risk. Not only does it increase acute risk, but it also decreases the value of the signal by crying wolf.
The argument here is the kind of nonsense cargo cult security that pervades the industry.
This is a company with one of the largest digital infrastructures in the world. An outage is understandable, inability to tell they're having an outage and inform users appropriately is not. Stop making excuses for people who are literally awash in resources.
It is always better for the company's rep for the issue to have been on your end. Admitting fault comes with a potential liability. It's gaslighting written as an SLA
Not the worst thing that a bunch of Facebook users are resetting their passwords.
That's the sound of millions of "password" becoming "passwordnew"
It would be a better UX, but, depending on the outage, that might be a really hard behavior to guarantee.
There was for a brief moment. I got that once
Well you can't expect to hire engineers with half a brain for the pitiful compensation Meta offers, can you?
I called out some comment for being racist a little earlier (yeah I know, just report and move on...) and figured they'd managed to pwn my account somehow. Good to know it's not just me.
Strictly speaking, just because there's an outage does not mean you're not pwn'ed.
Maybe the outage happened because they used a 0-day to pwn smcl
Crazier things have happened.
In an "anything's possible" sense then yeah. But the fact that FB was not letting me login with the credentials I knew to be correct was directly attributed to a global outage, rather than a me-specific issue. Which I can now verify by checking the devices that are authorised to my account.
Same same. I went through the password reset flow (I was overdue anyways), it never sent anything to my SMS, so I did it again with email, reset the password and went to log in with the new password, "Incorrect password" error. Old password, also incorrect.
Didn't help that I had just posted a lukewarm spicy take on how linguistic prescriptivism is BS.
All the while the website felt like it was unstable, hard to describe, but it felt like it was bouncing around between URLs too much and reloading a lot.
Definitely feels like a botched update on their end.
E: Instagram is misbehaving as well, banner loads but big "Something is wrong" error on the feed.
E: now youtube has "Something went wrong" - WTF. I can't believe I'm saying this, but thank goodness for reddit and X[itter]???
E: interesting, seeing a big spike across multiple platforms on downdetector, including AWS: https://downdetector.com/status/aws-amazon-web-services/ I'm not able to log in right now, but that could be PEBCAK, I have too many saved IDs and I don't want to fail2ban myself
downdetector reports has gone down but to me is still bugged out, been catching a livestream on youtube all along though, meta stocks are back up from the dip so I take it some regions are restored to normality
The password reset flow was broken too. And I got logged out on every device. My friend who works there said they can't login either.
Discord is also having issues.
Same, thought all of my friends were getting spammed and i'll look like a "boomer" who got phished.
Then i remembered i have a very long and secure password, then immediately panicked about someone having access to my Gmail.
The sense of security is more brittle than i thought.
On a psychological note, I think the threat detection part of our brain doesn't always notify our conscious thought that it's actively monitoring for threats. I've often noticed that when I'm carefully handling a hot frying pan then my ringing phone is more likely to startle me than usual.
When you've already got one threat on your hands you're less prepared for anything else.
That makes sense. I've noticed too that my brain seems to have a threat pre-emption module as well as a threat reaction module. For example, I'll sometimes be walking and texting at the same time, only to stop in my tracks and suddenly realize that there's a hidden stair in front of me.
Yes. My spidey sense went off and I told my work I'll be off for an hour while I redo all my passwords... might still do that but glad to know it's not necessarily me getting hacked.
don't bother. fb's forgot password flow is broken too.
As was the account hijack process. It just loops.
I panicked the same. Even when I tried to recover my password, it said my email address wasn't associated to any account. I thought I lost it forever
That's the natural endgame of the "user-facing services must not stop, if something they depend upon stops, they must only degrade" philosophy.
I heard for a while Netflix would fail open if auth was unavailable. Like it’s just movies just let em see it.
Facebook data is more sensitive. Not so much the data people go there to see, cool memes that their friends liked, but the list of friends and interests.
Other places I worked had the ability for Ops to push out a change saying the site was down for maintenance. After a while we stopped using it and just took the hit of a bunch of 5xx errors. Basically when the planned down times became shorter than the time to propagate the down setting.
me too, also Instagram, could be that Facebook got hacked ?
Pro tip: in chase meta actually got hacked, it would be good idea to not use that password on any other websites, change them immediately.
Yea, the wife came to me in a bit of a panic that her Facebook account got hacked. I tried logging in to FB to check if I had been unfriended, and I also got errors indicating my password was incorrect. My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago. So, no my password wasn't wrong, and I'm not a big enough target for someone to put enough effort into figuring out how to snarf up my password data and crack my GPG passphrase.
Anyway, when FB thought my password was wrong I calmed way down. I thought maybe FB corrupted their password DB or something, so I just tried to reset my password, got into an odd workflow loop, and then quacked "downdetector facebook".
My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago.
We have the same approach to password management!
Likewise, started password reset process that won't complete, asked my wife to double check my account wasn't compromised and posting cryptocurency crap or somesuch.
I was considering buying the Quest 3 this morning, this outage is timely. The fact that I can't use my perfectly working headset to play an offline game because facebook is down, makes me wonder if I should go for a different provider. Any recommendations? Excluding apple vision pro since it is too expensive.
The Quest 3 is kind of the obvious mainstream consumer choice for a VR headset.
Standalone wireless headset, reasonably powerful chipset, can optionally stream from a PC either wired or wireless, good optics/resolution, decent controllers/tracking, large game library, large suite of features (including hand tracking and color passthrough), all for a reasonable price. Not sure any other headset really competes on all those things at once.
can we really call it standalone if you need it to phone home to Meta? and doesn't it require an account one of their services too?
they did a fantastic job with hardware. I just wish they didn't couple the software so tightly.
It makes me wonder how the world would be if monitor manufacturers did the same thing, it would be unacceptable.
I view VR headsets and their peripherals as no different than a mouse, keyboard, and display.
Companies requiring all this nonsense to use your device, put in that light, is ludacris.
comparing it with a monitor is rather unfair, you have to bundle the computing along with it, not to mention the applications to make it an actually fair comparison. At that point, is it still ludacris?
A silly comparison. A standalone VR headset is more comparable to a smartphone than a monitor or keyboard. The latter have little to no compute.
I know they raised the price recently but it seemed pretty obvious to me they were selling these at a loss to try and get people locked in by the software.
can we really call it standalone if you need it to phone home to Meta?
Yes.
The parent's comment is about how it quite literally is not a standalone headset, as is currently being demonstrated.
It is a standalone headset. You don't need a tethered gaming PC or console for the Quest series of headsets.
As a principle I think it is a good idea to avoid any technically unneccessary coupling of hardware or software to something else that is not strictly user-servicing, else the priorities are inverted in favor of the vendor's priorities and not your own.
I have been in the industry for the last 25 years and have done more “technically unnecessary” things than there are grains of sand on Earth :)
why are you smiling about that?
Probably sarcasm because it's likely out of their control due to product & executive demands.
Why are you serious about that?
I'm pretty sure the priorities are almost always inverted these days, unnecessary coupling or not.
Unless it's a recent change, it works perfectly fine offline (wifi turned off).
As for alternatives, there is Pico, but Quest 3 may be superior in games selection. Or go wired which is of course less portable
I think you need to be online for the first setup of the headset, otherwise it won’t work at all.
Yes. You login first, and then it completes device setup. At least, that's how it works for the Rift.
In my case I was doing iRacing with a cable, it requires the Oculus app to run for the link to work. Which in turn needs the login to be active.
This is different from it being offline, it's like the device is kicked off the associated account. A "something went wrong" window pops up with a "generate device code" button, with instructions on how to remove and re-pair the headset.
There is no way to even access wifi settings or anything else to disconnect the device from the internet. If it's still a problem much later in the day, I'll try turning off my router to see what that does.
I could not play a steam game on a PC in offline mode when my internet was down. This issue is not exclusive to fb.
Doesn't Steam still offer an offline mode that works with most games?
It does offer an offline mode. It does NOT work with most games, because the publishers literally can't help themselves but add more layers of DRM on top of steam DRM and most of those these days require always online connections.
Have never had this problem. There must have been something specific to your scenario, like the game wasn't downloaded, or it's a specific title?
That's not a Steam thing though, but rather the specific software. Steam explicitly has an exit path for the user if Valve disappeared overnight that allows their downloaded games to continue working offline.
The difference being that we are discussing platforms, not the things that run on those platforms.
I can confirm that I was unable to use my Quest 3 this morning. I left it connected to the internet, it tried to phone home I guess, and then locked itself into a "please connect this headset to your Meta account" state.
I am so sick of companies "selling" computers that they continue to control. In what universe does Meta have the right to remotely lock my headset and prevent me from using it to run the software I installed on it? If I were to sell my current desktop computer, or phone, or whatever, on any marketplace, and leave a remote login account on it that I then used to continue to operate the computer as though it were mine remotely, installing software, playing games, and occasionally peeking at what the current owner was doing, that would be obviously criminal. How is this any different? Because I signed away my rights when I "agreed" to their Terms and Conditions box (which I was compelled to do to use the hardware I purchased)?
Something is so fundamentally broken in the current ownership/property landscape. We somehow ended up in a world where people don't own the most critical tools in their lives, companies have managed to recreate feudal fiefdoms within the bounds of the market.
I can understand your frustration but were you not aware of the software lock in when you bought it? I'm not defending the ownership erosion, but I avoided these things specifically because of who was selling it and how it was locked to them.
Not OP but I bought a Rift when it was still just Oculus.
...then Facebook bought Oculus...
...and then required you to have a Meta account to continue using the Oculus drivers.
It's a real "boil the frog" strategy and this is still early days for VR in terms of realized market value. The time to push back on this bullshit is yesterday. As we can all see, nobody can compete with Meta on price with the Quest 3, but the cost to purchase is heavily subsidized by the expected futures.
Ubisoft Director of Subscriptions really opened the floodgates of bad behavior when they came out saying "Gamers need to get comfortable with not owning their games".
I think these companies need to be reminded they do not own our PCs either.
I'm really starting to like that mantra of "If buying isn't owning then piracy isn't stealing".
Steam Index is great if you don't care about cords / needing a PC. Easy to do VRchat, IRacing, or even blade & sorcery game sessions for 3-4+ hours without any eye strain, headaches, motion sickness, or discomfort from the headset.
It also fits over / around glasses
Biggest reason not to IMO is of the rumors around an "index2".
Seconded, I think. I've been very happy with my Valve Index, but I don't know how it compares to newer headsets on the market.
More broadly, I find that Valve represents my interests far more than Facebook/Meta does. So I'd much rather send my money to Valve.
I'll third this - Valve is really unobtrusive about the steam related features of the index. It does have some requirements with steam for setup but if you want to run a local binary and mess around with dev tools it's extremely easy to do. It's also extremely well sealed and designed - I tend to sweat a lot and a few times I've been beat sabering for quite a while without any long term damage to the headset.
Biggest reason not to IMO is of the rumors around an "index2".
Another big reason not to buy a Steam Index is "not available in your country". The only VR headsets I've seen in stores here are the Quest 2/3 and the PSVR2; and the Steam store page for the Valve Index (and the Steam Deck) says "not available in your country".
I think the current buzz in the VR space is the "Bigscreen Beyond" which eschews all of the nice-to-haves in order to make the headset as light as possible, and the result is surprisingly compelling.
It looks compelling for high-end PC gaming VR enthusiasts, but if GP is more of a mainstream consumer it probably won't make sense for them.
At least from what I've read, there's a bunch of downsides for regular consumers: very expensive ($1000) -- SUPER expensive if you bundle in controllers and tracking points (~$1600), needs external tracking, wired instead of wireless, no built-in audio, can effectively only be used by one person (because each one is built custom to your face), and of course it's not a standalone headset, it has to be hooked up to a gaming PC.
You can’t have everything. If you want a VR headset you described, there may not be any good ecosystem of software yet and you’d have to wait. This means you are gonna use your headset even less, think about how often fb goes down, it goes down for 2-3 hours once a year and it also has to coincide with the time you are using it? It doesn’t make sense to be this risk adverse
I'm typing this from Immersed in the Quest 3, no issues.
I used my Quest 3 for a couple of gaming breaks during the outage with no issues.
Whatever you do, get one of the all-in-ones that doesn’t require setting up tracking beacons. They are universally a pain and will cause you to never bother playing.
Quest is the best option right now
Lucky you didn't already have a Quest 3, and you were in CyberSpace when Facebook crashed, because then you would be trapped in CyberSpace and die. That's how it works, you know.
Tip for Meta engineers: when your service is failing, don't just log people out and prevent logins. Display a cute image that shows that the service is drastically failing (like a whale or something), and then people will know to stop trying to repeatedly log in. The public might even come up with a catchy name for the whale.
Beyond unbelievable that going on an hour later, they're still showing "incorrect password" errors. How many hundreds of millions of people have wasted time frantically trying (in vain) to reset their passwords and pointlessly freaking out that their account might be compromised? What a bunch of careless, incompetent excuses for engineers.
What a poor bunch of overworked human beings, with almost no control over the product they work on. Frantically following the whims of managers, reduced to labour units in this late stage capitalist hellscape.
But well paid at least. Working at Meta seems pretty shitty except for the pay from the stories I have heard.
It probably depends what team you're on, but I would not describe it as "pretty shitty." Being oncall for a 24/7 service sucks, yeah, but for my team it is one week a quarter and I haven't had any outside-of-biz-hour alarms the last few shifts. Other than that -- my work is challenging and interesting, my colleagues are friendly and smart, and my manager is decent. Not a lot to complain about.
That's such a beautiful comment I would almost consider printing it and putting it on my wall
Imagine how many hundreds of millions of users waste their time using instagram and facebook on a daily basis. Safe to say they don't mind wasting their customers time
This is your regular reminder that the users are not their customers. Advertisers and media outlets are the paying customers.
Major outages are periods of intense stress and extremely difficult to operate in. The folks troubleshooting may be many things, but careless and incompetent are unlikely to be among them.
Blame the managers and product owners that don't think this is an issue, not the developers that likely raised it a million times already.
I can almost guarantee you're getting mercilessly downvoted because half of the people here are sympathetic Meta worshippers who desperately (1) wish they worked there and (2) know they'd probably contribute similarly to this same horribly engineered system.
Could be an outage in the auth service itself
Yes of course it was. The point is, an hour later, they could have hit a circuit-breaker to get people to stop trying and going crazy over an error that is completely inaccurate.
Unless it's an outage in their ability to log into their own servers, they should be able to swap out the login page with a static HTML page explaining the outage. Maybe a 503 status code.
For anything outside like-ing and post-ing, facebooks UI/UX is horrendous. Even Internet search does not help to find out how something trivial is done... The only way is to watch youtube videos.
> For anything outside like-ing and post-ing, facebooks UI/UX is horrendous.
It isn't perfect for even that IMO.
> Even Internet search does not help to find out how something trivial is done... The only way is to watch youtube videos.
That says more about where the web is heading than about facebook. Video is easier to monetise ATM⁰, and these days people don't put helpful stuff out there just to be helpful as much as they once did¹, so content creators are making them instead of simple web pages.
--
[0] Everyone wants to be the next big influenza who doesn't need a day job to get by.
[1] That sort of people are still out there, though they are somewhat drowned out as the signal-to-noise ratio heads inexorably towards “WHAT? WHAT?! I can't hear a single thing above the manscaping adverts!”.
How many other people here just assumed they had been banned for some arbitrary, uncontestable reason? I just use instagram to post hiking photos...
Having had a look at desperate Twitter posts during a major outage of a big German email provider with similar failure mode (login failed silently), it seemed like many people assumed that their email account was hacked. Close enough.
catchy name
Petunia?
What do you mean my login flow sucks? Its time complexity is really good!
This is a great idea! I know that I was flailing about trying to figure out why I couldn't login, so I'd suggest calling it a "Flail Whale".
Display a cute image that shows that the service is drastically failing (like a whale or something), and then people will know to stop trying to repeatedly log in.
Probably not so easy to implement in behemoth apps, consisting of 20'000 source files...
Surely the repeated logins can't be helping the situation. I suppose it is entirely auth related across all Meta products. The repeated strain could pose a cold start problem for example.
Right? I know I was like "oh, I haven't typed in my FB password in eons... maybe I changed it at some point and forgot? But if I change it what happens to all related services, is it going to log out my kids' Messenger Kids devices? Those are such a pain to log in. Should I change my password or not? What do I do?"
Then I saw the news that it was an outage.
Might be related to this event?
The cut lines include Asia-Africa-Europe 1, the Europe India Gateway, Seacom and TGN-Gulf, Hong Kong-based HGC Global Communications said. It described the cuts as affecting 25% of the traffic flowing through the Red Sea.
https://apnews.com/article/red-sea-undersea-cables-yemen-hou...
I'm 95% sure it'll turn out to be a mundane config error somewhere
It will be DNS related.
Handy troubleshooting tool for outages: https://isitdns.com/
reminds me of http://iscaliforniaonfire.com/ even if not directly related to topic at hand
The sysadmin's haiku:
It's not DNS
There's no way it's DNS
It was DNS
I'm pretty sure that report is just mainstream media reporting this week old cut https://www.datacenterdynamics.com/en/news/at-least-one-subs...
Pretty neat if a week after a cable is cut, FB falls over.
Especially when most of the source of truth databases are in the US and Europe, and that sort of data flow doesn't cross the Red Sea. FB has datacenters and points of presence all over, but outside the US/EU it's almost all caching.
Thanks for the reference. Yeah, may also be entirely unrelated.
Pretty neat if a week after a cable is cut, FB falls over.
that'd be one helluva cache!!
time to move away from undersea cables to satellites.
We have satellites. We use cables b/c they lack the speed and bandwidth necessary to support the total requirements of the modern internet. Satellite-only is only feasible if you're fine with going back to waiting minutes for your saucy jpegs to load (elder millennials, you know what I'm talkin' about).
LEO satellites would be too inconsistent, and further orbits have way too much latency.
This shouldn't affect europe. It just stopped working
Somebody might have fat-fingered a BGP configuration while trying to improve traffic routing that was impacted by the cut cables.
Yea I thought too that core of this is not at the services itself but at thr network somewhere.
It's not unreachable. I can easily see the FB page on my browser. It's just that even after resetting my password it doesn't accept it. Probably something's fucked up in the credentials database.
maybe a long tail consequence of further shifting traffic?
BY JON GAMBRELL Updated 8:25 PM PST, March 4, 2024
Timelines don't match nicely
Those lines were cut yesterday, so it seems like a poor candidate for explaining the current outages. Likewise the geography doesn't match up with the outages.
I was thinking is deploy of DMA "compliant" unbundling the day before it takes force.
Could be both.
Anyone have a good guesstimate of how much money is "lost" when Youtube/Facebook/Instagram/Whatsapp (more?) are down each minute?
Naively, divide ad revenue by time to get a dollars-per-time.
But thats naive because ad serving isn’t totally sold out so they can make up for it by increasing the density of ads in the next time window. If the outage is short, then the impact is small.
But some markets are totally sold out and there’s no making up for lost impressions.
They also do not refund spend during this time typically so wouldn't be a 1:1
They don't? I've worked in ad tech at some smaller places and we absolutely refunded spend during outages.
How is money even being spent if users aren't viewing the ads?
Not all ad spend is impression based, there are things like takeovers. And, sometimes impression counts arent guaranteed, theyre estimated
That's because they were some smaller places.
When you've more-or-less monopolized a lot of the web's content sharing you get to tell your clients to pound sand. Where else they gonna go? Twitter? The incel white supremacist dollar is not what advertisers call "the good dollar".
From my experience you'll receive a partial refund - and in some instances like inexplicable overspending, etc. - you won't receive anything. This may be an exception given a full sitewide outage, though
they can make up for it by increasing the density of ads in the next time window
no need to do that, for google search, people will come back later to make the search
Yes, but, they’ll need to serve X ads in a smaller amount of time
This is something I've thought about a while back. Like Facebook probably has a "maximum number of ads shown to users per post" value. So theoretically, they have a ceiling for how many ads can be bought in a specific time frame before having to increase the ceiling/find new users.
Do they share data about this?
before having to increase the ceiling/find new users.
Or raise the prices of ads.
they can make up for it by increasing the density of ads in the next time window
Not only that but the bigger spenders will have more budget so the bidding after a large outage should return higher bids on average leading to increased profit per ad slot.
This is rough napkin math, no need to downvote if anyone knows the real number and this is way off :)
Meta 2023 ad revenue was $131 billion. To make it easy, let's assume an even spread for # of users and ad revenue generation per hour/minute of the day and day of the year (which I'm sure is not the case).
This would be:
$358 million per day
$15 million per hour
$249k per minute
This also assume a minute down won't be somewhat or totally offset by a spike in users when it comes back online.
I didn't check your math, but your last number is probably per minute, not per hour.
Thanks, you're right . Edited it
I can’t answer your question, but when I was at Google I made a mistake that caused ads serving on Google results to become unclickable. For the postmortem they had me calculate (I don’t think a dollar amount but) the number of ad clicks that would have happened during the time it was down. Of course I looked up average cost per click rates. Not sure if I could share even if I remembered, but it really put things in perspective.
Overall it was a good learning experience. I didn’t get reprimanded; several months later I got a promotion.
I can't believe you typed all that and didn't include a number, what a tease
I assume they will run all those campaign views after yhe outage, so very little will be actually lost
What are the chances this is GPT-5 that has escaped and gone rogue?
Jest aside, i wonder when/if stuff like that will actually happen.
Negligible. It would have gone after Twitter first.
You mean X right?
I don't think that anyone but Elon ever means X.
forgot the /s
Comically twitter seems to be working fine and is where I was able to find confirmation that it wasn’t just me.
Yeah, exactly, so it must still be a good old human mistake, not Skynet.
My shadow is extremely good at mimicking my every move, but I don’t live in fear that it will make the jump to the third dimension, kill me, and assume my identity. Should I?!
Well, maybe - if the source of the shadow keeps getting more and more powerful, it will eventually turn you into a permanent shadow.
About 95%.
Clearly the work of Llama 3
Hope it stays broken. Imagine? People would blink, confused, and slowly start to wake from a decade-long stupor.
how many people, realistically, use it daily? I think not a lot
Facebook: global daily active users 2023.
Statista https://www.statista.com › statistics › facebook-global-dau Feb 9, 2024 — During the fourth quarter of 2023, the number of daily active users on Facebook reached 2.1 billion, a minor increase on the previous quarter.
So roughly 28% of the planet.
Facebook accounts != individual humans.
couple billions.
Like, everyone I know is on TikTok all the time and Reels is right up there
Boomers probably do.
Lots and lots. Look at how many people are in this thread.
Really?
Enough for Meta to be a $1.2 Trillion company. What a silly comment.
I love how no one here is talking about the "coincidental" timing that this is happening on Super Tuesday. Carry on, nothing to see here.
I have yet to see one plausible reason for why this would affect primary voting in any way. Voting is done at the local level on local hardware; we don't vote on Facebook.
Messenger is also down. For some people this is an important means of communication (many young people I know in the US). People may either be in a panic or otherwise cannot coordinate sufficiently the plans they may have had to get to the polls.
That being said this year's primaries have got to be historically uncontested that it could not matter less.
It’s not direct disruption.
Grassroots organizers use FB and messenger and Insta for “get out the vote” communications. People use these services to goad their friends to go and vote. People use FB as a search engine to identify their polling place.
Same reason we're also not talking about the ten million non-coincidences that didn't seem to happen today. It's a kind of survivorship bias that coincidences seem like they mean something.
Or you could imagine it as this: your animal brain has evolved to notice patterns. Seeing coincidences like this is akin to seeing faces in the stars. The challenge of evolved being is to override those impulses when they're not logically sound.
Sure. But if millions of evolved brains are "seeing the same pattern", they will behave as such, and so the meaning of that otherwise random noise in fact becomes something material.
And that’s why conspiracy nuts exist.
If this was 2016, I'd think there could be a possibility. Given that the candidates for the election are almost 100% a foregone conclusion at this point, what purpose would it serve?
Fueling clickbait conspiracy theories is one probable outcome that comes to mind. For example, even a winning candidate may claim that a "woke" employee sabotaged the site in an effort to subvert the will of the people to have free and open discourse, they won despite the media deck stacked against them, etc.
We've seen continuous layoffs for the past year or so, and the stability of everything is wonky now. Yesterday I had trouble with LinkedIn, last week it was Coinbase, a month or so ago Gmail was hanging. I don't think it's because of Super Tuesday.
Discord is having issues too! https://discordstatus.com/
Could this be related to Google's log in page change? Seemed cosmetic only, but funny timing that Google's page update happens the same time all these log in issues pop up.
They've been previewing that change for weeks, and I wondered: why do they need to change it at all? Is it some product manager justifying his need to exist?
Based on their hype banners, I was ready for a major overhaul, too. Or at least something obviously different about the login flow. It sure looks like somebody just clicked the left-align button and spit-shined the typeface a tiny bit.
Which, I guess, is the best possible redesign: one that freshens up without rocking the boat.
Could this be related to Google's log in page change? Seemed cosmetic only, but funny timing that Google's page update happens the same time all these log in issues pop up.
That rollout is staggered over time, so not all users receive it at the same time. It's unlikely to be related.
I think the downtime associated with other services could just be people choosing alternative sites for their social media time.
Facebook + Insta makes up a huge share of the social media market, and when they go offline, it'd be natural for their competitors to receive large sudden upticks in trafic they're not immediately prepared for on a Tuesday morning.
Scary idea but could indeed be. Who let the zombies out!
+
https://metastatus.com/ seems to be suffering a hug of death now...
It really should be a static page.
Built on React, though.
React can be used fine for static pages. That's orthogonal.
Such irony...
It could save others from clicking on downed sites, but not itself.
Tangent to this but I saw a link to metastatus last week and thought it would be a status page for other services' status pages. This makes it sound like a useful thing now, too bad the name is taken.
Could call it metastasis, apropos when one outage cascades into others.
Official page to check status: https://metastatus.com
It would have been smarter if they hosted this on a separate reliable server altogether.
It's on AWS, but likely just has issues with provisioned capacity now that it's actually being hammered.
Meta has a small collection of tools on AWS to deal with large SEV0 events like these. Another one of them is a basic communication tool that does not use Meta's own servers for anything (including auth), a super basic version of the internal SEV tool.
X-Amz-Apigw-Id:
X-Amzn-Errortype: TooManyRequestsException
X-Amzn-Requestid:
Looks like the endpoints are on aws lambda and its getting rate limited.
Might as well rename it from Meta Status to Metastasis.
Thats down too. lmao
Well, I've been spitballing 1-3 years for effects of layoffs to affect company performance, so this timeframe is about right
Yup. That it’s related to the elections is also predictable, due to stress.
Made worse in big corp due to affirmative action + lack of enough qualified candidates meeting diversity criteria.
Which is inevitable when you have coarse criteria applied to such a large industry this way so quickly, as it takes decades for anyone to be qualified for the senior roles, and many years for junior/mid level, even if there were no pipeline issues, which there are.
And unqualified folks in leadership, and mid level == stupid mistakes.
And, with the DOL rules, the company can’t even pay people differently, so no bueno even giving the high performers keeping things afloat better bonuses - unless they happen to meet the diversity criteria and it makes the stats look good.
Which it’s already hard enough to do properly when there is only one dimension, and impossible when there are 2-3.
so the bigger the company, the faster it has to cut its own throat.
Your spitball is flying in a completely different universe than my spitball. Imo anytime you have layoffs that cut so deep, you cut through informal capability, knowledge and relationships that take a long time to form. If anything DEI helps create internal resilience because the personal networks end up a little different, giving you wider and more angles of coverage.
HN talks about people in open source holding up major functionality with little to no recognition. That happens within corporations too. Indiscriminate layoffs may directly fire those people, or signal to them that it's better to move elsewhere leaving gaps that only get discovered over time.
Of course, and not breaking anything with layoffs is already hard when your sole criteria as a manager is ‘are they effective’. Which it’s never been that simple, especially in big corp, but it’s waaay more difficult now.
And so what happens when you’re required by the gov’t and leadership to also comply with coarse grained population statistics AND you can’t find qualified people that meet those statistics enough?
My ex was a reasonably qualified software engineer, and even 4 years ago was getting no-interview offers because she was a woman - as explicitly stated by the recruiters.
It’s only gotten worse since then for hiring managers. She was offended because they literally didn’t seem to care if she was qualified or not.
I can provide links to signed and in force legal agreements between the DOL and Google for instance which formalize the need for this, and can point towards public records of evidence submitted to court of emails (internal) between recruiters which state the same too, btw.
Actual job qualifications (as in skills) did not enter the conversation at all. Just course grained DEI attributes.
So then they end up disproportionately cutting from the ‘qualified’ tranches (the groups that DO have to be qualified to stay) first because your stats still have to look good. I’m not saying DEI folks overall have no one qualified or hard working in them - rather, that there are little to no structural incentives for them to be. In many cases, they’re also unfireable/unlayoffable.
And eventually, the ‘non-DEI’ folks leave, burn out, or give up because f-this. Why do so much extra work when you literally can’t even get paid more for it, or be recognized because it will piss everyone else off?
And even if you’re superhuman on that front - everyone burns out eventually. Which is also why you tend to see what you see in Open Source.
It’s never any one decision, but stochastic movement in this direction has been relentless and inevitable.
Blaming an outage on DEI... Man that's a new one for me.
Could you let us know where you work? I want to make sure I never apply there.
Bwahaha, just wait until you see the shrapnel flying over the next year.
You don’t think the steady erosion in system reliability and ever increasing outages is unrelated to these pressures do you?
I’ve seen the sausage being made at the middle manager level in big corp for a long time. It’s never any one person/hiring decision, but the pattern and it’s impact has been obvious (and getting unavoidable) for a long time.
That no one seems to want to talk about the actual issues, but doing character assassination and black listing (like this comment) is part and parcel of the problem.
Instagram has stopped working for me and I've also noticed other sites having issues like YouTube?
Yes both Youtube and Instagram seems to be down too for me.
Instagram is down but Youtube seems to be fine for me
Youtube seems to be about 1/3 i get "Something went wrong" error on the frontpage, so only sporadic.
Maybe because of the sudden increase of traffic?
YouTube seems fine
Nothing of value was lost.
Entire businesses and livelihoods would crumble without Meta’s platforms.
They can migrate to other platforms.
What platforms have anywhere near the reach and services that Meta does for a marketplace/advertising?
I'm not personally a fan of most social media, but saying "they could go somewhere else" is a pretty naieve/ignorant response.
If your business only exists at the sole discretion of Meta then it's not a viable business, at least not in the longterm.
Facebook is throwing up the sign in page, as if your sign in has failed or someone's logged you out. Instagram and Messenger also seem to be affected.
Yes, just checked Instagram and it says “Couldn’t refresh feed” and is generally not loading. Messenger has also logged me out.
For some reason I couldn't browse the home page but I could still access my messages (website).
And Threads.
And the Quest headsets.
This will be great for productivity!
I thought the average facebook user was of retirement age.
In a lot of developing countries, Facebook is often the only app that a significant part of the population uses. And that includes young adults as well.
Not sure about globally, but middle aged, 45 - 65, slightly more women.
I would bet those who are not working in full time jobs are a lot more active though.
Instagram is also affected.
This might be the end of Facebook if it lasts long enough.
Invalidating people’s sessions on apps? That’s a HUGE cost. There’s a huge % of users that won’t be able to get back in.
It might not have actually invalidated the session
It does on the app. Logs you out.
I've known several people who, when they got signed out, couldn't remember their password and access their email, and so made a new account. I'm sure Facebook will see a spike of this.
I hope so.
Which one this time - BGP or DNS?
Either way, global GDP should be up :)
Given the range. Doesn't look like DNS
So... No way it could be DNS?
Narrator: it was DNS
My sense is that it's BGP.
every one panicked that they got hacked. i got a slim hope that i also got hacked and that i will not bother to recover my account and just roll without FB :) im too week to quit my self.
Maybe post your password to Twitter or something. Let it escape into the wild, where it can be observed in its natural state.
I know your intention is to help, but please don't share your FB password (if that wasn't obvious already lol). Letting randoms log into your FB account will just have massive consequences with your friends and family thinking it's you talking to them etc.
Send me your password and I'll log in and change it to nonsense. Get out of that heck hole.
YouTube is down too. Only thing working is Twitter.
YouTube works fine for me in Charlottesville, Virginia, U.S.A.
It’s working again. It’s been experiencing outages since 6am I guess.
YouTube seems okay here. East coast US.
Seems like somehow YouTube is also affected? I'm starting to think that this is an attack of sorts and not Meta specific
Given Facebook is like 10% of total Internet bandwidth, I'm curious what happens when all that traffic suddenly gets "redirected" elsewhere by people going to other sites and platforms. Could YouTube cope with a sudden 10% uptick in streams for example?
Edit: YouTube seems now to be defaulting to a lower bitrate, leading me to guess it's a demand issue.
That seems to be the actual case. Most of the "usual web" is struggling. Even HN is struggling for me.
Google Play Store also affected
[deleted]
Meta doesn't do that. For one thing, "Patch Tuesday" is a Windows thing, and 0% of production traffic is served from Windows there. For another, they are constantly and always redeploying.
More likely that someone bungled a deploy of user auth. No doubt they are rolling back as we speak.
More likely that someone bungled a deploy of user auth
How would that affect YouTube?
Surplus load from Meta users flocking to other platforms
What could affect so many different services? Some DNS server? Clould provider like AWS?
Underwater cables seemingly got attacked; the Red Sea situation is likely related: https://apnews.com/article/red-sea-undersea-cables-yemen-hou...
Timeline doesn't seem to line up
Probably the authentication serer being down.
I've never seen an outage this bad and I've been on Facebook since the oughts.
The 2021 outage was worse, it completely killed the network access to FB data centers for hours, and even led to issues with FB employees accessing offices (since the badge readers were also offline). It was so bad it got it's own Wikipedia page: https://en.wikipedia.org/wiki/2021_Facebook_outage
I don't remember that one - must have been on a hiking trip. This one might turn out to be worse yet! Hopefully not.
Oh dear.
Facebook, WhatsApp, Instagram, Threads are all down. and its time to contact the CEO of Meta to bring them all back up.
At least Twitter / X is still up, so time to complain about it there.
WhatsApp works for me
I'm on it! I WhatsApped Zuckerberg directly. He's aware of the situation.
HN also seems unusually slow. Wonder what's going on
Happens every time a major outage happens, everyone pours onto HN to discuss it which gives the one-core server HN runs on a really hard time.
HN is always slow when everything else goes down, tech people swarm here to check for information, and it overloads the site.
Also can't logged into my openAI account.
Not saying is related, but timing is weird.
if you are logging in with facebook account that might be the reason
Indeed if it was the case. But i always create an account with my email for this kind of situation.
Meta stock in freefall. Going short at 20x leverage.
Risky move. The slight drop (and subsequent recovery) today is unlikely related to this outage in any way.
Can't tell if serious, but I've seen worse trades. Wouldn't call this a freefall yet: https://imgur.com/a/0UN3HZa
Ironically, Meta employees are also affected by this
Just like last time they had an outage.
Gonna have my Angle Grinder controller ready for the new Sysadmin Simulator release
I can confirm that IRC still works! Pfewwwww..... /me takes his coat.
unless your bouncer is offline!
buserror has quit (*.net *.split)
Status page show some services are recovering. I wonder why ad services is the first service to be mentioned as recovering instead of login.
No reason to allow people on the platform if you can't monetise them, especially after monetisation has been paused for a few hours
Shortly before I received password recovery codes to my email, from all facebook accounts I have.
My email requires 2FA, as does my facebook account, but when I went there and clicked "forgot password" there was an unknown email address added to my account. That shouldn't be possible.
Your mouse click broke it.
The status page seems down for me as well. I just get a never-ending loading screen.
If you have an adblocker, try disabling that. The page fails to load with ublock Origin running.
Doesn't this kind of thing happen in every film regarding an AI becoming sentient ??
For now, some countries preparing a larger war seems more likely.
Note that this web page only covers the status of Meta's offerings for business users. This doesn't track the downtime of Facebook, Instagram, WhatsApp, Messenger, etc. as normal users experience them.
Curiously, WhatsApp did not suffer an outage in Brazil.
Mastodon is up!
Just checked again to be sure, even lemmy.world is up.
It looks like AWS is down if we can trust downdetector which is reason why nothing works. Almost fifty percent of these services runs on this cloud.
We can’t trust downdetector. Its just user reports. And users dont know anythig
Yeah so much for graceful feature degradation.
Just last week, the Meta paper about their Defcon system landed on HN front page.
https://www.micahlerner.com/2023/07/23/defcon-preventing-ove...
OH MY GOD WHAT IS THE WORLD GONNA DO!!!!!?????
Touch grass. xD
Google login also seems to be having issues, multiple people reported to me that the login isn’t working and they’ve been logged out of their Google accounts.
Yes, I tried logging in today in two distinct Google accounts on separate Chrome profiles and it would sign me out in about ~ 5 seconds after logging in. And the login process was very sluggish.
It appears to be more than Meta, lots of things are showing a spike in reports, including AWS.
That might be the FB traffic instead going to the content producers? I mean half the internet users need to go somewhere else if they want procrastination.
Probably DNS ...it's always DNS....
BGP is my guess
I’m seeing a lot of non-FB services down too. Mostly AWS-based, but not all. My original conspiracy thoughts (this being Super Tuesday in the US) are giving away to thinking it’s some low level routing issue.
It's definitely related to Super Tuesday. Why kill 2 birds with one stone when you can kill 10?
You know shit's bad when even their status page is down.
Is it coincidence that Meta is trying to get portions of NSOs source code via the court system. Also today is apparently a big voting day. The techno-apocalypse is near.
Llama3 become sentient
Google login seems to be slow just now for 2 of our websites
Seems like even the status page doesn't work ... i'd say typical for this type of situations :D :D :D
For me it happened when my long-dormant messenger lighted up (for a perfectly valid reason, no surprise there) and made me go through those unbundling options presumably mandated by EU. Certainly a coincidence, but it does irrationally strengthen that satisfying feeling I get whenever I see the ad giant stumble.
Since we're in full-on speculation mode ... Russian cyber-attack ?
yes it's all broken now, so I can take a short break here :)
This is so funny because the status page itself actually shows 500 Internal Server Error to me in the API calls. So the status page clearly isn't isolated from the FB network itself. I highly suspect it is either a BGP convergence issue, or their OIDC service hits the dirt.
I have an active Instagram account. Today, coincidentally was the first time I thought of promoting my best few posts as an ad to improve reach to fellow photographers. Bad luck! The app now seems to be back online but my ads which were supposed to run for 48-72 hours, show <Disabled>. And there is a new "Pay Now" link, even though it was paid for and seems ad-spend is already showing it used some of that payment.
As an individual, this is pretty confusing. I don't have much to lose. I am glad I spent 10-15 bucks on favorite 2-3 posts only. I can imagine many others to be more affected. What is normally to be expected for SME users? Does Meta resume the ads automatically? Do they make good for the lost time since the clock seems to be ticking -- although no one saw any impression.
Seeing such a large web property go down like this is fascinating.
It's like a power plant grid failure, except for attention instead of energy.
When meta is down, a hoard of internet users desperately seek somewhere else to place their attention... But the system is designed with the expectation that meta will take all that traffic... And boom! everything starts falling over. Wild.
hackernews is also v slow for me
Oh, that's why I was logged out then.
Fixed. I can login to facebook.com now.
They layoff too many ppl.
Lovely bit of cloaked dramatic irony in "Leave The World Behind" t=1:11:20 [0]
Ruth (Myha'la): This is for the better.
G.H (Mahershala Ali): For who?
[0] https://en.wikipedia.org/wiki/Leave_the_World_Behind_(film)
Discord too ?
FB logs you out and claims "Wrong password" if you try to login. IG just doesn't load new posts or comments, but doesn't log me out.
Cloudflare reporting that they were implementing a fix for something SSO-related at 16:02.
Cloudflare also having issues as of 1 hour ago:
https://www.cloudflarestatus.com
I wonder if it's connected to this. Seems like most large players are having issues now.
Why the outage?
I am surprised by how many people still use Facebook around here, as my social circle deleted or stopped using Facebook a long time ago.
Has a meta developer used Chat GPT again?
good luck to those on call
Well if everything down maybe isn't problem at services itself but at the DNS side?
Funny. This page won't work too and is clearly down. So there is possiblity of pretty big amateurism from facebook or failure at dns servers.
At least it's not a mission-critical website . . .
Laid off too many people? Critical services no longer have owners?
Oh good, I thought iOS Firefox was forgetting to load its cookie jar again
Eh, leave them down.
My Meta Ray-Ban Stories glasses are working just fine FWIW
I find it remarkable that Typepad, my blog host, not only isn't down but also is MUCH FASTER than its usual slothlike, approximately 5 seconds response time.
On Super Tuesday in the US today, too. Very interesting timing for how much political nonsense is on social media.
Internet Outages Map - https://www.thousandeyes.com/outages/
I don't use Meta's services but I have a number of co-workers that are all convinced that their Meta accounts have been hacked.
It's DNS
So I guess this is what that national security warning and purchase of Palo Alto Networks stocks was all about in mid February.
I'm having one heck of a time getting into Discord or getting YouTube to show videos. Anyone else experiencing issues outside of the Meta realm?
SRE now have real work to do.
Obligatory DNS joke :-)
It's not the DNS, It can't be the DNS, It might be the DNS, It's the DNS
Ads Manager - Recovering from disruptions
LOLLLL .. oh dear
They recently started demanding that I log in on a phone app, to make some choice to use messenger on ipad. That is pretty much the end of messenger for me.
Threads is down too. Given its integration with Instagram that's not exactly surprising though.
Everyone beware, The “Black Hat Hackers” that have accomplished this massive Global hack represent a Clear and Present Danger, immediate and significant threat/danger to National Security, as well as all People’s personal and financial Security
Everyone Beware, The “Black Hat Hackers” that have accomplished this massive Global hack represent a Clear and Present Danger, immediate and significant threat/danger to National Security, as well as all People’s personal and financial Security
SSO in Cloudflare also seems to have problems.
Or people are using Facebook Auth for them. I don't really trust Down Detector, which despite the claims is really People Winging on Twitter Detector.
I'm confused. Isn't listening for spikes in complaints about outages a great way to detect them? I know for a fact some service companies monitor social media channels for this purpose (among others). I'd be surprised if that wasn't more or less standard practice.
I've checked Down Detector for ISP outages in my area many times now. It's always confirmed them before my ISP did.
When there's a major ISP outage, people report problems with all the major sites. When Facebook's down, people report problems with any site that has "Login with Facebook" as an option.
It's almost never actually an outage impacting all of FAANG at once.
> When Facebook's down, people report problems with any site that has "Login with Facebook" as an option.
If users log into your site with Facebook, then the login functionality of your site effectively is down when "Login with Facebook" is down.
From the user's perspective, your subcontractors, including authentication subcontractors, are a problem for you to deal with and never show them. From your perspective, you could have architected your site in a way that logging in doesn't "go down" when Facebook login is down.
If the user chooses "Login with Facebook" over other authentication options available, and they don't want to use other options, educating them with a good error message might help. Or you could remove the Facebook login option, if you (totally reasonably) don't want Facebook's failures to reflect poorly on you.
There are plenty of sites where "Login with Facebook" is a convenience but hardly the only way to log in. Reddit, for example, has "Login with Google" and "Login with Apple"; it would be highly misleading to claim "Reddit is down" if Google's OAuth flow was having an outage.
Nothing in the API or OAuth flow would make that doable in an automatic fashion with this outage. It'd have to be something you put up manually as a banner after hearing of the outage.
I don't particualrly care; we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it.
> Nothing in the API or OAuth flow would make that doable in an automatic fashion with this outage. It'd have to be something you put up manually as a banner after hearing of the outage.
That is fair: if I choose to architect my site such that a user-critical feature goes down when a 3rd party service goes down, it behooves me to monitor the 3rd party service and do whatever necessary to properly inform users what's going on.
I edited my post unfortunately after you replied, but another option is removing the parts of your site that rely on 3rd parties, if you don't want the failures of those 3rd parties to reflect poorly on you (which they reasonably would).
>we're talking about why DownDetector isn't necessarily ideal for assessing. It can be a useful signal, in some scenarios, but I've seen plenty of spurious signals come from it.
Indeed, and if a bunch of users say that a feature of your site is down, even if it's a result of a 3rd party failure: chances are, that part of your site is down, and it's partially your fault for relying on a 3rd party for that feature. The users correctly don't care what the root cause is, they expect you to either mitigate it or don't have a feature they rely upon be unreliable.
All that's fine, but totally misses the point.
Take a look at https://downdetector.com/status/aws-amazon-web-services/ ; scroll down to the comments.
"SSH and Dbconnect stopped on all of my EC2 instances. Anyone else?"
"I can't add a payment method"
The chart shows a big spike this morning, but there was no AWS outage, nor does Amazon use Facebook login.
Again, DownDetector can be a useful "is something unusual happening right now" signal, but it'd be a mistake to take its attribution at face value.
Ignore the comments on DownDetector for a moment and check out that huge spike in reports recently. Clearly something wrong happened with AWS's user experience. That's something AWS needs to resolve, in the eyes of their users.
>The chart shows a big spike this morning, but there was no AWS outage
Are you sure? If hundreds of users simultaneously reported there was some sort of outage, particularly a huge spike like we saw, chances are there was an outage.
>Again, DownDetector can be a useful "is something unusual happening right now" signal
Exactly! Specifically, "is something unusual happening right now with my site, in the eyes of my users?" Every site owner should know when that condition is true. What you think about your site "up-ness" isn't as important as what your users think about your site "up-ness". What you attribute your downtime to, isn't as important as what your users attribute your downtime to (you.)
But that's not the case. It's a false positive.
Pick a DownDetector service and open the page every day for a few days. You'll see it most of the time just reflects people waking up in the US timezones.
Exactly. If you click through down detector when things are _up_ you'll see people still complaining that $site is down. Could be a local power outage or even a flaky connection in their own home.
Down Detector is one of many signal sources and should have a "credibly" score associated with it that's proportional to the number of people complaining that something's down.
do you really think there are masses of people who can’t tell the difference between a single sign on service being down and individual sites being down and reporting it to downdetector?
Even if there were doesn’t the outage graph give you exactly the information your asking be curated?
I very much don’t like the idea of downdetector using some special sauce to hide or limit outage reporting.
Have you never seen The Website Is Down? https://www.youtube.com/watch?v=uRGljemfwUE
The answer is: way more people than a software developer might think. Ask anyone in IT, or go to anywhere bugs are reported and read a handful.
Yes, absolutely. 100%.
https://downdetector.com/status/aws-amazon-web-services/
Was there an AWS outage this morning? The graph sure looks like it, but there wasn't.
I can guarantee you with 100% confidence from experience that the call centers for AT&T, T-Mobile, Comcast, etc. are all blowing up right now because of users who assume that if the Instagram app isn’t loading it means the “wifi” is broken. Also keep in mind “wifi” doesn’t mean 802.11, it means “anything related to the internet” up to and including 4g/5g and Ethernet.
Yes? That's how all top-level reporting is going to work. It's not going to tell you which part of your service is inaccessible. It's just telling you that people can't access it. You obviously have to do additional investigation to figure out why people are having trouble.
Scroll up the thread a bit; https://news.ycombinator.com/item?id=39605354
Even here on HN, where people should know better, people take its incorrect attribution as useful info. TikTok isn't down. X isn't down. Google isn't down.
That's kinda the point though isn't it? DownDetector is showing an early indication of a major outage in both of your examples. The issue may not be caused by the indicated service, but it's still a useful information source especially when we can correlate reports on there with what we are seeing in our internal monitoring.
A big spike on DownDetector is an indication of something going on.
Its attribution of what/who is often incorrect. You'll see "maybe it's more than Big Site X!" comments come up on every HN thread like this citing DownDetector; it's almost never the case, and folks on HN should know better.
The OP means there is a lot of collateral noise from people who are just tech savvy. Eg. “oh no, I can access Facebook, my internet must be down. Let me login in Down Detector to file a complaint against my ISP”
The problem is the source of the reports and display of the reporting.
I'd trust Down Detector a lot more if it was filled with Hacker News community -- people who are able to understand that there's "DNS" and "Routing".. and that your phone can have internet access at home while your home PC does not.
I personally hate Down Detector's graphing because it can make it 'look' like there's an issue when there isn't really... Facebook with 500,000 reports looked as down as Google with 1,000 reports... For equally sized / used entities, I would not trust that "Google" is down with 1,000 reports. I had a coworker ask me what was going on with the internet because "everything is down.. Facebook, google, gmail, microsoft!" (when seeing the Down Detector home page)
DD should normalize the graphs against the service history in some way. A service shouldn't spike because it had 30 reports / hour for a day, then suddenly has 100... when it has a history of being out with 100,000+ reports. The 100 reports are probably mis-reporting, but you can't tell until you dig into each service, one by one, with separate page loads.
The problem is that people use FB login for other sites, and if FB login is down, many users report a problem with that other site, not with FB.
Yeah, it's wild that it's now treated as an authoritative source, especially by some news organizations.
It's as good as asking a neighbor what happened with a loud noise down the street. Sometimes you'll get something good, sometimes it'll be completely wrong.
Asking my neighbors if they know what some loud noise was or about some local disturbance has been extremely reliable in my experience. The one time someone gave me an explanation about something which wasn't mostly right they qualified it with something like "So-and-so said it might be such-and-such but I don't know if it's true".
You must have an exceptional neighborhood. Everywhere I lived, here's a handy map of "actual cause" :: "what the neighbors said it was"
Car exhaust :: gunshot Appliance delivery truck liftgate :: gunshot Transformer explosion :: gunshot Garbage truck :: gunshot 787 at 25000ft :: complete ruining of peace and quiet Any police activity :: probably someone robbed a bank
For the record, my city has (statistically indistinguishable from 0) homicides and bank robberies and, by American standards (I know, I know) no particular issues with gun crime.
I can imagine it being different in a city. I'm in a fairly quiet suburban area.
One time I heard a loud boom. A few hours later I saw a neighbor outside and asked if he'd heard it and if he knew what it was. He told me a house a few neighborhoods over had exploded. I was a bit skeptical of it but he turned out to be right.
Downdetector is nice because it answers my question of "is anyone else having issues with this?" When it takes AWS an hour to even acknowledge "increased error rates", and tells me that everything is a-ok in the meantime, I want another perspective.
Twitter's search used to be my go-to for this - a search for "AWS down" would typically be very illuminating - but it's tough to get it to genuinely spit out the most recent tweets with a keyword these days.
The New York Times just posted a news flash ... citing Down Detector :-P
Gmail is also on the list. You can't use FB auth to login to Gmail, can you?
Gmail was also experiencing issues: https://www.google.com/appsstatus/dashboard/incidents/shD5Vv...
YouTube was definitely doing something weird that doesn't seem likely connected to Facebook.
A couple hours ago after watching a video I went to my home page, which usually shows recommendations based on what I've recently watched plus a few videos labeled as sponsored that have nothing to do with any of my interests.
Instead everything on the home page was either a sponsored video, or a movie that was free to view with ads, or something from one of their music products.
I tried from an incognito window to see if it had something to do with being logged in. Normally going incognito loses the history-based recommendations but at least recommends user uploaded content. But now it has just like my logged in home page. No user content. Just ads and videos from Google's movie and music services.
Refreshing gave an error that said something went wrong. I then logged in on that page and again got something went wrong. Another refresh got a page with some user content. Another refresh was the ads and Google stuff page.
A little later it seemed to clear up and now my home page is back to normal.
i trust Down Detector more than the (majority of) companies who are silent during outages
hell, i'm surprised Down Detector hasnt been outright sued due to the graphs being an actual honest representation of availability that shitty companies cannot hide
People are using Facebook Auth for YouTube?
Seems likely. TikTok and YouTube are currently working for me, while Meta platforms aren't.
Their outage heatmap is also basically a population density map too. https://xkcd.com/1138/
HN seems to be struggling too, but that could just be everyone here to talk about the outages.
That's the standard HN experience, this site runs on a single core I believe.
Does it really or is this a joke?
Edit: I found the following, I wonder if it's still the case.
https://news.ycombinator.com/item?id=16076041
It's real. Single core performance improves all the time. People overestimate how much power it takes to handle lots of queries per second on a well-tuned system and well-written software in 2024.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I see the "sorry, we are receiving too many requests, try again in a few minutes" error several times a day on here. I don't think that HN is reliably able to handle the amount of users it currently has.
I have been using HN daily since I was a teenager. I've seen that message maybe 10 times outside of serious issues in last 15 years. It's strange to me that it happens so frequently for you.
That's difficult to believe to be honest. I get it several times a week.
I get something like that when I try to comment and then upvote too quickly.
I believe that's by design if you send an action request very quickly after a previous one. It's very easy to replicate. Open a post. Then click the upvote button and very quickly click the favorite button too. That will trigger it. I think it's used to rate limit.
dang, linked in one of the ancestor comments. But I still suspect you are correct.
I just tested it by quickly upvoting your comment and then favoriting it and the error was:
Note that this only seems to happen for actions. Doesn't seem to be the case if I am just loading a page quickly.
Hmm. I get a different message. Something like "We are having trouble handling your request. Sorry!"
I saw it just a few minutes ago, but I don't remember the exact wording...
It's a feature
Note that's the application server process being single threaded, but the server machine is 4 core, so nginx cache etc use other cores
A worthwhile distinction!
I looked up the CPU mentioned in the link from your other comment. It looks like HN handles enormous traffic on about 2x the power of the last Celeron chip ever made.
https://www.cpubenchmark.net/compare/2383vs5793/Intel-Xeon-E...
I wasn't overestimating anything, but with how easy it is to write concurrently software today, why limit your site to a single core.
Maybe it's a lisp thing. Who knows what mysteries lurk here
Doesn't he mean single socket by single core?
These are good for actual business needs, but bad for resume-driven development.
See also: https://news.ycombinator.com/item?id=28478379
You don't run a massively profitable VC company by just throwing money away at a second core.
Happens whenever Sama sneezes too
It's honestly despicable how billion dollar companies at the forefront of making AI have such insanely brittle networking.
"blah blah blah I don't like the company so all their engineers must be really stupid..."
"The metaverse is the future of this company! We've hired the very best brains in the world to make these 1998 era PS1 polygon avatars!"
whole company goes offline
The Metaverse engineers and AI researchers don't actually run the infrastructure, but by all means, keep shitposting. It's so clever.
Yes, but there's been a trend recently to cut costs on all engineering except AI. Might be related
You keep defending developers while I'm pointing the finger at corporate. But good victim complex.
Yep. I've also noted that the people making such claims never seem to cite their own work as an example of how to implement something at Facebook or YouTube scale that is less "brittle".
Armchair quarterbacking isn't just a U.S. football phenomenon.
Dealing in absolutes is a result of misaligned expectations.
"The Force SHOULD have saved my mother's life!" - Anakin Skywalker
iirc these all use GCP which would make sense for them all to be disrupted at the same time. I wouldn't have thought Meta was GCP reliant though?
all of them are using oauth, likely auth provider issue?
https://www.cloudflarestatus.com is reporting an issue with SSO login. So seems like you might be onto something..
Definitely not us.
And I assumed it would be DNS again. Would referral traffic cause issues like this?
There’s no way fb uses gcp
Down Detector is so unreliable. People that can't call an AT&T phone via Verizon will think (and report) that Verizon is down, when it's really AT&T. People can try logging in using Facebook's on click login and not be able to get in, so they think Tiktok is down. It's not all that useful. I hate when journalists cite it.
It's only slightly better than "my mom claims". My mom would ask if I had the internet at my house. Yup. all of it. in a rack in my bedroom closet. She'd also report the "internet is down" when a single website was having issues. To me, that is down detector carrying on the legacy of moms everywhere.
A single report on there is useless. A sudden flood of reports is a good sign that something interesting is happening.
Eh, mostly it's people misunderstanding what it represents.
If I can't login to tiktok because FB is down, then tiktok is effectively down for me. When it comes to technology most people don't care about the trip, they care about the destination.
So yea, tiktok isn't "down" but for a lot of people it might as well be, hence coupling your infrastructure/auth on other providers has side effects like this you must take into account.
It has false positives and noise for sure, but it's also very sensitive and shows issues very quickly.
I wouldn't trust it as a single source, but in a case like this where our internal monitoring shows a spike of issues with the Google APIs and we can see a huge spike in reported issues for Google on Downdetector starting at the same time, it's useful to confirm that the issues have an external source.
S**'s gettin' real: https://downdetector.com/status/pokemon-go/
Scrolling down their list, why is DoorDash the only one that didn’t have a spike this morning?
Everything is going down, except Steam. You know where to find me.
Some piece of core infrastructure went down because everyone got spike at the same time. Surprisingly DoorDash and Steam was up
What's your fave conspiracy theory? Massive cyberattack for Super Tuesday? Powers-that-be mandated takedown? Mossad sleeper agents activated? Covid-brain struck that one engineer attending to that one wire that kept everything going?
Houthis destroying underwater cables in the Red Sea.
i knew my packets took a wrong turn at Albuquerque
I guess the swamp got drained, so there's no more flow through the tubes.
There is no consistent scale on that graph, so any local maxima of reports received would look similar to any other.
Exactly this. FB topped out around 520,000 reports. Google topped out around 1,400. That's a massive difference in scale.
Both are above their baselines, but I bet some is just mis-reports, or increases in awareness due to more people checking in.
Meta seems to be the only one really affected from what I can tell.
We saw a big spike in latency and failures on the Google OAuth apis starting at the same time (15:21 UTC)
Additional fun factor: today is Super Tuesday - primary elections in a lot of US states.
This outage will result in absolutely no ridiculous conspiracy theories.
If your election integrity relies on Facebook, YouTube, or even DNS to be up... there are bigger issues.
Actually, if all of them including Xitter went down, maybe things would get better? All the sunlight photons might get sucked in by too many eyeballs though, and there could be grass trampling.
I agree. While I don't think it likely that Facebook or YouTube would enter into it, I'd pretty much bet that DNS being down would cause problems.
And yes, there are bigger issues with that. Much.
A lot of sites have features that say “log in with your X or Y account.” They connect to each other somehow. I never studied that protocol. I wonder if authentication failures across services could be tied to it.
For process of elimination, do all of these services do multi-platform logins? Or do some not connect to anyone else?
that actually infuriates me more than cookie banners. the one from Googs is the worst offender.
Most of those seem OK for me now, and DD agrees. This seems to have been a temporary blip for all of them, possibly some kind of service switchover/fallback "not entirely unrelated" to the Meta outage?
Edit: actually a more attractive theory, given the very short timelines and near simultaneity of all those failures, is that downdetector itself had a failure, possibly a Meta-dependence, that they noticed and corrected quickly.
GCP seems fine, and no issues logging into Google Cloud Console.
Fb is down but YT is still up for me.
Facebook audience is in the billions, so you will see 100k false positives when a big site like that goes down.
Services are coming back to live.
Interesting to see that all static content was still working during the outage (at least for Instagram). It was still possible to swipe through all reels (I assume the list was cached).
What could have this sort of blast radius? BGP?
Man the postmortem on this is gonna be fun.
“Yeah so it turns out when Facebook and Instagram goes down so does Google”
I do not envy the SREs at either company. I'm pretty sure all those other ones use Facebook or Google as their OAuth provider which is why they are all being reported as down.
Thanks for sharing this, from what I have read it looks like an issue beyond just Facebook.
Google was acting up for me as well, so that could be
Cloudflare or AWS ?
Downdetector has successfully detected 150 of the last 20 outages.
It’s mention should honestly be banned from this site.