return to table of content

Private Cloud Compute: A new frontier for AI privacy in the cloud

hexage1814
58 replies
12h49m

The thing with cloud and with anything related to it, anything that connects to the internet somehow... is that, unless it's open source and the servers decentralized, you are always trusting SOMEONE. Sure, Apple might make their best to ensure nobody – but them – have access to your data... but Apple controls all the end points. It controls the updates your iPhone receives, it controls the servers where this happens. Like, they are so many opportunities for them to find what you are doing. It reminds me of this article "Web-based cryptography is always snake oil"

https://www.devever.net/~hl/webcrypto

And to be fair, this doesn't apply only to this case. Even the data you have stored locally, Apple could access it if they wanted, they sure have power to do it if they so wish or were ordered by the government. They might have done it already and just didn't told anyone for obvious reasons. So, I would argue the best you could say is that it's private in the sense that only Apples knows/can know what you are doing rather than a larger number of entities .

Which, you could argue it's a win when the alternatives will leak your data to many more parts... But still far away from being this unbreakable cryptography that it's portrayed it to be.

noahtallen
38 replies
12h8m

I don’t think that’s completely fair. It basically puts Apple in the same bucket as Google or OpenAI. Google obviously tracks everything you do for ads, recommendations, AI, you name it. They don’t even hide it, it’s a core part of their business model.

Apple, on the other hand, has made a pretty serious effort to ensure that no employee can access your data on these AI systems. That’s hugely different! They’re going as far as to severely restrict logging and observability and even building and designing their own chips and operating systems. And ensuring that clients will refuse to talk to non-audited systems.

Yes, we can’t take Apple’s word for it. But I think the third party audits are a huge part of how we trust, and also verify, that this system will be private. I don’t think it’s far to claim that “Apple knows what you’re doing.” That implies that some one, at some level at Apple can at some point access the data sent from your device to this private cloud. That does not seem to be true.

I think another facet of trust here is that a rather big part of Apple’s business model is privacy. They’ve been very successful financially by creating products that generate money in other ways, and it’s very much not necessary or even a sound business idea for them to do something else.

While I think it’s fair to be skeptical about the claims without 3rd party verification, I don’t think it’s fair to say that Apple’s approach isn’t better for your data and privacy than openAI or Google. (Which I think is the broad implication — openAI tracks prompts for its own model training, not to resell, so it’s also “only openAI knows what your doing.”)

chem83
17 replies
11h33m

What makes you think that internal access control at Apple is any better than Google's, Microsoft's or OpenAI's? Google employees have long reported that you can't access user data with standard credentials, for example.

Also, what makes you think that Apple's investments on chip design and OS is superior to Google's? Google is known for OpenTitan and other in-house silicon projects. It's also been working in secure enclave tech (https://news.ycombinator.com/item?id=20265625), which has been open-source for years.

You're making unverifiable claims about Apple's actual implementation of the technical systems and policies it is marketing. Apple also sells ads (App Store, but other surfaces as well) and you don't have evidence that your AI data is not being used to target you. Conversely, not all user data is used by Google for ad targeting.

Spooky23
8 replies
6h47m

It’s not about technology. It’s about their business.

Apple generally engineers their business so that there isn’t an incentive to violate those access controls or principles. Thats not where the money is for them.

Behavior is always shaped by rewards and punishments. Positive reinforcement is always stronger.

whynotminot
6 replies
5h32m

One hundred percent this.

All these conversations always end up boiling down to someone thinking they’re being clever for pointing out you have to trust a company at the end of the day when it comes to security and privacy.

Yes. Valid. So if you have to trust someone, doesn’t it make sense for it to be someone who has built protecting privacy into their core value proposition, versus a company that has baked violating your privacy into their value prop?

TremendousJudge
2 replies
4h5m

That's a false dichotomy. You may have to trust someone but that someone could be something else than an opaque for-profit company.

whynotminot
1 replies
3h57m

Give me some examples of benevolent non profits that provide anywhere near the level of consumer services as a company like Apple.

talldayo
0 replies
2h35m

I'll do better, here's a benevolent nonprofit that goes beyond what Apple provides to ensure top-notch consumer service: https://grapheneos.org/

Teever
1 replies
2h24m

They're not trying to be clever, they're trying to point out the very important philisophy of maximizing self reliance that so many people like you eschew.

How do you distinguish between a company who 'has built protecting privacy into their core value proposition' and one who just says they've done so?

What are you going to do if a major privacy scandal comes out with Apple at the center? If you wouldn't jump ship from Apple after a major privacy scandal then why does your input on this matter at all?

Some people feel that is inevitable so it's best to just rip that bandaid off now.

whynotminot
0 replies
1h55m

I'm taking aim at the Google bros who try to raise these arguments to muddy the waters into a sort of false equivalence between Apple and Google.

If you're already using a dumb phone and eschewing modern software services, then I'm not really talking to you. Roll on brother/sister, you are living your ideals.

How do you distinguish between a company who 'has built protecting privacy into their core value proposition' and one who just says they've done so?

The business incentives. Apple's brand and market valuation to some extent depends on being the secure and privacy oriented company you and your family can trust. While Google's valuation and profit depends almost entirely on exploiting as much of your personal data as they possibly can get away with. The business models speaks for themselves.

Does this guarantee privacy and security? Does Apple have a perfect track record here? No of course not, but again if these are my two smartphone choices it seems fairly clear to me.

talldayo
0 replies
2h29m

It's not about being clever, it's about being perceptive. Apple's cloud commitment has a history of being sketchy, whether it's their government alliance in China, the FIVE-EYES/PRISM membership in America, or their obsession with creating "private" experiences that rely on the benefit of the doubt.

Apple doesn't care about you, the individual. Your value as a singular customer is worthless. They do care about the whole; a whole that governments can threaten to exclude them from if they don't cooperate with domestic surveillance demands. How far off do you really think American iCloud is from China? If Apple is willing to backdoor one server, what's stopping them from backdooring them all? If they're willing to lie about notification security, what's stopping them from lying about server integrity too?

And worst off, Apple markets security. That's it; you can't go verify their veracity outside the dinky little whitepapers they publish. You can't know for sure if they have privacy violation baked-in to their system because you can't actually verify anything. You simply have to guess, and the best guess you can make gets based off whatever Apple markets as "true" to you. In reality, we can do better with security and should probably expect more from one of the largest consumer technology brands in the world. Simply assuming that they aren't violating user privacy is an absurd thing to gamble your security on.

robmccoll
0 replies
4h5m

That's becoming less the case. As Apple's advertising and services revenue grows and hardware sales slow, they have increasing incentive to mine your data the same as any company does. They already use quite a bit data on the location and content personalization front. I would argue that Apple perhaps cares about protecting your data more from malicious third parties (again like any company should - it's never good for FAANG when data leaks or is abused), but they are better at it (and definitely better at marketing it).

theshrike79
6 replies
10h8m

What makes you think that internal access control at Apple is any better

There are multiple verified stories on the lengths Apple goes internally to keep things secret.

I saw a talk years ago about (I think) booting up some bits of the iCloud infrastructure, which needed two different USB keys with different keys to boot up. Then both keys were destroyed so that nobody knows the encryption keys and can't decrypt the contents.

p_l
2 replies
8h40m

The stories about Apple keeping things secret usually go about protecting their business secrets from normal people, up to doing probably illegal actions.

Using deniable, one-time keys etc. are... not that unusual. In fact I'd say I'm more worried about the use of random USB keys there instead of proper KMS system.

(There are similar stories with how doing a cold start can be difficult when you end up with a loop in your access controls, from Google, where a fortunately simulated cold-start showed that they couldn't access necessary KMS physically to bootstrap the system... because access controls depended, after many layers, on the system to be cold-started).

milkshakes
1 replies
8h27m

they used smartcards, not usb keys

p_l
0 replies
7h54m

Which probably were just key transport devices from offline secured KMSes

treprinum
1 replies
4h46m

Destroyed? Where? In all places where they were stored? Or just in some of them? How can you tell? You still need to trust them they didn't copy them somewhere.

theshrike79
0 replies
1h55m

It's impossible to use any technology if you don't trust anyone.

Any piece of technology MAY have a backdoor or secondary function you don't know of and can't find out without breaking said device.

padolsey
0 replies
8h56m

What's funny is that, in all these orgs, it ends up being the low-tech vulns that compromise you in the end. Physical access, social engineering, etc. However, I'm really impressed by the technical lengths Apples goes to though. The key-burning thing reminds me of ICANN' Root KSK Ceremonies.

cdata
0 replies
5h10m

That's not even getting to the fact that Apple is also running a display ads business: https://searchads.apple.com/

verisimi
13 replies
12h1m

It's completely fair, because regardless of third party audits, chips, etc, there are backdoors right along the line, that are going to provide Apple and the government with secret legal access to your data. They can simply go to a secret court, receive a secret judgment, and be authorised to secretly view your data. Does anyone really think this is not already the case? There is no transparency. A licensed third party auditor would not be able to tell you this. We have to operate with the awareness that all data online is already not private - no need to pretend/imagine that Apple's marketing is actually true, and that it is possible to buy online privacy utopia.

theshrike79
6 replies
10h5m

The best protection against "secret orders" is to use mathematics.

Build your system so that it can't be decrypted, don't log anything etc. Mullvad has been doing this with VPNs and law enforcement has tested it - there's nothing for them to get.

Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone.

There's no reason why they wouldn't go to same lengths on their private cloud compute. It's the one thing they can do that Google can't.

verisimi
4 replies
9h27m

Same has been proven with Apple not allowing FBI to open an iPhone, because it'd set a precedent. Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone.

Right, but I have no reason to think that this isn't a marketing ploy either, just another story. There is simply no way that Apple is as big as it is, without providing whatever data the government requires. Corporations and governments are not your friend.

theshrike79
3 replies
8h12m

Apple will obey government orders to give data they have and can access.

No government order short of targeting a specific backdoored update to a specific person will allow them to give data they can't access.

And if you're doing something that can make a TLA force Apple to create a targeted iOS update just for you, it's not something regular people can or should worry about.

Apple keeps normal people safe from mass surveillance, being protected from CIA/NSA required going Full Snowden and it's not a technological problem, you need to change the way you live.

verisimi
2 replies
7h41m

Do you not remember Edward Snowden? Eg this sort of info:

The scandal broke in early June 2013, external when the Guardian newspaper reported that the US National Security Agency (NSA) was collecting the telephone records of tens of millions of Americans.

The paper published the secret court order directing telecommunications company Verizon to hand over all its telephone data to the NSA on an "ongoing daily basis".

https://www.bbc.com/news/world-us-canada-23123964

You seem to think that 10 years, under cover of secret orders, that this is NOT going on now. Not Apple!

People's lovely trusting natures in corporations and government never ceases to amaze me.

theshrike79
1 replies
1h57m

"telephone data" != "contents of every phone call"

verisimi
0 replies
20m

You and I have no idea.

KaiserPro
0 replies
7h10m

Build your system so that it can't be decrypted

Now you can't debug anything.

Mullvad has been doing this with VPNs

Mullvad do not need to store any data at all. Infact any data that they store is a risk. Minimising the data stored minimises their risk. The only thing they need to store is keys.

Look, if you want to ask an AI service if this photo has a dog in, thats simple and requires no state other than the photo. If you want to ask it does it have my dog in, thats a whole 'nother kettle of fish. How do you communicate the descriptors that describe your dog? how do you generate them? on device? that'll drain your battery in a very short order.

Apple not allowing FBI to open an iPhone, because it'd set a precedent

Because they didn't follow process.

Future iPhone versions were made so that it's literally impossible for even Apple to open a locked iPhone.

They don't need to, just hack the icloud backup. plus its not impossible, its just difficult. If you own the key authority then its less hard.

brookst
3 replies
5h12m

If you’re presenting a conspiracy theory, you have to at least poke holes in the claims you consider false.

Under the system described in the linked paper, your scenario is not possible. In fact, the whole thing looks to be designed to prevent exactly that scenario.

Where do you see the weakness? How could a secret order result in undetectable data capture?

verisimi
2 replies
4h57m

No. The information is all out there - secret courts, secret judgements, its all been put out there. I don't need to dissect any technical information, to recognise that I cannot know what I do not know.

In case anyone was uncertain about whether to trust what we are told - we heard that the US government was taping millions of phone records from the Snowden revelations.

So, we are told there are secrets, and we are told that there are mechanisms in place to prevent this information from being made public.

You are also free to believe that the revelations are no longer relevant... I'd like to hear the reason.

IMO - the reverse is the case - in that you need to show why Apple have now become trustworthy. Why would Apple not be subject to secret judgements?

I know there is a lot of marketing spin about Apple's privacy - but do you really think that they would actually confront the government system, in a way that isn't some further publicity stunt? Can one confront the government and retain a license to operate, do you think? Is it not probable that the reality is that Apple have huge support from the government?

Perhaps this kind of idea is hard to understand - that one can make a big noise about privacy, and how one is doing this or that to prevent access, and all the while ensuring that access is provided to authorised parties. Corporations can say this sort of thing with a straight face - its not a privacy issue to private information - its a (secret) legal issue!

Sorry, but secret courts and secret judgements, along with existing disclosure that millions were being spied upon, means one needs to expect the worst.

brookst
1 replies
35m

Fair, go ahead and expect the worse, and handwave away any attempts to mitigate.

But I'm not sure where that leaves you. Is it just a nihilistic "no security matters, it's all a show" viewpoint?

verisimi
0 replies
13m

It is fair, I don't accept attempts to mitigate. The trust is gone, and nothing can recover it. The idea of trusting government and corporations was ridiculous in the first place as these entities are not your friends.

You wouldn't expect a repeat abuser to stop abusing just because of 'time' or a marketing campaign. And yet this is the case here. People keep looking to their tormentors for solutions.

Not expecting healing from those also inflicting the trauma, ie changing one's expectations, seems like a minimum effort/engagement in my view, but it's somehow inconceivable.

dwaite
1 replies
9h27m

Does anyone really think this is not already the case?

I don't think this is already the case, and I think the article is an example of safeguards being put into place (in this particular scenario) to prevent it.

verisimi
0 replies
9h22m

On the basis of not having information, cos all this occurs out of sight, you believe this is not the case. Ok.

Sporktacular
2 replies
9h47m

"ensuring that clients will refuse to talk to non-audited systems."

I'm trying to understand if this is really possible. I know they claim so but is there any info on how this would prevent Apple from executing different code to what is presented for audit?

p_l
0 replies
8h37m

Unless they pass all keys authorized by the system to third parties that ensure appropriate auditing, none.

And at least after my experiences with T2 chip, I consider Apple devices to be always owned by Apple first...

brookst
0 replies
5h15m

The servers provide a hash of their environment to clients, who can compare it to the published list of audited environments.

So the question is: could the hash be falsified? That’s why they’re publishing the source code to firmware and bootloader, so researchers can audit the secure boot foundations.

I am sure there is some way that a completely malevolent Apple could design a weakness into this system so they could spend a fortune on the trappings while still being able to access user information they could never use without exposing the lie and being crushed under class actions and regulatory assault.

But I reject the idea that that remote possibility means the whole system offers no benefit users should consider in purchasing decisions.

1vuio0pswjnm7
1 replies
8h24m

"I think another facet of trust here is that a rather big part of Apple's business model is privacy. They've been very successful financially by creating products that generate money in other ways, and it's very much not necessary or even a sound business idea for them to do something else."

If a third party wants that data, whether the third party is an online criminal, government law enforcement or a "business partner", this idea that Apple's "business model" will somehow negate the downsides of "cloud computing", online advertising and internet privacy is futile. Moreover, it is a myth. Apple is spending more and more on ad services, we can see this in its SEC filings. Before he died, Steve Jobs was named on an Apple patent application for showing ads during boot. The company uses "privacy" as a marketing tactic. There is no evidence of an ideological or actual effort to avoid the so-called "tech" company "business model". Apple follows what these companies do. It considers them competitors. Apple collects a motherload of user data and metadata. A company that was serious about privacy would not do this. It's a cop out, not a trade off.

To truly avoid the risks of cloud computing, online advertising and associated privacy issues, choosing Apple instead of Google is a half-baked effort. Anyone who was serious about it would choose neither.

Of course, do what is necessary, trust whomever; no one is faulting anyone for making practical choices, but let's not pretend choosing Apple and trusting it solves these problems introduced by so-called "tech" company competitors. Apple pursues online advertising, cloud computing and data collection. All at the expense of privacy. With billions in cash on hand, it is one of the wealthiest companies on Earth, does it really need to do that.

In the good old days, we could call Apple a hardware company. The boundaries were clear. Those days are long gone. Connect an Apple computer to a network and watch what goes over the wire wth zero user input, destined for servers controlled by the mothership. There is nothing "private" about that design.

troyvit
0 replies
3h28m

Of course, do what is necessary, trust whomever; no one is faulting anyone for making practical choices, but let's not pretend choosing Apple and trusting it solves these problems introduced by so-called "tech" company competitors. Apple pursues online advertising, cloud computing and data collection. All at the expense of privacy. With billions in cash on hand, it is one of the wealthiest companies on Earth, does it really need to do that.

Yeah. I feel like the conversation needs some guard rails like, "Within the realm of big tech, which has discovered that one of its most profitable models is to make you the product, Apple is really quite privacy friendly!"

devjab
0 replies
11h48m

I think it’s pretty fair. This example isn’t about Apple but about Microsoft, but we’ve had a decade long period where Microsoft has easily been the best IT-business partner for enterprise organisations. I’ve never been much of a fan of Microsoft personally, but it’s hard to deny just how good they are at building relationships with enterprise. I can’t think of any other tech company that knows enterprise the way Microsoft does, but I think you get the point… anyway they too are beginning to “snoop” around.

Every teams meeting we have is now transcribed by AI, and while it’s something we want, it’s also a lot of data in the hands of a company where we don’t fully know what happens with it. Maybe they keep it safe and only really share it with the NSA or whichever American sneaky agency listens in on our traffic. Which isn’t particularly tin-foil-hat. We’ve semi-recently had a spy scandal where it somewhat unrelated (this wasn’t the scandal) was revealed that our own government basically lets the US snoop on every internet exit node our country has. It is what it is when you’re basically a form of vassal state to the Us. Anyway, with the increased AI monitoring tools build directly into Microsoft products, we’re now handing over more data than ever.

To get the point, we’re currently seeing some debate on whether Chromebooks and Google education/workspaces should be allowed in schools. Which is a good debate. Or at least it would be if the alternative wasn’t Microsoft… Because does it really matter if it’s Google or Microsoft that invades your privacy?

Apple is increasingly joining this trend. Only recently it was revealed that new Apple devices have some sort of radio build into them, even though it’s not on their tech sheets. Or in other words, Apple has now joined the trend of devices that can form their own internet by being near other Apple devices. Similar to how Samsung and most car manufacturers have operated for years now.

And again if sort of leads to… does it really matter if it’s Google or Apple that intrudes on your privacy? To some degree it does, of course, I’d personally rather have Microsoft or Apple spy on me, but I would frankly prefer if no one spied on me.

TeMPOraL
10 replies
11h57m

unless it's open source and the servers decentralized, you are always trusting SOMEONE

Specifically, open-source and self-hostable. Open source doesn't save you if people can't run their own servers, because you never know whether what's in the public repo is the exact same thing that's running on the cloud servers.

dwaite
6 replies
9h21m

You can by having an attestation of the signed software components up from the secure boot process, and having the client device validate said attestation corresponds to the known public version of each component, and randomize client connections across infrastructure.

Other than obvious "open source software isn't perfectly secure" attack scenarios, this would require a non-targeted hardware attack, where the entire infrastructure would need to misinterpret the software or misrepresent the chain of custody.

I believe this is one of the protections Apple is attempting to implement here.

andersa
5 replies
8h43m

Usually this is done the other way around - servers verifying client devices using a chip the manufacturer put in them and fully trusts. They can trust it, because it's virtually impossible for you (the user) to modify the behavior of this chip. However, you can't put something in Apple's server. So if you don't trust Apple, this improves the trust by... 0%.

Their device says it's been attested. Has it? Who knows? They control the hardware, so can just make the server attest whatever they want, even if it's not true. It'd be trivial to just use a fake hash for the system volume data. You didn't build the attestation chip. You will never find out.

Happy to be proven wrong here, but at first glance the whole idea seems like a sham. This is security theater. It does nothing.

brookst
2 replies
5h4m

If it is all a lie, Apple will lose so much money from class action lawsuits and regulatory penalties.

It’d be trivial to just use a fake hash

You have to go deeper to support this. Apple is publishing source code to firmware and bootloader, and the software above that is available to researchers.

The volume hash is computed way up in the stack, subject to the chain of trust from these components.

Are you suggesting that Apple will actually use totally different firmware and bootloaders, just to be able to run different system images that report fake hashes, and do so perfectly so differences between actual execution environment and attested environment cannot be detected, all while none of the executives, architects, developers, or operators involved in the sham ever leaks? And the nefarious use of the data is never noticed?

At some point this crosses over into “maybe I’m just a software simulation and the entire world and everyone in it are just constructs” territory.

andersa
1 replies
4h53m

I don't know if they will. It is highly unlikely. But theoretically, it is possible, and very well within their technical capabilities to do so.

It's also not as complicated as you make it sound here. Because Apple controls the hardware, and thus also the data passing into attestation, they can freely attest whatever they want - no need to truly run the whole stack.

brookst
0 replies
3h31m

It is as complicated as I make it sound. Technically, it's trivial, of course.

But operationally it is incredibly complicated to deliver and operate this kind of false attestation at massive scale.

p_l
1 replies
8h35m

Usually the attestation systems operate on neither side having everything to compute a result that will match attestation requirements, and thus require that both server-side and client-side secret are involved in attestation process.

The big issue with Apple is that their attestation infrastructure is wholly private to them, you can't self-host (Android is a bit similar in that application using Google's attestation system have the same limitation, but you can in theory setup your own).

andersa
0 replies
8h19m

Attestation requires a root of trust, i.e. if data hashes are involved in the computation, you have to be able to trust that the hardware is actually using the real data here. Apple has this for your device, because they built it. You don't have it for their server, making the whole thing meaningless. The maximum information you can get out of this is "Apple trusts Apple".

Under the assumption that Apple is telling the truth about what the server hardware is doing, this could protect against unauthorized modifications to the server software by third parties.

If however, we assume Apple itself is untrustworthy (such as, because the US government secretly ordered them to run a different system image with their spyware installed) then this will not help you at all to detect that.

jjav
1 replies
11h15m

exact same thing that's running on the cloud servers

What runs on the servers isn't actually very important. Why? Becuase even if you could somehow know with 100% certainty that what a server runs is the same code you can see, any provider is still subject to all kinds of court orders.

What matters is the client code. If you can audit the client code (or better yet, build your own compatible client based on API specs) then you know for sure what the server side sees. If everything is encrypted locally with keys only you control, it doesn't matter what runs on the server.

flakeoil
0 replies
5h26m

But in this use case of AI in the cloud I suppose it's not possible to send encrypted data which only you have the keys to as that makes the data useless and thus no AI processing in the cloud can be made. So the whole point of AI in the cloud vs. AI on device goes away.

nardi
0 replies
11h19m

This is what the “attestation” bit is supposed to take care of—if it works, which I’m assuming it will, because they’re open sourcing it for security auditing.

nl
3 replies
7h37m

This isn't right.

If you trust math you can prove the software is what they say it is.

Yes it is work to do this, but this is a big step forward.

ADeerAppeared
1 replies
7h0m

The only thing the math tells you is that the server software gave you a correct key.

It does not tell you how it got that key. A compromised server would send you the key all the same.

You still have to trust in the security infrastructure. Trust that Apple is running the hardware it says it is, Trust that apple is running the software it says it is.

Security audits help build that trust, but it is not and never will be proof. A three-letter-agency of choice can still walk in and demand they change things without telling anyone. (And while that particular risk is irrelevant to most users, various countries are still opposed to the US having that power over such critical user data.)

nl
0 replies
5h43m

No, this really isn't right.

To quote:

verifiable transparency, goes one step further and does away with the hypothetical: security researchers must be able to verify the security and privacy guarantees of Private Cloud Compute, and they must be able to verify that the software that’s running in the PCC production environment is the same as the software they inspected when verifying the guarantees.

So how does this work?

The PCC client on the user’s device then encrypts this request directly to the public keys of the PCC nodes that it has first confirmed are valid and cryptographically certified. This provides end-to-end encryption from the user’s device to the validated PCC nodes, ensuring the request cannot be accessed in transit by anything outside those highly protected PCC nodes

Next, we must protect the integrity of the PCC node and prevent any tampering with the keys used by PCC to decrypt user requests. The system uses Secure Boot and Code Signing for an enforceable guarantee that only authorized and cryptographically measured code is executable on the node. All code that can run on the node must be part of a trust cache that has been signed by Apple, approved for that specific PCC node, and loaded by the Secure Enclave such that it cannot be changed or amended at runtime.

But why can't a 3-letter agency bypass this?

We designed Private Cloud Compute to ensure that privileged access doesn’t allow anyone to bypass our stateless computation guarantees.

We consider allowing security researchers to verify the end-to-end security and privacy guarantees of Private Cloud Compute to be a critical requirement for ongoing public trust in the system.... When we launch Private Cloud Compute, we’ll take the extraordinary step of making software images of every production build of PCC publicly available for security research. This promise, too, is an enforceable guarantee: user devices will be willing to send data only to PCC nodes that can cryptographically attest to running publicly listed software.

So your data will not be sent to node that are not cryptographically attested by third parties.

These are pretty strong guarantees, and really make it difficult for Apple to bypass.

It's like end-to-end encryption using the Signal protocol: relatively easy to verify it is doing what is claimed, and extraordinarily hard to bypass.

Specifically:

The only thing the math tells you is that the server software gave you a correct key.

No, this is secure attestation. See for example https://courses.cs.washington.edu/courses/csep590/06wi/final... which explains it quite well.

The weakness of attestation is that you don't know what the root of trust is. But Apple strengthens this by their public inspection and public transparency logs, as well as the target diffusion technique which forces an attack to be very widespread to target a single user.

These aren't simple things for a 3LA to work around.

robmccoll
0 replies
4h1m

It's not fully homomorphic encryption. The compute is happening in the plain on the other side, and given the scale of models they are running, it's not likely that all of the data involved in a computation is happening inside a single instance of particularly secure and hardened hardware. I don't think it's reasonable for most individuals to expect to be protected from nation-state actors or something, but their claims seem a little too absolute to me.

underdeserver
0 replies
35m

Unless you personally validate hardware designs, manufacturing processes, and all software, even when running locally you're trusting many, many people.

seydor
0 replies
12h45m

if wanted.

Or if someone compels them to

mbesto
0 replies
2h47m

They already have your private pictures. What difference is it that it's now running AI?

detourdog
0 replies
10h21m

If one has to use tech one has to trust someone. Apple has focused on the individual using computers since inception. They have maintained a consistent message and have a good track record.

I will trust them because the alternatives I see are scattered and unfocused.

BenFranklin100
11 replies
16h37m

If he really wanted no one to be reading his tweets he’d be using BluSky or Masto…

unshavedyak
7 replies
15h59m

Is there more to that thread? I can't read it if it exists, not sure if that is what the parent is talking about? But i don't have a Twitter account anymore, so maybe it's locked?

qingcharles
3 replies
15h46m

I don't know what you're seeing. It's a very long thread. Exceptionally good take on the whole thing. Apple has gone way out of their way to try and sell this thing. Above and beyond compared to how I imagine Microsoft or Google would have tackled this.

zooq_ai
2 replies
14h57m

If your AI model sucks, you have to use other gimmicks to lure customers. That's marketing 101.

Create irrational fear about piracy, push privacy focused products and profits as the sheeple promptly fall for this

astrange
1 replies
11h1m

I've never seen someone use "sheeple" in an anti-privacy argument.

zooq_ai
0 replies
1h41m

the most successful sheeple operation is the one the sheeple and the entire world is completely oblivious of it.

jokes aside, this is no different from people selling bunker beds, gold, ammunition, crypto, vpns. It is specifically for the set of gullible people who think they and their data is so important. Reality (except for 10,000 people or so) is, most lives and their 'precious' data is worthless. (I'm not talking about SSN, Bank Accounts -- those are well protected by tech cos HN seem to hate on)

capybara_2020
2 replies
15h34m

Without being logged into X, you can only see the first post in a thread.

jjav
1 replies
11h13m

Not even that anymore, all links show is "Something went wrong, but don’t fret — let’s give it another shot."

Impossible to see any content.

AnonC
0 replies
10h23m

That's likely due to tracking prevention or protection by your browser because X really, really wants to track you. If you disable the tracking protection and related settings, you may be able to see the single tweet.

rmm
0 replies
15h49m

Ok that made me spill my coffee.

Andrex
0 replies
3h27m

Or maybe (gasp!) a blog?

transpute
10 replies
18h27m

Thanks for the link.

> As best I can tell, Apple does not have explicit plans to announce when your data is going off-device for to Private Compute. You won't opt into this, you won't necessarily even be told it's happening. It will just happen. Magically.

Presumably it will be possible to opt out of AI features entirely, i.e. both on-device and off-device?

Why would a device vendor not have an option for on-device AI only? iOS 17 AI features can be used today without iCloud.

Hopefully Apple uses a unique domain (e.g. *.pcc.apple.com) that can be filtered at the network level.

wmf
3 replies
16h48m

If you have no threat model and want to opt out of random features just because... you probably shouldn't use Apple products at all. Or Google or Microsoft.

transpute
2 replies
16h44m

For years, Apple has a documented set of security policies to disable off-device processing (e.g iCloud, Siri), via MDM / Apple Configurator. Apple also published details needed for enterprise network filtering to limit Apple telemetry, if all you want from Apple servers are software security updates and notifications.

With a hardened configuration, Apple has world-class device security. In time, remote PCC may prove as robust against real-world threats. Until then, it would be good to retain on-device security policy and choice for remote computation.

sneak
1 replies
13h3m

Apple does not publish details to limit telemetry. Nowhere in MDM or in their docs do they tell you that you can safely block xp.apple.com (telemetry) but not gs.apple.com (boot ticket signing server for updates).

transpute
0 replies
11h25m

Thanks, both are listed as required for software updates, https://support.apple.com/en-us/101555

Is there a good non-Apple reference for the functions performed by their servers?

azinman2
2 replies
17h1m

You would have to activate a clearly LLM-powered software feature and have internet access. I don't know if settings will appear to disable this, but you could imagine it would be the case. This isn't just siphoning off all your data at random.

transpute
1 replies
16h56m

Would Spotlight be considered a "clearly LLM-powered software feature"? Will there be an option for "non-AI Spotlight"? Disabling dozens of software features, or identifying all apps which might use LLM services, is a daunting proposition. It would be good to have a PCC kill switch, which makes opt-in usage meaningful, rather than forced.

chefandy
0 replies
16h11m

Privacy "consent" is fundamentally broken. We've moved from "we're doing whatever the fuck we want" to "we're doing whatever the fuck we want, but on paper it's whatever the fuck you expressly asked for, whether you wanted to or not."

sneak
1 replies
13h4m

Almost certainly you will be able to disable it entirely and hide the UI to re-enable it via provisioning profiles via Apple Configurator 2 or MDM.

This is actually what you have to do now if you don’t want Siri and Mail to leak your address book to Apple.

transpute
0 replies
11h23m

> if you don’t want Siri and Mail to leak your address book to Apple.

By disabling Siri and iCloud, or other policies?

onel
0 replies
10h30m

I think the main reason might be the on-device AI is fairly limited features wise. For Apple to actually offer something useful they would need to switch between device/server constantly and they don't want to limit the product by allowing users to disable going to a server.

With OpenAI calls is different because the privacy point is stronger

tantalor
4 replies
16h33m

nobody can read his tweets without a X account

False; works fine for me logged out or incognito..

unshavedyak
0 replies
15h52m

Also can't see the thread.

steg132
0 replies
16h31m

I’m on iOS. I can’t see the thread. Incognito or normally.

OneLeggedCat
0 replies
15h17m

False

firecall
4 replies
16h23m

Threads also is popular.

Probably the mainstream Twitter alternative at this point?

jxi
3 replies
14h44m

Threads is far from mainstream and just filled with spam and OnlyFans spammers at this point.

threeseed
0 replies
10h48m

By every metric Threads is mainstream:

a) Top 10 App Store charts in every country.

b) Heavily promoted through Facebook and Instagram.

c) DAUs are higher than X.

fragmede
0 replies
12h27m

weird, i get a bunch of music and programming stuff on my Threads feed. it's not very deep, but what's on the surface is quite nice and not a bunch of almost-porn. Twitters become half porn though

JimDabell
0 replies
3h35m

That sounds far more like Twitter than Threads. I get so much spam on Twitter now that I hit rate limits reporting it all.

vaylian
2 replies
12h52m

(I wonder if Matt realizes nobody can read his tweets without a X account? Use BlueSky or Masto man)

He actually has an active Mastodon account, but this particular story is not on there (yet): https://ioc.exchange/@matthew_d_green

Tepix
1 replies
12h7m

Inactive since 2 months

Tepix
1 replies
12h9m

Thanks. I wonder how long that service is going to last.

dmix
0 replies
4h7m

its been around a loong time

wslh
1 replies
15h40m

Beyond all the hardware complexity, another attack vector is the network infrastructure.

astrange
0 replies
11h0m

That is covered in the article.

stavros
0 replies
11h52m

He's not wrong that, given that you want to do this, this is the best way. The alternative would be to not do it at all (though an opt-out would have been good).

brigandish
0 replies
17h15m

These two tweets stand out for me:

Ok there are probably half a dozen more technical details in the blog post. It’s a very thoughtful design. Indeed, if you gave an excellent team a huge pile of money and told them to build the best “private” cloud in the world, it would probably look like this.

and

And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal.

I'd prefer things stay on the device but at least this is a big commitment in the right direction - or in the wrong direction but done better than their competitors, I'm not sure which.

ein0p
27 replies
16h38m

It is not possible for this to be fully private in the United States because the government not only can force Apple to open up the kimono, it can also forbid it to talk about it. There’s not really anything Apple can do to work around this “limitation”. Thank your “representative” for extending the PATRIOT Act when you get a chance.

amiantos
6 replies
15h2m

Private Cloud Compute servers have no persistent storage so there would be nothing to see upon opening the kimono. You'd need some sort of government requested live wire tap thing to harvest the data out of the incoming requests, which might be a different situation. I'm, of course, just some dude on the internet, thinking up a counter-point to this concern, who knows if I am even remotely in the right ballpark.

visarga
2 replies
14h21m

mandatory 30 day retention policies or something like it

theshrike79
1 replies
9h56m

You can't mandate retention on stuff you're not storing anyway - or because of encryption can't store.

transpute
0 replies
11h19m

> have no persistent storage

How often do PCC servers reboot and wipe the temporary encryption key?

KaiserPro
0 replies
7h3m

Private Cloud Compute servers have no persistent storage so there would be nothing to see upon opening the kimono

It doesn't actually say there is no persistent storage, it says that the compute node will not store it for longer than the request. There's nothing to stop the data coming from a datastore outside of the "PCC" in another part of apple's infrastructure.

paradite
5 replies
14h32m

Slightly off-topic, "open up the kimono" sounds disturbing and creepy to me as an Asian. I suspect I'm not alone in this.

bn-l
1 replies
12h15m

Is this phrase worth an entire long form blog post from NPR?

rpastuszak
0 replies
10h30m

Honestly, why not? I love reading about etymologies and I know that many people here do as well.

_heimdall
0 replies
5h46m

I'm not sure how I've been in tech since before this article was written and today is the first time I've ever even seen/heard this phrase.

ein0p
0 replies
13h56m

That’s even better. I do think it’s disturbing and creepy when someone goes through my private data without my knowledge.

tantalor
3 replies
16h26m

Warrant canary

sneak
0 replies
12h59m

The orders in question aren’t search warrants and don’t require probable cause.

70,000+ Apple user accounts are surveilled in this manner every year.

egorfine
0 replies
9h7m

It makes zero sense for a company of this size. I bet they are served with gag orders like daily, so the warrant canary is going to expire the moment it is published.

afh1
0 replies
15h50m

Apple removed theirs year ago.

ls612
3 replies
14h39m

What Apple can do (and appears to be doing throughout its products) is not have the data requested. Or not have it in cleartext. NSLs can't request data that doesn't exist anymore.

ein0p
2 replies
13h28m

LLMs work on clear text inputs

ls612
1 replies
13h24m

But the setup is that Apple doesn't know which cleartexts currently being processed are associated with which user meaning that even live surveillance can't work without surveilling everybody, making any such program very quickly discoverable. Read the section about non-targetability in the link.

Apple deserves credit for correctly analyzing their threat model and designing their system accordingly.

ein0p
0 replies
12h3m

I’m willing to concede that Apple’s system is the best designed of the bunch. I’m not willing to call it “private”, however, if it processes unencrypted inputs in the jurisdiction of a nation state with pervasive government surveillance.

aalimov_
2 replies
15h2m

Their post sure makes it seam like it’s possible. Was there something that stood out to you?

quenix
1 replies
12h49m

I mean, the software running on the client (phone/mac/ipad) is closed-source and, if we assume Apple is compromised, can be made to circumvent all of these fancy protections at the push of a button.

If pressured by the government, Apple can simply change the client software to loosen the attestation requirements for private compute. And that would be the most inconspicuous choice.

rvnx
0 replies
12h26m

Or target a device by IMEI or iCloud to be candidate to receive a software update, and push an update that sends data to "dev-llm-assistant.ai.apple.com".

"oh it's our dev version ? what's the problem ? we need data access for troubleshooting"

sciolist
1 replies
16h6m

There's a difference between guaranteed privacy and certifiable privacy. Yes, the government can request one's data. However, Apple's system would reveal those intrusions to the public, even if Apple themselves couldn't say it.

afh1
0 replies
15h50m

How?

sneak
0 replies
13h0m

It seems to me that this security architecture is a direct response to the hostile regulatory environment Apple finds themselves in wrt USA PATRIOT and the CCP et al.

zmmmmm
21 replies
14h34m

Read through it all, it still comes down to "trust us". Apple can sign and authorise an update at any time that will backdoor it, and the government is the stroke of a pen away from forcing them to, all completely silently.

I get that there's benefit to what they are doing. But the problem of selling a message of trust is you absolutely have to be 100% truthful about it, and them failing to be transparent that people's data is still subject to access like this poisons the larger message they are selling.

troad
18 replies
14h18m

They already have root. Their software is closed source. There is absolutely nothing stopping them from uploading all of your data right now.

If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.

__MatrixMan__
12 replies
14h2m

That's true, but also it should be possible to make an OS that people can trust without trusting you, and as users we should encourage movement in that direction.

EternalFury
5 replies
13h45m

Good luck. It’s much easier to talk about it. The last open OS I have seen reach a semi-mainstream level of adoption was started in the early 90’s, more than 30 years ago, by some Linus guy.

fastball
4 replies
11h50m

And (basically) nobody running linux is individually verifying the source code of every little piece of software that goes into it (maybe Linus is), so you're still trusting someone.

detourdog
3 replies
10h11m

I can’t even find the appropriate documentation.

tsimionescu
1 replies
9h16m

How would documentation help you trust the code?

detourdog
0 replies
9h3m

How does one understand what is going on with out a clear set of documentation? Either one is so smart they can review everyline of everything they are running. My contention is that it is hard enough to find authoritative documentation much less developing an understanding of the code needed or running.

The choice is either many little trust relationships or a giant leap of faith. I feel better served by a giant leap of faith and access to all the technology I can use.

I prefer no documentation and consistent behavior to no documentation and a bunch of internet howtos on what might work.

dialup_sounds
0 replies
8h43m

It's all on Discord now.

abtinf
2 replies
13h49m

should be possible

What makes you think this?

brookst
0 replies
4h57m

It’s especially funny because I believe it is provably impossible. You’ll have to trust me that I’ve done the proof.

__MatrixMan__
0 replies
4h36m

Because there are so freaking many of us, and some of us trust each other. If we were better at coordinating about which parts of the code we trust and to what degree, we could determine which parts of it are untrustworthy and patch the problem out of it.

The GrapheneOS people are doing this, for example. It's not crazy to consider your device vendor as part of your threat model, because like it or not, they are a threat.

troad
1 replies
13h45m

I understand the sentiment, but it's impractical to live in a trust-less society. If you've ever had dental work done, you've put an awful lot of faith in a stranger pushing a drill into your head. Ditto for riding buses and bus drivers, etc etc.

Trust can be abused, certainly, but it also allows collaboration and specialisation, and without those I doubt we'd have gotten very far.

__MatrixMan__
0 replies
4h32m

I'm happy to trust many kinds of people, dentists included, just not the kind people who find themselves at the helm of companies like Apple and Google.

Better to trust many people narrowly (e.g. I don't trust the bus driver to drill my cavities) than to trust a small handful of people broadly (e.g. like Apple expects of their users).

Aerbil313
0 replies
4h49m

Any kind of practical OS would contain code of unpractical amounts to manually review and audit.

That's not to mention the argument that any software of a LOC count of higher than some number is impossible to audit because of complex state handling. Rice's Theorem applies to your brain too, probably, to some extent. Idk about purely functional Haskell though.

jchw
3 replies
13h44m

That's true, but if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere. The trouble with remotely pushing data somewhere is that you can't tell if anything has changed even if you wanted to. (Attestation only works if it's not compromised, and for obvious reasons, there's no way to know that an attestation mechanism is compromised.)

That said I really don't disagree with this point at all in terms of it being a valid problem. It's not a fixable problem either (it comes down to, again, building trustworthy computers) but it could be biased way towards being solved whereas today it is still "trust me bro". I don't think Apple will be the company to make progress towards this, though.

troad
2 replies
13h35m

if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere

If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.

I don't think Apple will be the company to make progress towards this, though.

I agree.

jchw
1 replies
13h27m

If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.

Y'know though, when you put it that way, it sounds inherent that security vulnerabilities will pop up, which is kinda true, at least for the foreseeable future, but to be pedantic, the security vulnerabilities are already there, it's discovering them that's the problem. If we could make secure computers... (time to formally prove everything from the ground up I guess.)

But, that said, I wasn't overlooking this, I'm just looping "getting pwned" into "active involvement". If you have some sufficiently isolated machines, they're probably fine indefinitely. The practicality of this is limited outside of thought experiments. However it's definitely worth noting that unlike a compromised remote, it is at least technically feasible to work on the problem of making local compromise more evident, whereas a remote compromise is truly impossible to reliably be able to detect from the outside.

troad
0 replies
13h9m

If you have some sufficiently isolated machines, they're probably fine indefinitely.

The eternal dream of unplugging, and living free on Amigas.

Vegenoid
0 replies
13h49m

While true, the gap between "we send your data to our datacenters but we don't look at it" to "we look at it a little bit without telling you" is much smaller than "we leave your data on your device alone" to "we upload data from your device", both on a technical and policy level.

Even if the org has been trustworthy to this point, I think this step makes it more likely (maybe still unlikely, but more likely) that in the future they do look at your data, as less things have to change for that to happen.

buzzerbetrayed
1 replies
14h25m

Your argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical.

Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well:

Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good.

In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
devjab
0 replies
11h36m

I think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part:

As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.

Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.

zer00eyz
19 replies
18h14m

I have a big question here.

Who is this for? Dont get me wrong I think it's a great effort. This is some A+ nerd stuff right here. It's speaking my languge.

But Im just going to figure out how to turn off "calls home". Cause I dont want it doing this at all.

Is this speaking to me so I tell others "apple is the most secure option"? I don't want to tell others "linux" because I don't want to do tech support for that.

At this point I feel like an old man shouting "Dam you keep your hands off my data".

al_borland
6 replies
16h50m

Apple needs to differentiate itself, and they have chosen privacy as a way to do that, which I'm all for. The headlines around Microsoft's AI efforts have largely been a nightmare, with a ton of bad press. If the press around Apple's AI is all about how over the top they went with security and privacy, that will likely make people feel a little better about using it.

I'm not a big user of OpenAI's stuff, but if I was going to use any of it, I'd rather use it through Apple's anonymizing layer than going directly to OpenAI.

gpm
5 replies
14h43m

I actually thought one notable thing in the presentation was that they spent all this time talking about their new private cloud compute architecture.

And then showed that they have a prompt asking if you're ok sending the data to OpenAI. Presumably because despite OpenAI promising not to use your data (a promise apple relayed) OpenAI didn't buy into this new architecture.

al_borland
3 replies
14h17m

Thank you for mentioning this. I thought I was going crazy, because I heard this too, but kept seeing comment after comment on other sites asking if a person could choose not to use OpenAI, or that it was happening magically in the background. The way I heard it, the user was in control.

I think this goes back to what Steve said in 2010.

https://youtube.com/watch?v=Ij-jlF98SzA

And yes, while the data might not be linked to the user and striped of sensitive data, I could see people not wanting something very personal things to go to OpenAI, even if there should be no link. For example, I wouldn’t want any of my pictures going to OpenAI unless I specifically say it is OK for a given image.

manmal
1 replies
5h54m

I’m not sure but I thought I saw it mentioned that OpenAI is still allowed to train on the data received from Apple customers.

FumblingBear
0 replies
3h27m

Actually the opposite. They’re explicitly not allowed to.

rekoil
0 replies
8h47m

I was under the impression that the OpenAI integrations were more about content generation and correction than the Apple Intelligence-driven personal stuff.

brookst
0 replies
4h47m

Different features.

OpenAI provides the chatbot interface we all know.

The PCC cloud serves all of the other integrated AI features like notification prioritization, summarization, semantic search, etc. At least when those can’t be run on device.

wmf
4 replies
17h27m

What if you can't turn it off and this extreme security is the justification for why?

transpute
2 replies
17h16m

If user data disclosure is forced, would user data be limited to PCC nodes located within the same legal jurisdiction, e.g. EU, UK, US, China, etc?

wmf
1 replies
16h53m

PCC is as government-proof as the iPhone itself so jurisdiction may not matter much.

transpute
0 replies
16h47m

Some jurisdictions require data to be processed within the jurisdiction.

m463
0 replies
17h16m

what happens when your apple id is turned off?

theshrike79
2 replies
9h54m

NSA already has all our data and if they don't, they have direct contacts at Meta and Alphabet to get it same-day delivery.

I'm trusting Apple more in this case, they have an incentive to keep things private and according to experts they're doing everything they can to do so.

"Indeed, if you gave an excellent team a huge pile of money and told them to build the best “private” cloud in the world, it would probably look like this." - Matthew D. Green

_heimdall
1 replies
4h50m

The NSA partners directly with telecoms companies, especially AT&T. Its easier when companies like Meta and Facebook will play along, but that's not the only way they get access to a bunch of our data.

theshrike79
0 replies
1h57m

How do telecom companies unravel public key encryption in transit?

hapticmonkey
1 replies
12h52m

It's for shareholders. Microsoft and Nvidia have a bigger market cap than Apple now, thanks to the AI investor boom. Apple need to show they can be all about AI, too. But Apple have the institutional culture to maintain privacy.

written-beyond
0 replies
10h2m

This is exactly what I was thinking during the entire keynote. It was blatantly the WWSC (worldwide shareholder conference) and hackernews commenters are eating it up.

Don't get me wrong, I've always appreciated apples on device ml/AI features, those have always been powerful, interesting and private but these announcements feel very rushed, it's literally a few weeks after Microsoft's announcements.

They've basically done almost exactly what Microsoft announced with a better UX and a pinky promise about privacy. How are they going to pay for all of that compute? Is this going to be adjusted into the price of the iPhones and MacBook? and then a subscription layer is going to be added to continue paying for it? I don't feel comfortable with the fact that my phone is basically extending it's hardware to the cloud. No matter how "private" it is it's just discomforting to know that apple will be doing inference on things seemingly randomly to "extend" compute capabilities.

Also what on earth is apple high on, integrating a third party API into the OS, how does that even make sense. Google was always a separate app, or a setting in safari, you didn't have Google integrated at an OS level heck you don't have that on Android. It feels very discomforting to know that today my phone could phone home to somewhere other than iCloud.

lurking_swe
0 replies
12h53m

Me

I don’t care if the government has access to the data. I just don’t want “bad actors” (scammers, foreign governments, ad-tech companies, insurance companies, etc) to have access to my private data. But i also want the power of LLM’s. Does that sound so far fetched?

I’m a realist. I already EXPECT the US govt has all my data. I don’t like the status quos, but it is what is is.

fundad
0 replies
3h10m

It’s for their competitors who have pushed a narrative that Apple was caught “flat-footed”. Well there is actually a line of LLM and cloud infrastructure at iPhone scale. This is not merely 2 years of work. They push the privacy because it’s expected of them.

Gemini could claim privacy but I think people would assume that if true, it would make it less effective.

SirensOfTitan
16 replies
17h59m

What I'm most curious about here is if a state actor comes to Apple with a subpoena and compels them to release information on an individual, what would Apple be able to release?

... I suppose this is ultimately a question that will be tested sooner or later in the US.

gpm
6 replies
16h43m

Probably everything uploaded after the intercept is in place if you can convince a court to compel it.

One option is to release a malicious software update, sign it, publish the signature on the public chain, and then simply not release the binaries until after whatever associated gag orders there are (if any) expire. Apple gave themselves a 90 day timeline for this before they'd even be in violation of their promises.

Another option is to use the cryptographic keys used to make the hardware that attests to the software running on it, to simply falsely attest to what software is running. Unless Apple's somehow moved those keys outside of the courts jurisdiction (which means outside of Apple's control in the case of most courts) that should be within the courts power. If they can still create new hardware, it seems likely whoever is making that hardware must still have access to the keys...

Both of these attacks are outside the "threat model" proposed, because they are broad compromises against the entire PCC infrastructure. The fact that they are possible and within the legal systems power... well... why are we advertising this as secure again?

The main value of this whole architecture in my mind isn't actually security though, it's that it's Apple implicitly making the promise that they won't under any circumstance use the data, or let anyone else use the data, for business purposes (not even for running the service itself).

aalimov_
5 replies
14h17m

One option is to release a malicious software update, sign it, publish the signature on the public chain,

In this option it would be Apple releasing a malicious software update?

If they can still create new hardware, it seems likely whoever is making that hardware must still have access to the keys...

This option reads like the keys are stored in apple-keys.txt

Both of these attacks are outside the "threat model" proposed, because they are broad compromises against the entire PCC infrastructure

They mentioned that the in-depth write up will be shared later, might they still address this concern in writing? Your wording makes you sound so certain, but this is just a broad overview. How are you so sure?

gpm
4 replies
13h41m

In this option it would be Apple releasing a malicious software update?

Yes, compelled by something like the all writs act (if the US is the one doing the compelling).

This option reads like the keys are stored in apple-keys.txt

They probably are. That file might live on a CD drive in a safe that requires two people to open it, but ultimately it's a short chunk of binary data that exists somewhere (until it is destroyed)...

might they still address this concern in writing?

Can I say beyond all doubt that this won't happen? Of course not.

On the first approach I'm quite confident though, because it's both the type of attack they discuss in their initial press release, and pretty fundamental to and explicitly allowed by their model of updating the software.

On the second approach I'm reasonably confident. Like the first issue it's the type of issue that they were discussing in their initial press release. Unlike the first issue it's not something that is explicitly allowed in the model. If Apple can find a way to make the attestation keys irretrievable while still allowing themselves to manufacture hardware I believe they'd do it - I just don't see a method and think it would have warranted a mention if they had one. I tried to insert a level of uncertainty in my original writing on this one because I could be missing a way to solve it.

Ultimately I'd rather over-correct now then have people start thinking this is going to be more secure than it is and then have some fraction of them miss the extremely-likely follow up of "and we could be compelled to work around our security".

gpm
2 replies
5h51m

I'm well aware of these. They don't solve the problem at hand. You need a way to put keys into new hardware. Thus you need a way to get keys out of wherever you've stored your cryptographic material. Thus it can't be on a HSM (or it can be if it's a master key signing child keys, but in that case the attack only needs a signed child key).

lgg
1 replies
3h46m

From: https://support.apple.com/guide/security/secure-enclave-sec5...

“A randomly generated UID is fused into the SoC at manufacturing time. Starting with A9 SoCs, the UID is generated by the Secure Enclave TRNG during manufacturing and written to the fuses using a software process that runs entirely in the Secure Enclave. This process protects the UID from being visible outside the device during manufacturing and therefore isn’t available for access or storage by Apple or any of its suppliers.“

gpm
0 replies
3h28m

Sure, and even Apple can't imitate a different server that they made.

They're making new servers though. Take the keys that are used to vouch for the UIDs in actual secure enclaves, and use them to vouch for the UID in your evil simulated "secure" enclave. Your simulated secure enclave doesn't present as any particular real secure enclave, it just presents as a newly made secure enclave that Apple has vouched for as being a secure enclave.

theshrike79
0 replies
9h50m

Push notifications are on a whole different level than "full access to your phone".

astrange
0 replies
10h50m

That's an ordinary subpoena and that data is not being specially collected and not e2ee encrypted. Has nothing to do with the guarantee in this article.

ricardobeat
1 replies
17h38m

Unless their statements regarding the design of these systems are blatantly false, or they are forced to add data collectors on purpose to target individuals, the answer is close to nothing.

You can opt into full E2E encryption [1] which makes it nothing, presumably at the cost of some convenience features.

[1] https://support.apple.com/en-us/108756

onesociety2022
0 replies
15h54m

PCC exists because full E2E is not feasible for these use cases. The LLM has to take your personal data (context window and prompt) to process it.

m463
1 replies
17h15m

The more common use case will be when you're locked out of "your" apple id.

gpm
0 replies
16h3m

The design appears to be entirely ephemeral. There's no personal data to recover here from "your" apple id.

throwaway41597
0 replies
16h1m

I'm very curious as well because my very limited understanding tells me the answer is nothing. The relay hides your identity. Your phone checks the attestations so it won't send your data to servers not running the published software which ensures encryption keys are ephemeral. Once your session is done, the keys are deleted.

Law enforcement would need to seize the right server among millions while it's processing your request and perform an attack on it to get the keys before they're gone.

My next question is what happens if/when the attestation keys are stolen.

jeffbee
10 replies
19h32m

A lot of this sounds like Apple has been 10-20 years behind the state of the art and now wants to tell you that they partially caught up. Verifiable hardware roots of trust and end-to-end software supply chain integrity are things that have existed for a while. The interesting part doesn't come until the end where they promise to publish system images for inspection.

ethbr1
7 replies
19h27m

Do you have analogous search terms for Microsoft, Alphabet, Google, and Amazon's approaches?

Your comment makes me curious on how guarantee-to-guarantee looks (and associated architectures).

threeseed
1 replies
19h4m

Apple's system goes further by having incoming requests choose and verify a server and then encrypt itself using the public key of the node to prevent MITM attacks.

And a one-time credential to prevent replay attacks.

As well as minor things like obfuscating IP addresses, metadata etc.

aaomidi
0 replies
18h54m

Apples system is also the entire pipeline. Borg SREs can still change behavior here. It’s a lot better than what most places have but does not go far enough.

sodality2
0 replies
18h1m

None for consumer-facing products, though

candiddevmike
1 replies
19h17m

Most of the stuff in the blog post reads like common security precautions: don't run as root, stateless immutable nodes, use secure boot, etc. All wrapped up in some Apple marketing pizzazz.

ignoramous
0 replies
18h47m

common security precautions ... marketing pizzazz.

If it were this common, Meta, Google, and others would have announced or launched something similar for its consumer apps/services; I can't seem to recall anything of note.

saagarjha
0 replies
14h52m

Perhaps one of these days we'll get a 'jeffbee that realizes that Google is not actually ahead of everyone in everything all the time. But not today, I guess.

7e
0 replies
18h11m

This is not true at all. Apple is the first to roll out end to end remote attestation of an enclave that includes an ML accelerator in the root of trust, with public verifiability of the entire stack. They are way ahead.

bayareabadboy
9 replies
17h44m

What are the longer term implications that Apple is doing this on their own hardware and not Nvidia? This seems like a big thing to me, an idiot.

wmf
4 replies
17h25m

If you're one of the richest companies in history you can "simply" invest 15 years into developing your own chips instead of buying Nvidia GPUs.

transpute
3 replies
17h10m

> simply invest 15 years into developing your own chips instead of buying Nvidia GPUs

https://www.notebookcheck.net/Apple-and-Imagination-strike-G...

  Following the loss of Apple, easily its biggest client, Imagination was bought out by a Chinese-based investment group. Apple subsequently released its first in-house designed mobile GPU as part of the A11 Bionic SoC that powered the iPhone X.. The new “multi-year license agreement” gives Apple official access to much wider range of Imagination’s mobile GPU IP as well as its AI technologies. The A11 Bionic also included the first neural processing engine in an iPhone
https://9to5mac.com/2020/01/01/apple-imagination-agreement/

  Apple described Imagination’s characterizations as misleading while hiring Imagination employees to work for Apple’s GPU team in the same community.

wmf
2 replies
16h55m

That probably has nothing to do with the Neural Engine though.

transpute
1 replies
16h48m

Probably a coincidence that Apple GPU and NPU both appeared at the same time (A11).

jrk
0 replies
15h13m

It is a coincidence. They are unrelated hardware blocks and very different architectures.

onesociety2022
1 replies
15h57m

But this is just inference. What did they use to train their foundation models?

theshrike79
0 replies
9h51m

The M-series CPUs are stupidly effective in LLM operations. Even my relatively old M1 mac mini can do decent speeds of 7B models.

And Apple clearly has made some custom server hardware and slapped a ton of them on a board just to do the PCC stuff.

ls612
0 replies
14h35m

Apple doesn't want to pay the Jensen Leather Jacket Fee and has $200 billion in cash it is sitting on to make it happen. If anyone can create an Nvidia substitute for AI chips its Apple and their cash hoard combined with their world-class design team and exclusive access to all of TSMC 3nm and next year 2nm production they could possibly want.

dindobre
0 replies
10h24m

This feels like the biggest part of the news to me

tiffanyh
5 replies
15h34m

I wonder who Apple will be colocating with for data centers.

And what the PCC chassis looks like for these compute devices (will it be a display-less iPad)?

jnaina
1 replies
15h24m

Starts with A and ends with S

tiffanyh
0 replies
4h5m

Apple DatacenterS

Gotcha, makes sense :)

jachee
1 replies
15h17m

Apple’s rich enough to build and own their own datacenters. Savvy enough, too. I’d imagine the chassis are custom Apple-NOC-specific M-chip powered servers.

tiffanyh
0 replies
4h7m

So basically, a Mac mini.

jrk
0 replies
15h16m

The have built and operated a growing number of their own data centers for years. Presumably this will go into those.

advael
5 replies
19h0m

I really want to see this OS, and have cautious optimism that this could be the first time we'll see a big tech company actually provide an auditable security guarantee!

I think depending on how this plays out, Apple might manage to earn some of the trust its users have in it, which would be pretty cool! But even cooler will be if we get full chain-of-custody audits, which I think will have to entail opening up some other bits of their stack

In particular, the cloud OS being open-source, if they make good on that commitment, will be incredibly valuable. My main concern right now is that if virtualization is employed in their actual deployment, there could be a backdoor that passes keys from secure enclaves in still-proprietary parts of the OSes running on user devices to a hypervisor we didn't audit that can access the containers. Surely people with more security expertise than me will have even better questions.

Maybe Apple will be responsive to feedback from researchers and this could lead to more of this toolchain being auditable. But even if we can't verify that their sanctioned use case is secure, the cloud OS could be a great step forward in secure inference and secure clouds, which people could independently host or build an independent derivative of

The worst case is still that they just don't actually do it, but it seems reasonably likely they'll follow through on at least that, and then the worst case becomes "Super informative open-source codebase for secure computing at scale just dropped" which is a great thing no matter how the other stuff goes

ignoramous
2 replies
18h42m

could be the first time we'll see a big tech company actually provide an auditable security guarantee

AWS Nitro Enclaves [0] come close but of course what Apple has done is productize private compute for its 1b+ macOS & iOS customers!

[0] https://docs.aws.amazon.com/enclaves/latest/user/nitro-encla...

transpute
1 replies
18h47m

> even if we can't verify that their sanctioned use case is secure, the cloud OS could be a great step forward in secure inference and secure clouds, which people could independently host or build an independent derivative of

Yes, the tech industry loves to copy Apple :)

Asahi Linux has a good overview of on-device boot chain security, https://github.com/AsahiLinux/docs/wiki/Apple-Platform-Secur...

> My main concern right now is that if virtualization is employed in their actual deployment, there could be a backdoor that passes keys from secure enclaves in still-proprietary parts of the OSes running on user devices to a hypervisor we didn't audit that can access the containers.

  We’ll release a PCC Virtual Research Environment: a set of tools and images that simulate a PCC node on a Mac with Apple silicon, and that can boot a version of PCC software minimally modified for successful virtualization.
This seems to imply that PCC nodes are bare-metal.

Could a PCC node be simulated on iPad Pro with M4 Apple Silicon?

advael
0 replies
18h35m

Yes, the tech industry loves to copy Apple :)

Yes, most technology is built on other technology ;)

This seems to imply that normal PCC nodes are bare-metal.

I realize that, but there's plausible deniability in it, especially since the modification could also hide the mechanism I've described in some other virtualization context that uses the unmodified image, without the statement being untrue

Shank
5 replies
19h5m

In a first for any Apple platform, PCC images will include the sepOS firmware and the iBoot bootloader in plaintext, making it easier than ever for researchers to study these critical components.

Yes!

Software will be published within 90 days of inclusion in the log, or after relevant software updates are available, whichever is sooner.

I think this theoretically leaves a 90-day maximum gap between publishing vulnerable software and potential-for-discovery. I sincerely hope that the actual availability of images is closer to instant than the maximum, though.

gigel82
4 replies
15h44m

Well, a 89-day "update-and-revert" schedule will take care of those pesky auditors asking too many questions about NSA's backdoor or CCP's backdoor and all that.

gpm
3 replies
15h5m

No, because the log of what source was used will still show the backdoored version, and you can't unpublish the information that it was used. Reverting doesn't solve the problem that people will be able to say "this software was attested 90 days ago and it hasn't been released".

If you're trying to do a quiet backdoor and you have the power to compel Apple to assist, the route to take is to simply misuse the keys that are supposed to only go into hardware for attestation, and instead simply use them to forge messages attesting to be running software on hardware that you aren't.

Or just find a bug in the software stack that gives you RCE and use it

brookst
2 replies
4h52m

simply use them to forge messages attesting to be running software on hardware that you aren't

Well, your messages have to be congruent with the expected messages from the real hardware, and your fake hardware has to register with the real load balancers to receive user requests.

RCE

That’s probably the best attack vector, and presumably why Apple is only making binary executables available. Not that that stops RCE.

But even then you can’t pick and choose the users whose data you compromise. It’s still a sev0 problem, but less exploitable for the goals of nation states so less likely to be heavily invested in for exploiting.

gpm
1 replies
3h30m

Well, your messages have to be congruent with the expected messages from the real hardware,

Yes, which is why you need the keys that are used to make real hardware. Provided you have those very secret and well protected keys (you are Apple being compelled by the government) that's not an issue.

and your fake hardware has to register with the real load balancers to receive user requests.

Absolutely, but we're apple in this scenario so that's "easy".

brookst
0 replies
36m

I think I misunderstood your point -- I took it to mean someone impersonating a server, but you're saying it's Apple. So the part you're attacking (as Apple) is:

The process involves multiple Apple teams that cross-check data from independent sources, and the process is further monitored by a third-party observer not affiliated with Apple. At the end, a certificate is issued for keys rooted in the Secure Enclave UID for each PCC node.

So, in your scenario, the in-house certificate issuer is compelled to provide certificates for unverified hardware, which will then be loaded with a parallel software stack that is malicious but reports the attestation ID of a verified stack.

So far, so good. Seems like a lot of people involved, but probably still just tens of people, so maybe possible.

Are you envisioning this being done on every server, so there are no real ones in use? Or a subset? Just for sampling, or also with a way to circumvent user diffusion so you can target specific users?

It's an interesting thought exercise but the complexity of getting anything of real value from this without leaks or errors that expose the program seems pretty small.

piccirello
4 replies
17h27m

The Secure Enclave randomizes the data volume’s encryption keys on every reboot and does not persist these random keys, ensuring that data written to the data volume cannot be retained across reboot. In other words, there is an enforceable guarantee that the data volume is cryptographically erased every time the PCC node’s Secure Enclave Processor reboots.
Timber-6539
3 replies
17h7m

Feels like an uptime screenshot would be appropriate here

transpute
2 replies
16h54m

PCC node execution should be per-transaction, i.e. relatively short lived.

wmf
1 replies
14h48m

The server can't afford to do one transaction then reboot.

transpute
0 replies
14h39m

Intel and AMD server processors can use DRTM late launch for fast attested restart, https://www.semanticscholar.org/paper/An-Execution-Infrastru.... If future Apple Silicon processors can support late launch, then PCC nodes can reduce intermingling of data from multiple customer transactions.

> The server can't afford

What reboot frequency is affordable for PCC nodes?

kfreds
4 replies
9h38m

Wow! This is incredibly exciting.

Apple's Private Cloud Compute seems to be conceptually equivalent with System Transparency - an open-source software project my colleagues and I started six years ago.

I'm very much looking forward to more technical details. Should anyone at Apple see this, please feel free to reach out to me at stromberg@mullvad.net. I'd be more than happy to discuss our design, your design, and/or give you feedback.

Relevant links:

- https://mullvad.net/en/blog/system-transparency-future

- http://system-transparency.org (somewhat outdated)

- http://sigsum.org

jiveturkey
0 replies
1h19m

It's not, AFAICT from the press release.

Confidential Compute involves technologies such as SGX and SEV, and for which I think Asylo is an abstraction for (not sure), where the operator (eg Azure) cannot _hardware intercept_ data. The description of what Apple is doing "just" uses their existing code signing and secure boot mechanisms to ensure that everything from the boot firmware (the computers that start before the actual computer starts) to the application, is what you intended it to be. Once it lands in the PCC node it is inspectable though.

Confidential Compute goes a step further to ensure that the operator cannot observe the data being operated on, thus also defeating shared workloads that exploit speculative barriers, and hardware bus intercept devices.

Confidential Compute also allows attestation of the software being run, something Apple is not providing here. EDIT: looks like they do have attestation, however it's different to how SEV etc attestation works. The client still has to trust that the private key isn't leaked, so this is dependent on other infrastructure working correctly. It also depends on the client getting a correct public key. There's no description of how the client attests that.

Interesting that they go through all this effort just for (let's be honest) AI marketing. All your data in the past (location, photos, contacts, safari history) is just as sensitive and deserving of such protection. But apparently PCC will apply only to AI inference workloads. Siri was already and continues to be a kind of cloud AI.

rekoil
0 replies
9h21m

This was my take from the presentation as well, immediately thought of your feature. Will be interesting to hear your take on it once the details have been made available and fully understood.

ThePhysicist
0 replies
7h46m

Yeah it seems so, though most of these systems (e.g. Intel SGX, AMD SEV, NVIDIAs new tech) use the same basic building blocks (Apple itself isn't member of the confidential computing consortium but ARM is), for me it's the quality of the overall implementation and system that sets this apart. I'm also quite bullish about trusted computing, seems it gains significant momentum. I would like some technologies to be more open and e.g. allow you to control the whole stack and install your own root certificates / keys on a hardware platform, but even so I think it can provide many benefits. With Apple pushing this further into the mainstream I expect to see more adoption.

throwaway369
3 replies
17h16m

[Deleted]

zie
2 replies
17h3m

Well the options from China's perspective is: Come to the table and meet some/all of our demands or stop doing business here.

Since Apple devices are now on the Chinese Governments poopy list, I assume Apple is only meeting some, not all of China's demands. I assume if Apple did everything the Chinese govt wanted, they wouldn't be on the poopy list. Personally I see being on the Chinese govt poopy list as an endorsement that it's probably a net positive for privacy and security compared to those not on the list. :)

Around WhatsApp, it's probably part of the whole compromise mess above. WhatsApp now does E2E and that's something China is not a fan of, so it's probably China's doing that it's not in the app store in China any more. Apple is just following the laws China forces them to follow.

It should be noted I've never been to China(yet) and have zero 1st hand knowledge.

astrange
1 replies
16h2m

It's a silly oversimplification that nothing in China is ever allowed to have privacy ever. China has privacy/data protection laws just like other countries do. Even an authoritarian government doesn't want other random private actors getting to see everything.

zie
0 replies
14h8m

I agree, but I was talking specifically about the govt.

The govt basically requires total access doesn't it? I mean every govt basically wants it, and the US has tried many times, but so far hasn't quite gotten complete access everywhere.

clipjokingly
3 replies
16h20m

Is it possible to have zero knowledge AI?

wslh
2 replies
15h37m

Yes, the issue is that they are really slow.

rjeli
1 replies
15h9m

ZKML is actually not horrible, probably only 100-1000x overhead atm. Unfortunately it doesn’t solve the problem, you would need FHE which has much higher overhead

tharant
0 replies
14h20m

FHE? I, a noob, assume that acronym maybe has something to do with homomorphic encryption?

Also, got any links for interesting ZKML papers/projects?

cherioo
3 replies
18h34m

Can some ELI5 how remote attestation is supposed to work? It feels like asking a remote endpoint “are you who you say you are”. What’s stopping remote endpoint always responding “yes”

wyes
0 replies
16h54m

They rely on Trusted Execution Environments and the fact that hash functions are one-way functions.

Verifier -> requests a Prover to attest its software state

Prover -> goes into RoT, verifies authenticity of Verifier (and request), computes hash of attested memory region, sends hash digest

Verifier -> receives digest and compares to known hash

What’s stopping remote endpoint always responding “yes” The attestation code is inside of a RoT, so a bad actor shouldn't be able to call this code, only callable by receiving a request from a Verifier
transpute
0 replies
18h8m

> What’s stopping remote endpoint always responding “yes”

It requires a small, trusted remote observer hardware component, e.g. TCG TPM/DICE, Apple Secure Enclave, Google OpenTitan, Microsoft Pluton.

2021 literature review, https://arxiv.org/abs/2105.02466

2022 HN thread on remote attestation, https://news.ycombinator.com/item?id=32282305

GrantMoyer
0 replies
15h18m

My understanding is that it's similar to TLS authentication.

The remote endpoint has special hardware which keeps secret signing keys (similar to a TLS server's signing keys). The hardware refuses to reveal the private keys, but will sign certain payloads under certain conditions. In addition, Intel or AMD or whoever also has super duper mega secret master keys (similar to a CA's signing keys), which they use to sign the device's signing keys. The certificate signing the device keys is also stored on the device.

So, each time the endpoint is asked to attest its software, it says yes and signs its response with its keys, and it also sends a certificate showing its keys are signed by the master key. That way, the client knows the special hardware really said yes and that Intel or AMD or whoever said that particular special hardware is legit.

CGamesPlay
3 replies
14h57m

All of this is interesting, but how easy is this to circumvent? When Apple changes their mind for whatever reason, don't they just return a key to a fake PCC node, which would bypass all of their listed protections? Furthermore, what prevents Apple from doing this for specific users?

a2128
1 replies
14h23m

According to the article, it would be difficult to tie any request to a user:

Target diffusion starts with the request metadata, which leaves out any personally identifiable information about the source device or user, and includes only limited contextual data about the request that’s required to enable routing to the appropriate model

If this is the case, I wonder how the authentication would work. Is it a security through obscurity sort of situation? Wouldn't it be possible for someone, through extensive reverse engineering, to write a client in Python that gives you a nice free chat API and Apple would be none the wiser?

filleokus
0 replies
7h7m

Don't know if they use it (or if it would somehow weaken/break the privacy claims you cited), but Apple has an SDK called DeviceCheck[0].

Essentially, your server send a nonce which the client signs using a key pair derived from the Secure Enclave. The server can then verify the signature by an API provided by Apple's servers, and they respond whether it was signed by a Secure Enclave resident key or not.

I'm guessing this could be helpful to make it hard(er) to write a Python client.

[0]: https://developer.apple.com/documentation/devicecheck/establ...

JimDabell
0 replies
2h56m

iOS won’t send requests to it unless that node appears in the transparency log.

If it appears in the transparency log, the whole world will be able to see that a suspicious node has started serving requests.

If Apple changes iOS to remove that restriction, the whole world will be able to see that change because it’s client side.

If Apple tries to deliver a custom version of iOS to a single user, the iOS hardware will refuse to run it unless it has a valid signature.

If it has a valid signature, that copy of the firmware is irrefutable evidence that Apple is deliberately breaking its privacy promises and spying on people in a way they specifically said they wouldn’t, which would be extremely harmful to their business.

Apple seems to be going all-out in binding themselves in a way that makes it as difficult as possible to do what you are suggesting.

thomasahle
2 replies
19h48m

Did Apple say anything about what training data they used for their generative image models?

jml78
0 replies
19h13m

Yes, basically if you opted out of Apple scraping, your data isn’t used

goupil
2 replies
13h46m

It's sad to see so many discussions on security and so little on privacy. How about solutions that could combine both, such as homomorphic encryption for AI?

wmf
0 replies
12h58m

Privacy is the entire point of this discussion?

Homomorphic encryption is mostly a fantasy at this point.

dymk
0 replies
4h28m

There’s not a single homomorphic encryption implementation out there which can do real world quantities of computation

gigel82
2 replies
15h36m

The only way to trust this is them selling "cloud compute" servers that folks can deploy and monitor in their own infrastructure. Nothing else can be guaranteed to not include malicious code to exfiltrate the data.

Or better yet, make the APIs public and pluggable so that one can choose an off-device AI processor themselves if one is needed.

astrange
1 replies
10h48m

Your own infrastructure is definitely less secure than this or even, say, Google. You do not have the capability and teams of SREs to detect intrusions, and an attacker would know that your server processes your data.

gigel82
0 replies
23m

Maybe, but I can totally firewall my own servers to my heart's desire, including completely blocking it off from the internet and only allowing connections via my own network's routes.

ethbr1
2 replies
19h58m

This entire platform is the first time I've strategically considered realigning the majority of my use to Apple.

Airtag anonymity was pretty cool, technically speaking, but a peripheral use case for me.

To me, PCC is a well-reasoned, surprisingly customer-centric response to the fact that due to (processing, storage, battery) limitations not all useful models can be run on-device.

And they tried to build a privacy architecture before widely deploying it, instead of post-hoc bolting it on.

> 4. Non-targetability. An attacker should not be able to attempt to compromise personal data that belongs to specific, targeted Private Cloud Compute users without attempting a broad compromise of the entire PCC system. This must hold true even for exceptionally sophisticated attackers who can attempt physical attacks on PCC nodes in the supply chain or attempt to obtain malicious access to PCC data centers.

Oof. That's a pretty damn specific (literally) attacker, and it's impressive that made it into their threat model.

And neat use of onion-style encryption to expose the bare minimum necessary for routing, before the request reaches its target node. Also [0]

> For example, the [PCC node OS] doesn’t even include a general-purpose logging mechanism. Instead, only pre-specified, structured, and audited logs and metrics can leave the node, and multiple independent layers of review help prevent user data from accidentally being exposed through these mechanisms.

My condolences to Apple SREs, between this and the other privacy guarantees.

> Our commitment to verifiable transparency includes: (1) Publishing the measurements of all code running on PCC in an append-only and cryptographically tamper-proof transparency log. (2) Making the log and associated binary software images publicly available for inspection and validation by privacy and security experts. (3) Publishing and maintaining an official set of tools for researchers analyzing PCC node software. (4) Rewarding important research findings through the Apple Security Bounty program.

So binary-only for majority, except the following:

> While we’re publishing the binary images of every production PCC build, to further aid research we will periodically also publish a subset of the security-critical PCC source code.

> In a first for any Apple platform, PCC images will include the sepOS firmware and the iBoot bootloader in plaintext, making it easier than ever for researchers to study these critical components.

[0] Oblivious HTTP, https://www.rfc-editor.org/rfc/rfc9458

transpute
0 replies
19h7m

It's very encouraging.

Another good step in this direction would be publishing a list of all on-device Apple software (including Spotlight models for image analysis) and details of any information that is sent to Apple, along with opt-out instructions via device Settings or Apple Configurator MDM profiles.

Apple does publish a list of network ports and servers, so that network traffic can be permitted for specific services. The list is complicated by 3rd-party CDNs, but can be made to work with dnsmasq and ipset, "Use Apple products on enterprise networks", https://support.apple.com/en-us/101555

manquer
0 replies
18h35m

Oof. That's a pretty damn specific (literally) attacker, and it's impressive that made it into their threat model.

How so ? There are any number of state and state sponsored attackers who it should apply it including china, North Korea , Russia , Israel as nation states and their various affiliates like NSO group .

Even if NSA its related entities are going to be notably absent. If your threat model includes unfriendly nation state actors then the security depends on security at NSA and less on Apple, they have all your data anyway.

If nation state actors are interested in you, no smartphone that is not fully open source on both hardware and OS side that has been independently verified by multiple reviewers is worth it, i.e. no phone in the market today, everything else is tradeoff for convenience for risk, the degree of each is quite subjective to each individual.

For the rest of us, the threat model is advertisers, identity thieves, scammers and spammers and now AI companies using it for training.

Apple will protect against other advertisers insofar to grow their own ad platform , they already sell searches to Google for $20B/year and there is no knowing the details of the OpenAI deal on what kind of data will be shared.

croes
2 replies
12h36m

Who pays for the costs of private cloud compute, is it free of charge for the iPhone owner (at least until they turn it into a subscription)?

What about second hand iPhone users?

repler
0 replies
8h16m

Exactly - nothing is for free. They explicitly state that PCC data gets destroyed after a response is returned.

Are the anonymized queries (minus user data context) worth anything?

It’s gotta be some kind of subscription/per query charge model to pay for the servers, electricity, and bandwidth.

AnonHP
0 replies
1h48m

Who pays for the costs of private cloud compute, is it free of charge for the iPhone owner (at least until they turn it into a subscription)?

My guess is that this is similar to how iOS upgrades are “free”, how Apple Maps is free, how iMessage is free, how iCloud Mail is free, etc. To a good extent, it’s all paid for by the price paid by the customer for the hardware.

I’d also wager that there will be a paid service/subscription that will get baked into iCloud+ at some point in time (maybe a year from now). This will offer a lot more and Apple will try to attract more customers into its paid services net.

WatchDog
2 replies
13h46m

I'm interested in how this compares to AWS nitro enclaves, which they mention briefly.

The main difference seems to be verifiability down to the firmware level.

Nitro enclaves does not provide measurements of the firmware[0], or hypervisor, furthermore they state that the hypervisor code can be updated transparently at any time[1].

Apple is going to provide images of the secure enclave processor operating system(sepOS), as well as the bootloader.

It also sounds like they will provide the source code for these components too, although the blog post isn't clear on that.

[0]: https://docs.aws.amazon.com/enclaves/latest/user/set-up-atte....

[1]: https://docs.aws.amazon.com/pdfs/whitepapers/latest/security...

yolovoe
0 replies
6h15m

Nitro does measure firmware. If any firmware is unexpected, server will essentially stop being connected to the EC2 substrate network and/or server wiped clean automatically. People will be paged automatically, security will likely be pulled in, etc.

There is no reason to measure hypervisor firmware as it’s not firmware in the case of EC2. The BIOS/UEFI firmware on the mobo is overwritten if it’s tampered with. Hypervisor code (always signed, like all code) is streamed via a verifiably secure system on the server (Nitro cards, which make use of measured boot and/or secure boot).

No idea what the customer facing term “Nitro enclaves” means, but EC2 engineers are literally mobilized like an army with pages when any security risk (even minor ones) is determined. Basic stuff like this is covered. We even go as far as guaranteeing core dumps don’t contain any real customer data, even encrypted

ram_rattle
0 replies
11h36m

Aws had to do it this way because of their custom silicon, Intel, ARM and AMD do provide firmware/hypervisor level attestation

solarkraft
1 replies
16h52m

I was sceptical of the announcement, but this actually sounds really well thought out.

One key part though will be the remote attestation that the servers are actually running what they say they're running. Without any access to the servers, how do we do that? Am I correctly expecting that that part remains a "trust me bro" situation?

wyes
0 replies
16h39m

Attestation will run on the RoT.

While we’re publishing the binary images of every production PCC build, to further aid research we will periodically also publish a subset of the security-critical PCC source code.

I expect that they'll publish the attestation source code.

But, basically what will happen is the Verifier will request a certain memory region to be attested, then that region will be hashed and the digest will be sent back to the Verifier. If the memory is different from what is expected, the hash digest will NOT match.

rmbyrro
1 replies
18h17m

Most ironic thing is they abbreviate this as "PCC". (reads chinese communist party in many languages)

The absolute worst acronym for anything even remotely related to personal privacy.

transpute
0 replies
17h55m

Inverted acronym?

ramesh31
1 replies
16h17m

Here's the answer to the "what's taking Apple so long to get on the LLM train?" folks. Per usual, they lag a bit and then do it better than anyone else.

JimDabell
0 replies
2h54m

It’s also because they have a twelve month release cycle.

paul2paul
1 replies
11h43m

We don't need "a new frontier". I want to be the only one who holds the private key to my encrypted data. I think it's pretty lame to sell privacy when it's not.

dyauspitr
0 replies
2h31m

The problem with that is it’s not possible for you to be the only one to hold the private key and have the cloud run your data against a model.

m3kw9
1 replies
17h11m

I wonder how they will do this in china?

mlindner
0 replies
16h25m

Apple already runs China-only software on their devices, I suppose it just won't run there.

candiddevmike
1 replies
19h19m

Did Apple need to license the phrase Core OS like iOS?

thirdhaf
0 replies
19h15m

That phrase distinguishes the internal group responsible for that part of the architecture, don’t think it’s a marketing term.

Havoc
1 replies
19h0m

Sounds good. Still won’t send anything sensitive there but I appreciate the effort and direction, especially when current industry trend seems to be fuck you were rewriting our TOS to take your data.

m-s-y
0 replies
18h6m

Apple’s doing this specifically to avoid the possibility of what you’re describing.

The transparency & architecture together are intended to be more than enough to publicly detect any major retooling of the system.

yla92
0 replies
17h52m

And finally, we used Swift on Server to build a new Machine Learning stack specifically for hosting our cloud-based foundation model.

Interesting to see Swift on Server here!

https://www.swift.org/documentation/server/

vlovich123
0 replies
15h3m

What I haven’t heard from the announcement is whether the private cloud has external network access. Presumably it wouldn’t otherwise the guarantees of your request staying in your cloud is meaningless. Conversely, a lot of trivial network stuff can be involved (eg downloading the model). Anyone know which balance Apple is choosing to strike initially?

v4dok
0 replies
7h59m

This is Confidential Computing https://en.m.wikipedia.org/wiki/Confidential_computing

with another name. Intel, AMD and Nvidia have been working for years on this. OpenAI released a blog some time ago where they mentioned this as the "next step". Exciting that Apple went ahead and deployed first, it will motivate the rest as well.

tzs
0 replies
6h49m

The Secure Enclave randomizes the data volume’s encryption keys on every reboot and does not persist these random keys, ensuring that data written to the data volume cannot be retained across reboot. In other words, there is an enforceable guarantee that the data volume is cryptographically erased every time the PCC node’s Secure Enclave Processor reboots.

I wonder if there is anything that enforces an upper limit on the time between reboots?

Since they are building their own chips it would be interesting to include a watchdog timer that runs off an internal oscillator, cannot be disabled by software, and forces a reboot when it expires.

system7rocks
0 replies
12h36m

I trust Apple

rldjbpin
0 replies
32m

not trusting any of the privacy/security mumbo-jumbo when their icloud free tier still allows for a paltry 5 gigs, when even google offers thrice as much for their public service.

i am happy for those who see the positives here, but for the skeptic a toggle to prevent any online processing would be more satisfactory.

nisten
0 replies
8h39m

Complete horseshit marketing speak.

Was the cloud non-private before? Was it not secure in the first place? Do my Siri searches no longer end up as google ads metadata now? Are the feds no longer able to get rubber stamp access to my i C L O U D now?

You are a naive idiot for believing that this is anything but security theater to adress the emotional needs of AI anxiety in and outside the company.

Just my opinion.

nerdright
0 replies
15h12m

Even if you don't like Apple's monopolistic approaches, you have to admire how they go an extra mile to stay true to their mantra of selling privacy.

This is clearly a company with an identity, unlike Microsoft and Google who are very confused.

nardi
0 replies
11h15m

Many people in this thread are extremely cynical and also ignorant of the actual security guarantees. If you don’t think Apple is doing what they say they’re doing, you can go audit the code and prove it doesn’t work. Apple is open sourcing all of it to prove it’s secure and private. If you don’t believe them, the code is right there.

krosaen
0 replies
5h34m

I wonder if they will ever make this available to developers - I can think of many products that would be nice to have at least part of the cloud infra being hosted in a trusted provider like this, e.g indoor cameras for health metrics: sounds awesome but I would never trust a startup to handle private data this sensitive.

jaydeegee
0 replies
17h52m

Outside of all the security aspects which look to be handled quite well on the surface I do enjoy that the client mainframe architecture is still a staple of computing.

j0e1
0 replies
17h30m

The Apple Security Bounty will reward research findings in the entire Private Cloud Compute software stack — with especially significant payouts for any issues that undermine our privacy claims.

Let the games begin!

dymk
0 replies
4h42m

I would love to be able to run a PCC node locally on my M2 MacBook or similar for my iPhone to offload to, even if it’s only for doing what 15 Pro iPhones can do on-device.

There’s precedent for this sort of thing as well, like Apple TVs or iPads acting as HomeKit hubs and processing security can footage on-device.

Maybe they’ll open that up in the future.

asp_hornet
0 replies
11h6m

This thread reads like a whole bunch of sour grapes. Hopefully this challenges other companies to do better

EternalFury
0 replies
13h42m

Let’s not be too picky. This is a good thing.