Cleaning up my 200GB iCloud with some JavaScript

Okay, so it could be a bug, but it is possible that iCloud is secretly storing more than one version of the file in some cases? (After all, Apple does similar things with other media files.)

The example given at the end is interesting:

So iCloud says the video is 128MB, I download it and the video is actually 48MB, and my free storage increases by ~170MB when I deleted it. Interesting!

This suggests that iCloud isn't simply misrepresenting the size of the example file, as then you'd expect that deleting the 128MB file would clear ~128MB of iCloud space. Instead, the deletion clears roughly the space it reports (128MB) plus the space of the downloaded version (48MB): 128MB + 48MB = 176 MB - which might be close enough, allowing for rounding errors, as iCloud reports the free space (from the article's example) to the nearest 10 MB.

Differential backups or any sort of versioning seemed like one of the most obvious culprits (that and or total redundant storage to preserve the file) but the issue with all of this is it’s entirely opaque.

Ultimately you’re increasingly tethered to some service for your storage that you pay for periodically based on total storage yet you have little-to-no information how to best optimize that storage if you want to operate in a fixed cost bracket or lower storage/cost ratio. So as a consumer, do I just wave my hands and keep throwing more and more money at the problem, especially now that devices are increasingly pushing everything, including storage, as a subscription service to meet my actual functional needs (that realistically could be met by local storage options if manufacturers didn’t have a vested interest in pushing me towards service based storage solutions)?

The modern business strategy in technology is simply hiding behind complexity. The cost is too complex for you to understand, it gives too much information away about our internals to competitors, and so on. Yet somehow these metrics are derived to assure the business is operating above cost because when the rubber meets the road it must be done, yet when the consumer wants to understand it’s suddenly too complex. The problem is that tech in many cases is growing to scales that really is too complex and business managers know this, so it’s often a valid excuse to hide behind. Conveniently that’s where they focus on investment and padding margins though.

So as a consumer, do I just wave my hands and keep throwing more and more money at the problem,

Yes.

I can go and buy 1TB of Microsoft OneDrive or 2TB of Google Drive for less than a Franklin a year, and most people won't even need 1TB let alone 2TB. Both Microsoft and Google also offer 100GB plans for a Jackson a year, which is what I purchase myself. The average person can get by paying a Washington per month to Apple for 50GB of iCloud.

The amount of money I would save from managing photos myself locally isn't worth the time spent nor the money spent on the hardware.

EDIT:

For the downvoters, consider this: If I were to manage all this myself, I would need at least three storage mediums with one being a different form factor to satisfy the 3-2-1 backup scheme. I would also need to procure arrangements for that third backup copy in the 3-2-1 scheme. And I would need to spend time managing it all.

That is going to cost me more than a Franklin per year. Life is short, my time is precious, and my money is ultimately expendable.

For the downvoters, consider this: If I were to manage all this myself, I would need at least three storage mediums with one being a different form factor to satisfy the 3-2-1 backup scheme. I would also need to procure arrangements for that third backup copy in the 3-2-1 scheme. And I would need to spend time managing it all.

That is going to cost me more than a Franklin per year. Life is short, my time is precious, and my money is ultimately expendable.

When you're looking at cloud services, you need to perform your own off-site backup. Apple, Google, Microsoft, etc. will maintain copies that they'll restore in the event of a hardware failure. But, if your account gets compromised or a buggy sync or bad API event happens, your data is gone. They're not going to go restore it from tape for you. This is a big part of why I do have an in-home NAS. Maybe you have everything sync'd with a laptop and that has you covered, but Apple's expanded storage options are outlandishly expensive so I doubt many with the 2TB+ plans are able to do that. (Yes, you could use external storage, but that's also rather inconvenient for a Photos.app library.)

We could both get what we want if these storage operations weren't wrapped up in proprietary APIs. If I use iCloud I get a seamless experience on macOS, but no access at all on Linux. If I use Dropbox I get access on Linux, but little more than photo sync on an iPhone. Given the decades of precedent with filesystems and I/O APIs, I suspect we could have an abstraction layer and an implementation layer that would allow for interoperability. Anyone that wants to pay for iCloud are free to do so, others could use their preferred storage engine. But, allowing access into the walled garden is far less profitable.

For most people, storage needs are going to increase over time (more + higher resolution photos & videos, larger apps, document storage, etc.). 6TB for a family is not unreasonable and that's what? Three Franklins and three Jacksons per year + whatever for an external drive for your offsite backups. What comes after the 6TB option? Storage costs have decreased drastically over time, unless you're using a proprietary service; consumers are not benefiting at all from those gains in efficiency.

Three Franklins and three Jacksons per year + whatever for an external drive for your offsite backups.

I know you're just following the theme set by the parent commenter, but there are a bunch of us folk on HN who aren't US residents, and have no idea how much those presidents mean in terms of currency.

In addition to being confusing for non-Americans, it also confusing for Americans because $100 bills in American slang are "Benjamins," not "Franklins." It's most notable use is in the song "It's All About the Benjamins"

I've heard it both ways myself; I like Franklin better since the others are all last/family names too.

I'm sorry. I use the currency and had to think about what the values were. I was trying to follow the theme by the previous author, but I can definitely see how that'd be hard for others to follow. It's $360 USD (Franklin = $100 USD, Jackson = $20 USD).

A Franklin is more or less two Turings and a Jackson is just less than a Turner.

What comes after the 6TB option?

Well, 12TB.

And if you are head of household and share storage, you can combine storage plans. Mine currently shows “2.3TB of 14TB used”.

Thanks. I overlooked the 12 TB. I'm not sure doubling the capacity and doubling the cost is really ideal for many, but it's nice to know it's there.

6TB for a family is not unreasonable and that's what?

Microsoft in particular has a 6TB for $100/year family plan, sharable with up to 5 other family members for a total of 6 persons each with 1TB. Google's plans can all also be shared with up to 5 other family members, though their bytes-per-dollar can't compete with that particular Microsoft family plan.

Basically: Local storage with personal management needs to be very easy, cheap, and carefree (which it isn't) to compete practically with cloud storage.

The only exception is if one's needs are niche and specific. I actually have a Synology NAS at home that I keep most of my data on, but that's because my data is mostly "bottle of rum" and "Linux ISO" in nature and thus not something I can throw on cloud storage in the first place.

As someone who has a Proxmox cluster at home (storage on RAIDZ, hot backups with PBS, cold backups on external HDDs) I literally recommend cloud storage for most people that ask me about backup solutions as it's simply not worth it for the average user. Those people already have all their data in the cloud anyways and share it on Facebook et al and they don't really care about the privacy side.

Remember it's not just buying the equipment, it's maintaining and understand it as well (e.g. I have to be familar with how ZFS works, how to restore a failed node, write some scripts, etc.). And with every backup solution you also need to be familar with the restoration process and test it occassionally to make sure it actually works as expected.

Another danger of doing it yourself I have found is that if you give a dev a Proxmox, they are going to play around making toy Kubernetes setups instead of implementing the backup system they intended to make.

Yeah I have always backed up to external drives and had cloud storage. But now I learn there isn’t rot, so I have to either build a RAID and have refresh software that rewrites and validates, or just buys a new drive every 3 years and backup all over again. And that’s a simple setup.

"640KB is more than anyone will ever need"

It's a little absurd to think people don't need more than 2TB - especially on HN. Gamers will likely have 2TB in games alone, videographers often have many TBs of videos and photos from weddings and events in their life, many that care about health may have a few TB in genomic data mirrored on their computers to analyze, etc.

I would imagine it's hard to find people that wouldn't have TBs of data, if they were allowed to do so. The reason many people don't have TBs of data is they're limited by these exact companies you're claiming 'solve the problem' by offering limited storage.

It is notable however, that having better tools to organize, deduplicate, and compress data would be helpful to reduce some of the size of data that many people have. Over the years I've noticed my family will have multiple tar.gz archives, zip archives, etc, which (after extraction/unencryption) will share 20% files here, 10% files there, a 4kb jpg that's the same as a 100MB PNG here and there, etc. So yes, those 10TB archives may end up being 5TB if someone spent the time to really comb over, understand, make good decisions, and organize that data. But I have not yet seen anything that can scratch that surface yet, other than perhaps https://github.com/jjuliano/aifiles - but I won't use it until it's local only and has guarantees not to destroy data without explicit permission. An overlay filesystem that shows compression/deduplication with LLM capability like aifiles is probably the best option here.

However, I wouldn't imagine that most people's life data is less than 2TB even with all of this - it's mostly imposed as an artificial constraint by these companies.

Gamers will likely have 2TB in games alone

If I were to install all the games on my steam account it would be many hundreds of terabytes of total storage. In the end I have about a terabyte of games on my computer. And of that 0 bytes are in my cloud storage.

I've been an amateur photographer for over 15 years. I tend to curate the photos I keep, largely because I don't need 20+ pictures of the same scene. Its more of a burden to casually flip through my photos if the majority of them are near duplicates. In the end my total collection is only several hundred gigs.

Most people aren't videographers.

Most people in my family have far less than even 50 gigs of actual data they care about. They maybe take a dozen compressed photos a week, maybe 30 minutes of videos a month. A lot of my friends take even fewer photos and pictures.

You could argue all your games are in the cloud since of the hundreds of terabytes of games you have, you're only keeping 1TB of them on the computer.

Now, consider a world where you are making $42,000 a year. That's 20 dollars an hour and a very common wage. How do they handle the same when there are so many competing Franklins for them?

bump this, and the implicit vendor lockin that this ideology creates

You still need to manage backups. You’re trusting everything to the vendor. I run a docker image weekly that pulls my google photos, copies to my USB drive in my pi, and also copies to backblaze b2.

got a link for this image?

And if you invest that $100 at 7% that becomes after x years...

,End Balance after x years, 4% per year

10, 1,578.36, 63.13

15, 2,788.81, 111.55

30, 10,207.30, 408.29

40, 21,460.96, 858.44

Partly you’re getting downvoted because you’re not accounting for the value of the content to the user. Even if the chance of a cloud provider cancelling an account or deleting content is 0.0001%, the value of a single photo to Microsoft will be magnitudes less than to an individual.

I imagine the other reason is because they’re not mutually exclusive: For instance, Synology makes it easy to have both an in-home NAS and cloud sync.

This is the same argument behind paying for a streaming service. Could I "find" everything I want to watch somewhere else and maintain it myself on a Plex server? Sure, but the cost-benefit analysis just doesn't make sense to me.

Particularly the older I get the more I value my finite free time. Throwing $20 at something to remove a problem that would take me hours (not to mention a large startup cost) to do myself is just an obvious choice.

Hackers don't understand something that regular people more readily do: It works, it's cheap, and I'm paying for the convenience. We can all have our ideals about how tech should be, but these choices are driven by practicality.

Presidents on US currency:

  - $100,000: Wilson
  - $1,000: Cleveland
  - $500: McKinley
  - $100: Franklin*
  - $50: Grant
  - $20: Jackson
  - $10: Hamilton*
  - $5: Lincoln
  - $2: Jefferson
  - $1: Washington
  
  * not a president

> I can go and buy 1TB of Microsoft OneDrive or 2TB of Google Drive for less than $100 a year, and most people won't even need 1TB let alone 2TB. Both Microsoft and Google also offer 100GB plans for a $20 a year, which is what I purchase myself. The average person can get by paying $1 per month to Apple for 50GB of iCloud.

While we're at it, iCloud+ offers these monthly storage plans now:

  United States:
  50GB: $1
  200GB: $3
  2TB: $10
  6TB: $30
  12TB: $60

See everywhere in the world here:

https://support.apple.com/en-us/HT201238

That's probably indeed the case, but it doesn't excuse it.

When you buy a hard drive or USB stick, you get a certain amount of GBs to use as you please. If you put a 1GB file on it your free space decreases by 1GB (yes this is filesystem dependent and you might lose a few KBs for metadata, but the choice of filesystem is up to you and not mandated by the storage decide). It doesn't matter that the NAND controller probably used a few megabytes of the overprovisioned area to store its block mapping tables, or maybe even duplicated your data for its convenience - you were never charged for that overprovisioned area.

Here, you are sold a storage device (that you access over HTTP instead of SATA/PCIe), but when you write a 1GB file, they duplicate/convert/etc it for their convenience yet still charge you to store those duplicates you haven't asked for. That's new and unexpected.

Don't get the complaints, you can use different messaging apps, that don't do that to your oCloud

That's new and unexpected.

For you (and me) perhaps, but not for most people[1] who appreciate having the file system abstracted away.

[1]: https://news.slashdot.org/story/21/09/27/2032200/students-do...

I think the confusion here is also rooted at the fact that photos and videos in the Photos library are not simply files copied to storage. Photos also stores metadata about edits, renditions and thumbnails and other data needed for various functions of the Photos app.

So when you’re syncing photos to iCloud, it’s not just the individual files that get synced but it’s the “Photos Library” managed container of the Photos app.

If you add individual files directly in Finder or the Files app then their size matches exactly both in iCloud and on the local file system.

On iPhone, the edits to pictures and videos are just metadata and the original file is kept. I just tested recording a video, cutting it in half and downloading the file from iCloud. The cut file it's smaller than the iCloud reported size however if I choose to download the unmodified original the size matches the iCloud reported size.

I kind of hate opaque cloud solutions like this, because their storage usage is inscrutable and their interfaces are basically never optimized to help you clear things out. Whether that’s intentional and malicious or just not a priority, I don’t really care, I just want to be in control of the stuff I’m using.

I have a very similar issue with iMessage in iCloud, and unlike photos, there’s no web interface. You have to interact with it on your device. Thankfully I have a Mac so I can load up the chat database from there and see what’s using the space, but cleaning it up has been a nightmare: I have a script to find the big attachments, but if I delete them then some agent that manages the database goes into a loop for like five minutes per deletion. Then the change gets synced to iCloud (at least, I assume). So unless I fully reverse what the deletion process does and whether it is possible to do a batch operation I’m basically at the mercy of the front end they provide to clean things up, which as I mentioned earlier is absolutely not designed to make it easy to do this.

The opaqueness is definitely a problem. My iMessage database has a corrupted spot somewhere in the message history; if I accidentally make the Messages app read that spot (via searching or scrolling back through old conversations) it'll silently start maxing out the CPU and burning energy for no reason. There is no error message, nothing obviously wrong-looking, and as far as I know everything works.

Hey, this sounds like an interesting problem. I am always looking for edge cases to test, if you have time would you mind checking if https://github.com/ReagentX/imessage-exporter works for you and if it crashes in that spot?

Holy crap. I've been looking for ways to archive my imessages somehow. imessage-exporter looks like exactly what I wanted. What a cool project!

Cheers! It was fun to build.

Oh, so that's what's happening? You made me realise I have the same exact issue!

Well I can't for sure prove it's indeed DB corruption, but I don't have many other theories.

It's unlikely to be a "static" bug affecting everyone because it would've been caught in QA and/or user reports and fixed already (this is not a new issue, been happening for a year at least).

Therefore my hypothesis is that it's dependent on corrupt persistent data that a relative minority of users have which is not easy for Apple to detect/replicate.

Sounds like a nightmare. Do you know what could have caused the database corruption?

Couple years of macOS/iOS reinstalls along with the typical shittiness of iCloud and the iMessage clients. I would be more surprised if it wasn't corrupt.

Search on the messages app is broken for me on iOS and macOS. It really can’t find things that are there. So I have to scroll and look around. The sizes reported are also inaccurate. While iCloud will report some large number, when I dig into the largest messages section it doesn’t show anything close to that size (with just a few items in that list).

Since the macOS Messages app is some Catalyst abomination, it won’t even support expected keyboard shortcuts for navigation, selection and deletion. Having to do everything using the trackpad is a slow process.

It’s frustrating that messages is so bad.

Why are message apps like this? Tons of effort for almost useless stuff like stickers but super basic media management.

Even in open source apps where in theory we can contribute functionality like Signal it’s difficult. E.g. it’s impractical to delete lots of media in groups in the iOS app. I added a PR last year to help this but still waiting for a merge even though it has been approved :(

Edit: PR for the curious

https://github.com/signalapp/Signal-iOS/pull/5685

Though Telegram’s chats are not end-to-end encrypted by default (and E2EE is not available for group chats), its storage management on device and on the cloud are quite good.

I have the same problem on iMessage. iCloud reports 7+gb of message storage being consumed, but each of my devices and my top conversations only report 1-2gb of content. I haven't found a way to force sync the missing content so I'm stuck paying the $1/month plan.

I haven't found a way to force sync the missing content so I'm stuck paying the $1/month plan.

And this is why they (and many companies) behave this way, there's no incentive to fix a problem that you will just pay them more money to fix. Fixing the bug means reduced revenue. Complaining about the problem sadly won't help, you can only stop using the app and stop buying their products.

no web interface

Same with HomeKit Secure Video (HKSV). The only way to access recordings is through the Home app. I'm surprised nobody has figured out a way to browse them outside of the app.

The opaqueness is malicious. Apple offers a free iCloud account but no method of cleaning up space. Normal usage of a phone with all optional features disabled will eventually fill up the free tier space.

The user’s options are quit the ecosystem or pay up. Apple’s lock-in monopoly means they get to shove their customers into another monthly fee.

This is a classic example of the harm done to consumers by allowing anticompetitive practices. (The consumer has no choice but to pay)

When the US government gets off their butts and investigates Apple for antitrust, this will be one of the key findings. The fine should be some multiple of the amount of money they have illegally extracted from their customers.

Not iCloud but Google makes it easy to drill down into how your storage quota is being used - https://one.google.com/storage

Photos.app doesn't show file size… so I thought, why not build a little Photos.app extension or a separate app that queries for large files?

Turns out that the API doesn't expose "file size", at least I didn't find a straight-forward way.

I think that all "photos" or "videos" are just a view of the underlying "photo or video object". If you crop a video, the full-size video will remain. Only if you export the video, it will be cropped and the smaller file size will manifest.

I guess that's why the file sizes differ.

[Edit: someone created an AppleScript to query file sizes - I didn't test it, yet: https://discussions.apple.com/docs/DOC-250000422 ]

Since you're willing to write since code apparently: have you checked the internal databases? They're just sqlite, moderately understandable at a glance last time I looked.

Looks promising - will look into it!

So it appears you can conveniently query the DB for filename (+directory), UUID, and (original) file size. But I can't find a good way of opening a Photo by filename or UUID.

After all, the file might not be present locally, so opening it should go through the Photos.app. But once you call AppleScript to open the file, you might as well use the AppleScript to comb through the database like the script I linked to earlier.

Check out this which I believe has that feature: https://github.com/RhetTbull/osxphotos

Side note, been a while sense i looked, is there a sqlite db for the contacts table?

Yes

I think that all "photos" or "videos" are just a view of the underlying "photo or video object". If you crop a video, the full-size video will remain. Only if you export the video, it will be cropped and the smaller file size will manifest.

Yup, the Photos app keeps the unmodified original file, and then any edits/crops are stored separately. You can always revert to the original file and redo your edits. So they might be storing multiple copies of the same image, with and without edits.

Which API were you looking at for "file size"?

I was able to get the size data from Photos.app with the PhotoKit API [1]. I've only tested it with my library of ~26k items, but it was useful for getting an indicator of the biggest items. (Although I didn't think to check whether exporting a 1GB video caused my iCloud usage to drop by 1GB.)

[1]: https://alexwlchan.net/2023/finding-big-photos/

Ahh, I did consider PHAsset.fetchAssets but my understanding was that the method will download the file if not present locally - which wouldn't be acceptable for an app, I guess.

Do you know more? The introduction says "Retrieve asset metadata or request full asset content.", but I can't find clarification when it actually accesses full content.

Yeah, that was what I thought when I first worked with these APIs! But when you use PhotoKit, you have to explicitly opt-in to downloading files from iCloud.

AFAICT, PHAsset is only metadata. When I'm downloading the full-sized images, I use PHImageManager.requestImage() and pass in the PHAsset I'm looking at [1][2]. I know there's something similar for video, but I've never used it.

You can control the behaviour by passing a PHImageRequestOptions instance. This includes an isNetworkAccessAllowed bool which controls where Photos.app will download the file from iCloud if not present locally, and it defaults to false.

[1]: https://developer.apple.com/documentation/photokit/loading_a...

[2]: https://developer.apple.com/documentation/photokit/phimagema...

[3]: https://developer.apple.com/documentation/photokit/phimagere...

It’s (much) more complicated than that. HEIC format allows multiple “frames” to be stored in the container, each one derived from the source (or each other). So there may literally be just one file but it has (algorithmically) the definition for generating (losslessly) the other files, too. So there isn’t/wouldn’t be even a separate “smaller copy” at all as it is generated on-the-fly.

this is the best plausible reason for why the file size differs. I've also noticed some unepxected media restoration (thought I cropped/edited a video, it is still there in full length and resolution).

This also helps explain why my iPhone storage always seems to be at its limits, despite my obsessive management.

Anyway, we all know foss unix ftw :P

thought I cropped/edited a video, it is still there in full length and resolution

It's been possible to create a clip from a video file that merely changes what parts of the video are displayed without effecting the data in the original since the Classic Mac OS days.

If you want to completely remove unwanted portions of a video to reduce the size without a loss of quality, there are many options. LosslessCut is a cross platform option that is both free and open source.

https://github.com/mifi/lossless-cut

You might know this already but the Photos catalog/photiolibrary is just a folder. You can actually right click and “show package contents”. From there, you can use something like Daisy Disk to display all the files based on their size. A simple drag and drop has worked for me. Once you find the large file, you can search for the name in the photos app. I’ve deleted some quite large videos this way to clear up some space.

This only works for files saved locally though. Doesn’t help with large files on the iCloud server.

On top of that, something I still haven't looked into the details on: when you import RAW+JPG photos, the "original" can be set to either one. You can import and make some edits without realizing which is being used unless you go through the menus. So my freshly imported pic could be 5MB or 50MB despite just being a single thumbnail in the library.

The thing I detest the most is the weird tier that jumps from 200GB to 2TB. There’s no way to pay for incremental storage for 500GB or 1TB, which would be perfect for many families.

In today’s age where storage is a commodity, this ought to be priced per GB used.

You can chuck as many 50GB increments on your 200GB if you need it.

Really? How? I can only jump from 50 GB to 200 GB when I look at the available plans

Actually I may be wrong there. I was on 200Gb with Apply Family but have stuck 50GB on top of that for £0.99 month.

I can only swap that out for a 200GB increment. This is weird because I'm sure you used to be able to do that.

Oh okey, I knew that sounded too good to be true.

I didn’t even get an option for 200GB increment.

That's weird, so you can only go from 5 GB -> 50 GB -> 2 TB?

Sorry, I meant I never got any additional increment options, even when I was at 200GB.

So I jumped 5, 50, 200, 2000.

Right, it's the same here. The poster clarified what they actually experienced.

You can stack the storage of your iCloud subscription on top of your Apple One subscription, once.

After you subscribe to Apple One, you can buy more iCloud storage if you need more. With both Apple One and an iCloud+ plan, you can have up to 14TB of total iCloud storage.

https://support.apple.com/en-us/108104

That way they can subsidize the free tier

The free tier is 5gb which nearly grows on trees these days

It still costs money and corporations are expected to produce a profit by their shareholders

Surprisingly, I've received a similar storage notice from Apple before the holidays and I decided to download all photos/videos into my own media server instead of hosting it on iCloud. There is not an easy, straight-forward way to download your archive from iCloud. I am slowly getting there using multiple machines and devices.

The issue with the recent changes with Apple is that they increase the prices for no good reason. We're always going to take photos/videos and their sizes keep increasing with modern tech and capabilities.

What about the Photos app on MacOS if you have one? I keep a local copy of my iCloud Photo Library and sync it through the MacOS Photos app.

Photos app for offline storage is also woefully neglected by Apple. They don't really design it for use with an external drive (you are expected to locate your library on a internal storage or permanently attached storage). The reason is that the photo library constantly gets corrupted on external drives, forcing a long rebuild/repair process.

Often the photos app doesn't even detect the iphone even when plugged in, and it's a serious bug that Apple has neglected for years.

Everything Apple does seems to be designed to drive hardware sales. Why support external drives, when you can be up-sold for larger internal storage (at a huge markup)? The "Photo Library" could simply be a database file with references to photo locations, alas that might confuse Mac/iOS user with "files" vs "photos".

What about the Photos app on MacOS if you have one?

If you ever try a large-scale import/export into macOS Photos be prepared for 100% CPU, endless spinner cursors and the process ending up killed because it ran out of memory.

> There is not an easy, straight-forward way to download your archive from iCloud.

Let me revolutionize your life:

https://github.com/icloud-photos-downloader/icloud_photos_do...

Thanks! That tool is what I was looking for. I think you saved me hours of frustration.

AFAIK, it doesn’t support shared iCloud libraries but https://github.com/steilerDev/icloud-photos-sync does.

Edit: Seems like it has gotten support since I’ve last looked: https://github.com/icloud-photos-downloader/icloud_photos_do...

There is an easy way to get your Data. It's just kind of hidden.

It's works the same way as Google Takeout. To get to it do this:

Sign in to your Apple ID account page at appleid.apple.com on a Mac, iPhone, iPad or PC. Go to “Data & Privacy” and select “Manage Your Data and Privacy.” On the following page, go to “Get a copy of your data” and select “Get started.”

privacy.apple.com is a neat shortcut, it's the first option there.

There is not an easy, straight-forward way to download your archive from iCloud.

I use PhotoSync[0] to copy photos from my iPhone to a NAS. It's an excellent program.

It will even download photos from iCloud as needed and can do format conversions. I run it every few days to push new photos to my NAS so that I've always got a local copy (these also get backed up to Backblaze B2 nightly). The format conversion lets me keep HEIC+JPG pairs of my photos, so I have the original and something that's more readily usable.

What I really want is something that will do the same thing, but with iCloud Drive. I keep a bunch of stuff in there and it bothers me that I don't have a reasonable way to back it up. Apple's recommended methods[1] leave a lot to be desired.

[0] https://www.photosync-app.com/home

[1] https://support.apple.com/en-us/HT204055

Interesting article. I have my own problem, that caused me to have to upgrade my iCloud plan, and thus may not be a high priority fix for Apple.

If you shoot RAW+JPEG (not a super-rare thing to do, for photo enthusiasts) then Apple Photos links the two images. Which is useful, rather than having a bunch of kinda-duplicates littering your library you can easily toggle between RAW and JPEG.

But this combining, along with the file system design described in this article, makes it impossible (as far as I can tell, anyway) to easily separate them and delete the RAWs. So years later, I have HUGE RAW files that I'll never touch that I can't delete, because I want to keep the much smaller JPEGS.

Any method that I've found to clean them up (exporting the originals, deleting them from the library, and then re-importing the JPEGs only seems easiest) will lose all of the years of metadata that I've built up in the library.

So I have to upgrade.

Not in your situation at this point, but as a photo nerd I will eventually be so I took to Google.

Any method that I've found to clean them up (exporting the originals, deleting them from the library, and then re-importing the JPEGs only seems easiest) will lose all of the years of metadata that I've built up in the library.

Apparently when you File/Export Unmodified Originals it will export the RAW+HEIC and a separate sidecar file containing the metadata. You can then move the RAW file away and import the HEIC file, which will autoimport the sidecar metadata file too.

You lose edits though, although it seems you can "copy edits" somehow. Surely a technically inclined person can AppleScript their way through this...

Yet it seems needlessly cumbersome and should be a built-in function in Photos.app, it's clearly not prioritized because it helps funnel people into higher iCloud tiers.

it's clearly not prioritized because it helps funnel people into higher iCloud tier

Which would be annoying by itself. But then iCloud has an arbitrary limit of 2TB as well, which does not seem to be able to be increased.

The 2 TB limit (that I've been bumping up against for nearly 2 years) has finally been raised - a few months ago they added 6 TB and 12 TB tiers

Cool! I don’t see myself needing them any time soon, but max 2TB was too close for comfort.

Possibly stupid question: why can’t the metadata be exported and imported? Is there other metadata aside from the exif data? Or does Apple not export all of it? And in case you’re talking about additional features like face recognition, doesn’t the app do that again once you imported the jpeg?

Im not in Apple ecosystem at all, so I don't have an answer to your question.

But in RAW/JPG world, the metadata and edits is already a solved problem with Sidecar files + EXIF data. Sure, EXIF fields are kinda messy but I'm sure it's better than Apple has rolled by their own.

ugh yeah, currently in the same boat and haven't taken on the work to deal with this hassle. definitely looking for a solution!

Am I reading this right?

If this is widespread, this could be seen as apple bloating figures to push people to upgrade, which could lead to a lawsuit, no?

IANAL

Funny, this was may take on the article too. I suppose one could download x amount of files from one cloud storage and upload it to another to see if there is any obvious discrepancy. Perhaps Apple are still using their old file system format for icloud? It's been around for a good few years now and perhaps they simply haven't bothered to ever change the type of storage format they use. Maybe those with older accounts have their images / videos on the 'older' drives?

I don't believe iCloud is a filesystem, it's not block-level. It's object-level, like S3, with a bunch of OS-side jank to make it look like a conventional directory and filesystem (which often fails miserably and dangerously).

It's similar levels of reliability to an FTP account mounted with curlftpfs[1] - except the latter at least fails in understandable ways and can be debugged.

[1]: https://news.ycombinator.com/item?id=9224

That's exactly my mental model of it as well, some janky home made s3 fixed with tape. I really advise anybody to save their data somewhere else as well before in inevitably stops working.

'Failing miserably and dangerously' is the new 'Move fast and break things'

If you edit a photo/video in file rather than saving as a new file, iOS retains the original file so that you can undo/revert. It doesn’t show the original anywhere in Photos or iCloud gallery. That might be the reason.

You can also edit with third party software, and they can embed whatever metadata they want to allow those edits to be modified, so there might not actually be an upper bound on how much larger you can make a photo or video by editing.

this could be seen as apple bloating figures

No, the OP doesn’t take into account that adding media to Photos is not a simple “copy and store a file”. The Photos app (like all apps) has its custom Photos Library file format. So when one ads a picture or a video, the Photos app analyses it and stores all kinds of metadata that’s needed for the Photos app to work, including edit history and other bits. This is what eventually gets synced to iCloud.

I've tried iCloud, Onedrive, Google Photos. Out of all these three, I stuck with Google Photos on my Iphone.

iClouds bad pricing and storage issues led me to switch to Google Photos, It's way better. The lack of native support on iOS can be a little cumbersome though.

The only thing I wish these cloud providers offered was a way to deduplicate photos / videos. That would make my life so much easier.

iCloud Photos has deduplication

Yeah I ran it and shrunk my library by half. It was kinda scary since I had 100000 photos, I couldn’t really inspect the process

It literally shows each photo set it thinks are duplicates, you can literally manually verify

I didn’t have time to manually verify 10s of 1000s of duplicates. I spot checked and it looked fine..

I've used an app called PhotoSweeper[1]. It can access your Photos library directly. You can manually review them side-by-side or let it run automatically. You can also set thresholds of % similarity to reduce false removals. I believe it can also move the duplicates into a separate album for review later.

https://apps.apple.com/us/app/photosweeper/id463362050?mt=12

The only thing I wish these cloud providers offered was a way to deduplicate photos / videos.

Same here ... but of course that would make it easier for customers to not have to pay more, so sadly don't think it's ever happening.

What makes it worse is even if you run a deduplication app on the folder your syncing to Google Photos, the synchronization is one way only, so the changes does not remove the duplicated items on Google Photos.

Good timing..I've been getting the same warning mails (though 50gb). Was planning to go through photos but checking vids first makes sense

Been getting the same warning emails (200GB in my case). So I tried this now, but it doesn't seem to highlight anything after pasting in the gist into Chrome. Does it work for you?

If you have a Windows machine lying around, the iCloud app lets you download all your media out of iCloud Photos. Once it's all on Windows it's a lot easier to find the largest files!

You can also do that with Photos (leaving "optimize storage" off) and downloading all to your Mac, I believe. I haven't actually done yet that since it is quite a large download.

the windows icloud app doesn't download the highest quality versions of each file though right?

after pasting in the gist into Chrome.

No idea what this means so can't tell you whether it works for me

The original article had a JavaScript snippet you post into the browser to do the highlighting for you.

That's one of the reasons I self-host. I just don't trust the cloud providers regarding clarity and transparency... even if my self-hosted solution was far less reliable, less secure and less performant (I imagine it's not ;), I probably wouldn't change.

I personally use immich[1], a very complete solution with iOS / Android App, Server-Component and Sync / Backup option.

[1]: https://immich.app/

It looks interesting but the warning at the top tells me I should wait for it to become more stable?

Yeah, that's what I thought... but using it for 6 months now they have been pretty reliable regarding upgrades and new features.

The features I appreciate the most are:

  - Auto-Synchronisation on Android AND iOS (this is hard to find in any other app)
  - Photo sharing (Accounts can have partners to share all their photos with - ideal for me and my wife)
  - Deduplication
  - The Web Interface

The feature I miss the most is tags[1].

Once the app stopped working but there was a clear message on the repository / homepage that the server has to be upgraded. Since it is docker based, it was very easy to upgrade without losing any data. Same applies for backup...

So if you ask me, there is no need to worry - but I would not use it as only option to store my photos and it does not replace an (off-site)-backup.

[1]: https://github.com/immich-app/immich/discussions/1651#discus...

I would if you want to use it as backup. But if you just use it to display photos and have another process for sync it’s absolutely fantastic!

To me it’s better than any other interface I’ve tried. Commercial or home lab.

I think Apple just hasn't realized yet what a great sales opportunity a UI for quota management is. There's no better place to sell more storage. That's what Google does. See https://one.google.com/storage and https://photos.google.com/quotamanagement which work well for finding stuff to delete, in the hopes that eventually you'll tire of deleting things and go ahead and buy more storage.

Slightly off topic: anyone know how to backup iCloud data (mainly photos and videos, but also notes) to an outside platform?

I’m an Apple fan, but still don’t like the idea of having all my digital life on iCloud only.

Disclaimer I haven't tried this yet, but I found this project recently that supposedly downloads all iCloud photos (e.g. to a NAS for local backup). https://github.com/icloud-photos-downloader/icloud_photos_do...

I've been using a version of this for Unraid for over a year. Downloads everything from my iCloud storage to my home server.

You need to re-authenticate every 90 days due to 2fa restrictions but it works great.

I use Image Capture with my phone plugged in to save them on an external hard drive.

This is not for incremental backups but for all data tied to your Apple ID — Apple provides a data download option to get your contacts, photos, videos and a whole lot of other data. You can visit this KB [1] for details and instructions. There are some exclusions, like data that’s end-to-end encrypted.

[1]: https://support.apple.com/en-us/102208

I'm pretty sure that iCloud stores the full version and downloads a lower quality optimised version to your phone.

If you open iPhone settings and browse to:

  > Apple ID > iCloud  > Photos

There is an option to 'Optimise iPhone Storage' which is enabled by default. This states:

  > If your phone is low on space, full resolution photos and videos are automatically replaced with smaller device sized versions. Full-Resolution versions can be downloaded from iCloud at any time.

This seems perfectly reasonable to me.

Downloads should still return the original file though, so that can't be used to explain the storage size discrepancy.

There are two kinds of download options on the iCloud website, the “full quality” (e.g. HEIC file for photos) or a “quick” / “small” download which is a JPG.

This doesn't explain the case when the author uploaded and downloaded a new file, and there was no big diff in file size. The file size diff only seems to be happening to the author's older video files.

This isn't related.

This deals with the storage on the phone, which you can find in Settings > Manage storage. It has nothing to do with the storage in the cloud.

On this topic...anyone know the best way to export an entire iCloud Photo Library at full resolution?

If you have a Macbook, you can sync iCloud library at full resolution using Photos app and then export.

There is an export service that returns a archive

Interesting, also, Apple should make a mid-tier of iCloud. If they had a 500gb (or 400 or whatever) they could have gotten this guy to pay more.

At the cost of many others not buying the more expensive plan because it was their only option.

Surely there are plenty of us who would happily (or at least less reluctantly) pay incrementally more as our storage needs increase, but who baulk at trebling what we pay in one jump.

Compression?

Maybe adding a lot of metadata? I know AWS block storage adds an additional 4kb metadata file for everything you upload, maybe icloud doing something similar?

Must be bad compression if the files are getting larger?

Additionally, if you take a 4MB photo and add it to a shared album with 5 people, those 5 people all have to use up 4MB from their storage. Instead of reusing the same image and having pointers to it, it looks like it's "copied".

That doesn't make sense to me. Apple says here that photos in a shared album do not count against your iCloud storage.

https://support.apple.com/en-us/HT202786

Shared Albums don't count against your iCloud Storage.

It would make sense if they save the video in a few formats for being able to load them fast in the Apple apps, most video hosts do the same.

But that shouldn't count in the user's storage quota.

I’ve been on a similar hunt. My devices are littered with copies of videos that were shared via iMessage, WhatsApp, and other means. Each of these apps then store the copy locally which then gets backed up in iCloud.

Unfortunately I haven’t found a way to automatically delete all the videos/photos that I send which still have the original in the photo stream. It would be awesome to be able to automate that.

This isn't perfect, nor automated, but you can search the Messages app for photos/videos and delete them all. Assuming you've saved them already, this should clear up some iCloud space.

https://www.igeeksblog.com/how-to-use-smart-search-filters-i... https://www.igeeksblog.com/delete-multiple-imessage-photos-a...

iCloud backup is real mess.

Free 5GB is not enough for backing up iPhone system data any more.

Most insane is that if you remove some apps from backup, then it iCloud usage goes from 4.5GB to 4.6GB.

and you're basically given to choose either stop backing up your iMessage or pay $1/mo for the extra 50gb

I'm being sarcastic here, but the solution could be quite simple: iCloud is billing you (both in terms of storage size and money) for both your data and the redundant copies.

Other companies would build the redundancy costs into the final pricing, but Apple is known to "think different".

Apple needs to face criminal penalties for the dark patterns with which they prey upon their users.

Apart from the iCloud fiasco, I very much like these kind of programming. We generally lost the fun of it try to make enterprise/corporate programs, we forgot to use them for personal tasks.

there is something weird happening on iCloud, my phone storage is off even though the total size of data isnt as big as it says, i throw a lot of .zip/rar files on icloud and it seems like the data on icloud is more than it is on my HDD....

Videos anonymized by slow internet loading

I chuckled!

I believe the size difference has to do with the encoding, can’t tell for a fact since you didn’t show that part, but maybe it gets re-encoded when you download it hence the difference.

I need a good way to cleanup a gmail account…

Apple interested in maximizing recurring subscription revenue?

You don’t say. I’m sure this is just a harmless bug and not the source of $20M extra revenue per year across their hundreds of millions of active iCloud users.

Can confirm, have some ideas: Apple uses different file formats "internally" and also different resolutions, maybe saving them all in parallel as "one" video, which might cause this issue. So the total space per image is actually the sum of original .mov, optimized .mp4 and thumbnail image.

Also, (automated) version history might have caused this! Maybe (automatic) image optimiztion saved old and new versions of everything.

Thanks for the idea/solution!

I converted this into a TamperMonkey/Greasemonkey script. Also added a feature to "hide" all elements that do not match the threshold.

https://github.com/seffignoz/icloudcleanup

I am so tired of these patterns. Google Photos also does not allow any easy or good way to clean up junk. Why would they? They want you to run out of space so you can pay for their cloud storage. They also have low incentive to keep photo/video storage space-efficient for the same reason.

The thing that is taking up loads of my iCloud storage is the 'live' video version of photos where the tiny snippet of movement has a larger filesize than the photo that it accompanies. The only way I have found so far to deal with this is to download the files locally, delete from iCloud, delete the local videos and upload the remaining still images. This is time consuming, clunky and makes me nervous that I'll delete something or lose something important in the process. Is there anything out the that help to automate that process but where there is some level of editorial control so that I can identify and keep the handful of 'live' images that I might actually want to keep?

It's better to self-host

It's called a bug 'turning a blind eye'.. ie. a scam.

Tangent but does anyone else find iCloud sync random / slow?

The smart thing about having both an iPhone and a Macbook should potentially be; snap a bunch of photos, instantly have them available on the computer - but no. Apple apparently chooses a random time depending on 100 factors to upload the photos in the next 30 minutes to 7 days.

So you often have to airdrop a bunch of photos files completely invalidating the purpose of the sync function.

It seems like a giant class action lawsuit with a juicy target if Apple is overbilling customers on usage for videos in iCloud. I know lots of people paying for more storage in iCloud would appreciate the reprieve of lower costs.

There’s will be a time when there will be many open (or relatively more open) devices like FairPhone et cetera; and OSes that do not track users and suffocate you with dark patterns and hostility all around - like Graphene and Lineage; and hopefully they’d all be able to join their hands together and not reimplement the wheels of N types for the 1000th time all incompatible with each other. Hope is that there will be services and softwares on top of that spec and API of these OSes that will be the user’s choice to pick - for contact, for calendar, SMS, IM, call , media, sync, backup, device account management etc.

And such devices will be widely available, also in non-first world countries, with proper OEM warranties and support. Hell, a local manufacturer can just build for that spec.

I know it’s like a wishful dream. But if this happens, and when this happens, I hope a lot of us will be able to breathe free and hopefully would be candidly able to shit upon legacies of those so called fucking visionaries, who were barely not subhuman and were just rather pathetic jerks, who ensured such shit-show of walled gardens and opaquely implemented utterly inferior systems where users go and get stuck in one of the two houses of the duopoly to experience a lot of shitty things including, but not limited to, some variation of Stockholm syndrome and clear and conscious apologism.

Why? Because neither of the two is acceptably good and they have become so big and they have closed it down so much that nobody else can even make a dent even if they try. And they try!

So yeah, until then I will rant and feel shitty about both my iPhone 14 (as they call it - my “daily driver”) and Pixel 5a (my bread and butter phone; form factor wise less shitty one though).

Don’t you remember how painful it was to import/export an mp3 file into an IPod?

It was especially painful on a Windows PC.

The biggest selling point of other mp3 players for me was the ability to transparently copy and paste files into the filesystem.

In terms of iCloud cleanup, has anyone found a way to clean up iMessage attachments > x years old, greater than n MB or from certain contacts? It would clean up so much space for me. Apple only clears > 1 year old from the settings.

Can't you just use the Apple Shortcuts app to do a lot of this?

Pretty sure that can enumerate and query things like the filesize of the photos and videos in your iCloud.

With the iCloud desktop app for windows I'm able to sort by size...

That's an interesting bug/problem(?) he ran into.. Though, on the topic of reducing storage usage, in the Photos iOS app, you can remove photo duplicates under "Albums" at the very bottom -- if you have duplicates that it recognizes, there will be a line item for this, otherwise the line item will disappear.

Just discovered you can see file sizes for some items under > icloud > recommended for you

Finds duplicates, screenshots and big vids. Had a random vid that was 1.8gb for reasons unknown

Doesn't always seem to be a accessible though

Was the video with size mismatch edited? Apple keeps both versions (same goes for photos, they include image and MOV for live photos).

some possible reasons to investigate:

1. older videos have a larger file size because they are encoded less efficiently (h264 vs h265)

2. gigabytes vs gibibytes (like how a 500GB drive formats to ~465 GiB)

3. rounding errors

4. Apple fucking its customers

5. combo of the above