Missing Photos - 7404 out of 27803

[raoulwouters]

I got the integration working and it is successfully downloading. However Iclouldpd only 7404 files while my icloud libary has 27803 photo's and 688 video's. How do I download the remaining? What am I doing wrong.

[raoulwouters]

The send run, it recognized even less photo's

[infyx]

I am seeing the same issue on a new install. Have about 10k total and its only seeing about 6k.

[boredazfcuk]

What folder structure are you using?

Something like none or just using year will greatly restrict the number of files which can be downloaded.

[raoulwouters]

The standard folder structure. So year/Month/Day

[AndreyNikiforov]

Any chance you have shared libraries, @raoulwouters and @infyx ? Only assets from one library at a time can be downloaded and by default primary one is selected.

[AndreyNikiforov]

Any chance you have shared libraries, @raoulwouters and @infyx ? Only assets from one library at a time can be downloaded and by default primary one is selected.

Another thought - raw files. They were not supported until recent version of upstream project.

[raoulwouters]

No, I have some shared albums. But they are very small. No shared Library

[infyx]

Any chance you have shared libraries, @raoulwouters and @infyx ? Only assets from one library at a time can be downloaded and by default primary one is selected.

Now that you mention it, yes I have a sizable shared library and albums. Been a while since I checked so I’ll have to see if all of them downloaded eventually.

[boredazfcuk]

I've made a slight change to the way it counts the number of files available in icloud. Please update to the latest version and see if that works.

If not, try disabling the check by setting the skip_check variable in icloudpd.conf to true.

[raoulwouters]

@boredazfcuk unfortunately this did not help. I first upgraded the package and afterwards it found about 365 extra photo's
Then I set skip_check to true and the total #photo's found is 6299 so less then before and far from the now 28554 photo's in my library

[boredazfcuk]

I noticed something similar today. I found a bunch of photos in an album which was created by an app. None of the files that were saved to the album were downloaded. I've also noticed that photos in the "Hidden" album will not be downloaded either.

Try finding a missing photo or two, then remove them from the album, trigger a download, and see if they appear.

[luke-liuqi]

assets

as you mentioned, i meet same situation, But I have a shared photo library on Icloud, any chance to download all libraries?

[kushpvo]

What folder structure are you using?

Something like none or just using year will greatly restrict the number of files which can be downloaded.

Could you explain why folder structure as none would greatly restrict the number of files which can be downloaded? I'm using none and i have a library of about 26k photos and videos

[boredazfcuk]

When taking photos, Apple names the files sequentially, e.g. IMG_000.HEIC, IMG_0001.HEIC all the way through to IMG_9999.HEIC... that's a total of 10,000 individually named files.

It will then loop back around and start calling the photos IMG_000.HEIC, IMG_0001.HEIC etc again. This doesn't matter on an iDevice as they store all the photos in a database. Apple could name every single photo IMG.HEIC and it wouldn't matter to the iDevice.

However, when storing photos in a filesystem, every file in a folder needs to have a unique file name. If you have your folder structure set to none you only have one target folder for all your files. icloudpd will download IMG_0000.HEIC, IMG_0001.HEIC all the way through to IMG_9999.HEIC, loop back around to IMG_0000.HEIC and go "hey, this file already exists in the destination folder, so I can skip downloading it". In reality the two photos are different, but only the first one is downloaded.

Having folder structure set to Year allows you to have 10K photos per year, having it set to Year/Month allows you to have 10K photos per month, having it set to Year/Month/Day allows you to have 10K photos per day. This makes it very unlikely that you will have two photos with the same name.

There has been a recent change to allow for two types of de-duplication, see file_match_policy in the configuration guide for more details. Please be aware that changing these could trigger a re-download of your entire library.

[kushpvo]

When taking photos, Apple names the files sequentially, e.g. IMG_000.HEIC, IMG_0001.HEIC all the way through to IMG_9999.HEIC... that's a total of 10,000 individually named files.

It will then loop back around and start calling the photos IMG_000.HEIC, IMG_0001.HEIC etc again. This doesn't matter on an iDevice as they store all the photos in a database. Apple could name every single photo IMG.HEIC and it wouldn't matter to the iDevice.

However, when storing photos in a filesystem, every file in a folder needs to have a unique file name. If you have your folder structure set to none you only have one target folder for all your files. icloudpd will download IMG_0000.HEIC, IMG_0001.HEIC all the way through to IMG_9999.HEIC, loop back around to IMG_0000.HEIC and go "hey, this file already exists in the destination folder, so I can skip downloading it". In reality the two photos are different, but only the first one is downloaded.

Having folder structure set to Year allows you to have 10K photos per year, having it set to Year/Month allows you to have 10K photos per month, having it set to Year/Month/Day allows you to have 10K photos per day. This makes it very unlikely that you will have two photos with the same name.

There has been a recent change to allow for two types of de-duplication, see file_match_policy in the configuration guide for more details. Please be aware that changing these could trigger a re-download of your entire library.

Damn, that makes sense. I've been running the container for few days and hoping that it finally downloads all the files.. now i know why. Thank you so much for the explanation. I'm going to update my folder structure now (even though i'll have to redownload everything)