-
|
Hi, Pretty sure I'm missing something obvious but since I'm new to both python and github I don't know where to look... I've got an archive file that saves picture IDs and skips redownloading them, but I just tested the same download command twice in a row and it just skipped over the 30 already downloaded and downloaded the previous 30 (on twitter, the most recent 30 files were skipped and the second 30 were downloaded). Is there a way to make it only check the most recent files and stop downloading once it hits something already in the archive, instead of skipping over the archive to download unlisted things? I'm not sure if this is done in the config file or if it's done as part of the download command. If the latter, the string I'm using atm is py -3 -m gallery_dl https://twitter.com/authorname/media --range 1-30 Thanks. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
This is controlled by the |
Beta Was this translation helpful? Give feedback.
-
|
I've now had a few days to let accounts build up a few pictures and I edited the config file so the skip option now says "abort:5", but it still seems to be downloading 30 files at a time. Does the archive file wipe itself periodically or something? I'm certain I ran the commands for these accounts not even a week ago so they shouldn't be downloading this many things, if these codes are supposed to stop an account's process as soon as it hits several already downloaded pictures. Thanks. Edit: To be completely clear, what I'm trying to do is run the commands every few days for a bunch of accounts to download any new pictures, stop downloading once it hits a duplicate (or three in a row for safety) and the only reason I have the 1-30 thing is to stop the command from having to download, in some cases, hundreds of pictures I already manually went through in the past. Edit 2: I think I'm an idiot. The skips have been working when I run them on the same day, so I assumed my archive was properly set up, but now I'm thinking the archive I made is the wrong file type and so isn't working properly. I'm not sure how to create an sqlite3 file to use as an archive.. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure what could be causing this, some thoughts:
nvm, I see now you know that it looks for consecutive files. Not sure what's the problem them. You could try running gdl with the verbose flag to check for any problems. You could also try to look into the archive file to see if the expected entries are actually there (sqlitebrowser is a free GUI program that can open it). |
Beta Was this translation helpful? Give feedback.
-
|
You should let gallery-dl create the archive file, it will do so if the specified file doesn't exist yet. |
Beta Was this translation helpful? Give feedback.
-
|
I hadn't realised I needed another package beyond python when I first set gallery-dl up so I never had a way to get gallery-dl to automatically create an archive file for me. I still don't know how to make it do it itself, but I've just installed sqlite3 and manually made an archive file which I linked to in config and it grew from 0 bytes to 20 over the course of a command session, so I'm HOPING that means it works now. Would still probably need a few days to check. Sucks I can't port over the archive data from the notepad file I for some reason thought would suffice, but oh well. If you don't mind, I'd like to leave this open for a few days til I can confirm it's all working right? I'm not sure if you would prefer I close this now and make a new discussion if something goes wrong or keep this one up til I've finally got it sorted. |
Beta Was this translation helpful? Give feedback.
-
|
Feels like there is some misunderstanding here. The sqlite support is built into python, so you don't need to install it separately. And you don't need to create an archive file yourself, you can just add the path/name where you want it to the config and it should be created the first time gallery-ld tries to use it. |
Beta Was this translation helpful? Give feedback.
-
I just looked into this again, and this shouldn't, and for me doesn't, happen.
As said in the other answer: Also, are you sure that gallery-dl is actually loading settings from your config file? There have been a couple of cases where people saved them as config.json.txt or gallery-dl.conf.json |
Beta Was this translation helpful? Give feedback.
-
|
I'm pretty certain it's taking info from the config file as there were a few accounts that weren't downloading until I added authentication information and then started working, and when I do change the skip function the behaviour changes It's just been a bit difficult to interpret the results because, originally, I'd run a 1-30 on a Tuesday and then again on a Friday and it would download 30 files despite the account only having added 3-4 files since I last ran it. I'm pretty sure it's because I misunderstood how to make an archive and made a text file with .sqlite3 as an extension but now I've actually installed the package and made an actual sqlite3 file that seems to grow in size each time I run the commands, so I THINK it should work from here on |
Beta Was this translation helpful? Give feedback.
-
|
Show the censored |
Beta Was this translation helpful? Give feedback.
I just looked into this again, and this shouldn't, and for me doesn't, happen.
--range 1-30only processes the first 30 files, regardless of what happens to them (download, skip, simulate, etc).gallery-dl https://twitter.com/supernaturepics/media --range 1-30always downloads the first 30 files and stops, or it skips those same files and stops if I run it a second time. The same happen…