Downloading Every Tweet/Post (Text, Images, Videos, Replies, etc)

[Xylemon]

Hello, first off I want to say thank you for this tool! As an archivist, gallery-dl so far is a perfect companion to YT-DLP, with one exception:

I'm trying to download an entire Twitter profile, this includes their text only tweets, images, videos, etc. However, if I run "gallery-dl https://twitter.com/user/" (replace 'user' with an actual username) it won't download their text tweets, just media, but if I run "gallery-dl https://twitter.com/user/tweets" it will do what I desire, but gallery-dl will just stop after a certain amount of tweets. I was testing this with "gallery-dl https://twitter.com/dril/tweets", and it will always give up around early 2021.

I have a server in which I am automating YouTube channel archiving with cron/scripts, and I want to be able to do the same with gallery-dl, so being able to do "gallery-dl twitter.com/user" in a cronjob would obviously be desirable.

I am using the latest pre-compile Linux binary release:

gallery-dl -v
[gallery-dl][debug] Version 1.22.0 - Executable
[gallery-dl][debug] Python 3.10.4 - Linux-5.17.9-1-default-x86_64-with-glibc2.35
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.9

Here is my config I'm using for this purpose, I have gallery-dl set to change the metadata file timestamp and to download posts as json/txt, but I've removed all other sites for testing:

{
    "extractor":
    {
        "base-directory": "./",
        "parent-directory": false,
        "#": "set global archive file for all extractors",
        "archive": null,
        "cookies": null,
        "cookies-update": true,
        "proxy": null,
        "skip": true,

        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "retries": 4,
        "timeout": 30.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,

        "#": "replace invalid path characters with unicode alternatives",
        "path-restrict": "auto",
        "path-replace": "_",
        "path-remove": "\\u0000-\\u001f\\u007f",
        "path-strip": "auto",
        "extension-map": {
            "jpeg": "jpg",
            "jpe" : "jpg",
            "jfif": "jpg",
            "jif" : "jpg",
            "jfi" : "jpg"
        },

        "#": "write plaintext descriptions, and json metadata",
        "postprocessors": [
            {
                "name": "mtime",
                "event": "post"
            },
            {
                "name": "metadata",
                "mode": "json",
                "filename": "{post_id|tweet_id|id}.json",
                "mtime": true
            },
            {
                "name" : "metadata",
                "event": "post",
                "filename": "{post_id|tweet_id|id}.txt",

                "#": "write only the values for 'content' or 'description'",
                "mode" : "custom",
                "format": "{content|description}\n",
                "mtime": true
            }
        ],

        "twitter":
        {
            "username": null,
            "password": null,
            "cards": true,
            "conversations": false,
            "pinned": true,
            "quoted": true,
            "replies": "self",
            "retweets": false,
            "syndication": false,
            "text-tweets": true,
            "twitpic": true,
            "users": "timeline",
            "videos": true,
            "postprocessors": [
            {
                "name": "mtime",
                "event": "post"
            },
            {
                "name": "metadata",
                "event": "post",
                "filename": "{tweet_id}.json",
                "mtime": true
            }
        ]
        },
        "ytdl":
        {
            "enabled": false,
            "format": null,
            "generic": true,
            "logging": true,
            "module": "yt_dlp",
            "raw-options": null
        }
    },

    "downloader":
    {
        "#": "restrict download speed to 1 MB/s",
        "rate": "1M",

        "#": "show download progress indicator after 2 seconds",
        "progress": 2.0,

        "#": "retry failed downloads up to 3 times",
        "retries": 3,

        "#": "consider a download 'failed' after 8 seconds of inactivity",
        "timeout": 8.0,

        "#": "write '.part' files into a special directory",
        "part-directory": "/tmp/.download/"
    },

        "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "colors": {
            "success": "1;32",
            "skip"   : "2"
        },
        "skip": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}

[nisehime]

This is probably due to gallery-dl downloading /media timeline by default. It looks like you can write scripts, so you just can do like this

1. Run gallery-dl "twitter/user/with_replies" or "twitter/user/tweets" if you don't need their replies.
2. Somehow copy the tweet id of the last downloaded tweet.
3. Run gallery-dl "twitter.com/search?q=from:{USERNAME} max_id:{COPIED-ID}"

But be aware that when using "twitter/user/with_replies" gallery-dl unfortunately downloads non-target user's tweets from a conversations too. Or rather conversations in both normal and replies timelines are troblesome and will miss tweets. (#2252 (comment)).

[Xylemon]

Edit: Sorry I somehow missed the "copied ID" suggestion, I'll try this! When you say last downloaded tweet, you mean oldest or newest I downloaded?

Thanks for the reply, as I mentioned I did try "twitter/user/tweets" but it seems to stop after a certain number of tweets, or it's not grabbing every tweet. I'm not exactly sure, but it's not grabbing everything (I have filtered) even with "/tweets". I'm hoping to grab every text tweet, I hope this isn't beyond the scope of gallery-dl

Here's an example: If I try "twitter.com/ID_AA_Carmack" and ""twitter.com/ID_AA_Carmack/tweets", I only am able to download around 1,023 tweets, but his profile has over 15,000, and I'm certain the rest aren't replies. In my first example, dril was also doing something similar, and his account is also mostly text tweet based.

Also thanks for the with_replies suggestion, I'll try that in the future but I'll wait till filtering on that is more exact.

[nisehime]

it seems to stop after a certain number of tweets

This is the limitations of twitter's web API. It returns only latest ~1000 tweets. You can use search to get older tweets, but search also have some limitations and random behaviour. It can skip some tweets or return nothing at all.

When you say last downloaded tweet, you mean oldest or newest I downloaded?

Oldest

[Xylemon]

it seems to stop after a certain number of tweets

This is the limitations of twitter's web API. It returns only latest ~1000 tweets. You can use search to get older tweets, but search also have some limitations and random behaviour. It can skip some tweets or return nothing at all.

When you say last downloaded tweet, you mean oldest or newest I downloaded?

Oldest

OK, I tried entering the oldest tweet ID I had saved and sadly it returns nothing:
[twitter][info] No results for https://www.twitter.com/search?q=from:ID_AA_Carmack max_id:76690078931488768

Oddly enough, entering the newest tweet ID got me some more tweets, so it sounds like the behavior you mention with Search (i.e. being random), but not all of them, which unfortunately seems to be expected.

I suppose I could try something very manual like Nitter's search + some Firefox addon to generate a list of URLs for all the tweets I don't have, but I'd really not have to do something so time consuming.

[nisehime]

https://www.twitter.com/search?q=from:ID_AA_Carmack max_id:76690078931488768

It works for me when I'm logged in. (See also #2658).

Oddly enough, entering the newest tweet ID got me some more tweets

Those were just the replies I suppose, not exactly randomness of the search.

I suppose I could try something very manual like Nitter's search

Well, maybe that's the only alternative. I don't know how nitter works though, maybe it has the same issues to some degree

[Xylemon]

https://www.twitter.com/search?q=from:ID_AA_Carmack max_id:76690078931488768

It works for me when I'm logged in. (See also #2658).

I suppose I'll have to get a dummy account and try this, though I was really hoping to avoid needing an account especially when there's no NSFW content on it.

Well, maybe that's the only alternative. I don't know how nitter works though, maybe it has the same issues to some degree

It's too bad gallery-dl can't use Nitter's date search feature, it seems like Nitter has no issues getting everyone's tweets: https://nitter.42l.fr/ID_AA_Carmack/search?f=tweets&since=2010-01-01&until=2010-12-31

[nisehime]

It's too bad gallery-dl can't use Nitter's date search feature

You might open a request issue. (Though considering that gallery-dl's nitter support is just translating nitter requests into twitter's API, this won't help you)

[Xylemon]

I also am trying to save another account under a ~1000, and it too fails to do so, only grabbing 300+, even with the search and max_id method.

[nisehime]

Are you trying /with_replies timeline? I don't know how twitter counts the number of tweets but I guess that's all of them including retweets and replies. Although I suspect this counter is actually buggy. I remember on some profile it showed more number of media than there actually were.

[Xylemon]

I tried your suggestions @nisehime, did some slight changes to my config, and gallery-dl does exactly what I want now!

Seems I was misunderstanding the replies setting, I changed "replies" from "self" to "true" (and "syndication" as well, but I don't know if this affected anything) and gallery-dl started downloading the missing tweets. Doing that with the URLs you provided like "search?q=from:USER" and "/with_replies" seem to download everything what I want, didn't even need an account either.

Thank you @nisehime for your help and patience. Gonna consider this solved.

[nisehime]

Keep in mind that can break at any time considering how restrictive twitter becomes. So I recommend to make an account at some point.

By the way, this link

https://www.twitter.com/search?q=from:ID_AA_Carmack max_id:76690078931488768

works for me while I logged off right now. Meanwhile it certainly didn't previous time. So it might be just a coincidence for you. Again, with a profile it probably would be more stable.