User ID causes 503 error and trips timelines command

[JanaLasser]

I am collecting timelines from about 30k user accounts. I encountered a single user ID that trips the timelines command and causes twarc2 to crash. Here is the error message I get:

requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://api.twitter.com/2/users/27971300/tweets?expansions=author_id%2Cin_reply_to_user_id%2Creferenced_tweets.id%2Creferenced_tweets.id.author_id%2Centities.mentions.username%2Cattachments.poll_ids%2Cattachments.media_keys%2Cgeo.place_id&tweet.fields=attachments%2Cauthor_id%2Cconversation_id%2Ccreated_at%2Centities%2Cgeo%2Cid%2Cin_reply_to_user_id%2Clang%2Cpublic_metrics%2Ctext%2Cpossibly_sensitive%2Creferenced_tweets%2Creply_settings%2Csource%2Cwithheld&user.fields=created_at%2Cdescription%2Centities%2Cid%2Clocation%2Cname%2Cpinned_tweet_id%2Cprofile_image_url%2Cprotected%2Cpublic_metrics%2Curl%2Cusername%2Cverified%2Cwithheld&media.fields=alt_text%2Cduration_ms%2Cheight%2Cmedia_key%2Cpreview_image_url%2Ctype%2Curl%2Cwidth%2Cpublic_metrics&poll.fields=duration_minutes%2Cend_datetime%2Cid%2Coptions%2Cvoting_status&place.fields=contained_within%2Ccountry%2Ccountry_code%2Cfull_name%2Cgeo%2Cid%2Cname%2Cplace_type&start_time=2022-09-14T00%3A00%3A00%2B00%3A00&end_time=2022-09-15T00%3A00%3A00%2B00%3A00&max_results=100

For my use case, this is very unfortunate because the error is not handled gracefully and the collection of timelines from the list of users crashes and has to be started again, excluding the pathological user ID. Maybe the timelines command could profit from handling this exception explicitely?

[SamHames]

Hi there, thanks for the report.

So Twarc already does handle 503 errors - 503 errors indicate Twitter server issues that are (usually) unrelated to the specific request, and you'll only see them raised by twarc when the Twitter API is having persistent issues (the current defaults are 30 retries, with increasing wait time in between each).

I don't think this is something we can handle in the timelines or similar commands - we can't can't just skip to the next user on a 503 because we're likely to just get the same response, and we'll also have a result file that doesn't represent the input - the user account in question is now retrievable for example.

I do agree with your point that it's very inconvenient when this happens though - we've had issue #589 open for a while which would allow for resuming an interrupted collection, but it's a bit fiddly and no one has had time to tackle it.

[JanaLasser]

Thank you for the reply - these arguments make a lot of sense to me. Maybe a solution could be to skip a given user ID but print out a warning in stdout and the log?

Anyways, I am going to close this, since the drawbacks of handling this issue are probably greater than the benefits.