Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export All Hangs on Error: Internal Server Error 500 #272

Open
moonmd opened this issue Dec 3, 2024 · 3 comments
Open

Export All Hangs on Error: Internal Server Error 500 #272

moonmd opened this issue Dec 3, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@moonmd
Copy link

moonmd commented Dec 3, 2024

GET https://chatgpt.com/backend-api/conversations?offset=1140&limit=20 500 (Internal Server Error)
chatgpt.com-1733192011542.log
chatgpt.com-1733191997581.log

Steps to reproduce:

Details

Microsoft Edge
Version 131.0.2903.70 (Official build) (64-bit)
uname -srvm
Linux 6.8.0-49-generic #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov  6 17:42:15 UTC 2 x86_64
Tampermonkey
Size: 5.1 MB Version 5.3.2
@moonmd
Copy link
Author

moonmd commented Dec 3, 2024

Fetch Timing

Name                           Status  Type   Initiator                 Size     Time
conversations?offset=0limit=20    200  fetch  userscript.html?name=...  1.8 kB   964 ms    
rgstr                             202  fetch  lrt789z2rawt9v6n.js:214    630 B    70 ms    
conversations?offset=20limit=20   200  fetch  userscript.html?name=...  1.8 kB   513 ms    
conversations?offset=40limit=20   200  fetch  userscript.html?name=...  1.8 kB   974 ms    
conversations?offset=60limit=20   200  fetch  userscript.html?name=...  1.8 kB   479 ms    
conversations?offset=80limit=20   200  fetch  userscript.html?name=...  2.0 kB   776 ms    
conversations?offset=100limit=20  200  fetch  userscript.html?name=...  1.8 kB   7.07 s    
conversations?offset=120limit=20  200  fetch  userscript.html?name=...  1.8 kB   615 ms    
conversations?offset=140limit=20  200  fetch  userscript.html?name=...  1.8 kB   756 ms    
conversations?offset=160limit=20  200  fetch  userscript.html?name=...  1.8 kB   826 ms    
conversations?offset=180limit=20  200  fetch  userscript.html?name=...  1.8 kB   800 ms    
conversations?offset=200limit=20  200  fetch  userscript.html?name=...  1.7 kB   949 ms    
conversations?offset=220limit=20  200  fetch  userscript.html?name=...  1.9 kB   1.34 s    
conversations?offset=240limit=20  200  fetch  userscript.html?name=...  2.2 kB   888 ms    
conversations?offset=260limit=20  200  fetch  userscript.html?name=...  1.9 kB   1.14 s    
conversations?offset=280limit=20  200  fetch  userscript.html?name=...  1.9 kB   1.97 s    
conversations?offset=300limit=20  200  fetch  userscript.html?name=...  1.9 kB   1.71 s    
conversations?offset=320limit=20  200  fetch  userscript.html?name=...  1.9 kB   1.81 s    
conversations?offset=340limit=20  200  fetch  userscript.html?name=...  1.8 kB   2.64 s    
conversations?offset=360limit=20  200  fetch  userscript.html?name=...  1.9 kB   4.34 s    
conversations?offset=380limit=20  200  fetch  userscript.html?name=...  1.8 kB   3.72 s    
conversations?offset=400limit=20  200  fetch  userscript.html?name=...  2.2 kB   3.93 s    
conversations?offset=420limit=20  200  fetch  userscript.html?name=...  1.9 kB   3.26 s    
conversations?offset=440limit=20  200  fetch  userscript.html?name=...  1.9 kB   4.33 s    
conversations?offset=460limit=20  200  fetch  userscript.html?name=...  1.9 kB   4.13 s    
conversations?offset=480limit=20  200  fetch  userscript.html?name=...  2.0 kB   5.80 s    
conversations?offset=500limit=20  200  fetch  userscript.html?name=...  1.9 kB   6.83 s    
conversations?offset=520limit=20  200  fetch  userscript.html?name=...  1.8 kB   5.48 s    
conversations?offset=540limit=20  200  fetch  userscript.html?name=...  1.9 kB   7.11 s    
conversations?offset=560limit=20  200  fetch  userscript.html?name=...  2.1 kB   7.49 s    
conversations?offset=580limit=20  200  fetch  userscript.html?name=...  1.9 kB   9.43 s    
conversations?offset=600limit=20  200  fetch  userscript.html?name=...  2.0 kB  11.43 s    
conversations?offset=620limit=20  200  fetch  userscript.html?name=...  1.9 kB  11.44 s    
conversations?offset=640limit=20  200  fetch  userscript.html?name=...  1.9 kB  11.00 s    
conversations?offset=660limit=20  200  fetch  userscript.html?name=...  1.9 kB  16.09 s    
conversations?offset=680limit=20  200  fetch  userscript.html?name=...  1.9 kB  13.74 s    
conversations?offset=700limit=20  200  fetch  userscript.html?name=...  1.9 kB  15.20 s    
conversations?offset=720limit=20  200  fetch  userscript.html?name=...  2.2 kB  15.30 s    
conversations?offset=740limit=20  200  fetch  userscript.html?name=...  1.8 kB  13.91 s    
conversations?offset=760limit=20  200  fetch  userscript.html?name=...  1.8 kB  15.70 s    
conversations?offset=780limit=20  200  fetch  userscript.html?name=...  2.1 kB  17.84 s    
conversations?offset=800limit=20  200  fetch  userscript.html?name=...  1.9 kB  17.17 s    
conversations?offset=820limit=20  200  fetch  userscript.html?name=...  1.9 kB  16.90 s    
conversations?offset=840limit=20  200  fetch  userscript.html?name=...  1.9 kB  16.83 s    
conversations?offset=860limit=20  200  fetch  userscript.html?name=...  2.1 kB  19.47 s    
conversations?offset=880limit=20  200  fetch  userscript.html?name=...  1.9 kB  17.77 s    
conversations?offset=900limit=20  200  fetch  userscript.html?name=...  1.9 kB  17.73 s    
conversations?offset=920limit=20  200  fetch  userscript.html?name=...  1.8 kB  19.21 s    
conversations?offset=940limit=20  200  fetch  userscript.html?name=...  1.8 kB  22.43 s    
conversations?offset=960limit=20  200  fetch  userscript.html?name=...  1.9 kB  23.59 s    
conversations?offset=980limit=20  200  fetch  userscript.html?name=...  1.8 kB  21.32 s    
conversations?offset=1000limit=20 200  fetch  userscript.html?name=...  1.8 kB  19.83 s    
conversations?offset=1020limit=20 200  fetch  userscript.html?name=...  2.1 kB  21.54 s    
conversations?offset=1040limit=20 200  fetch  userscript.html?name=...  1.9 kB  24.44 s    
conversations?offset=1060limit=20 200  fetch  userscript.html?name=...  1.9 kB  26.06 s    
conversations?offset=1080limit=20 200  fetch  userscript.html?name=...  1.8 kB  26.45 s    
conversations?offset=1100limit=20 200  fetch  userscript.html?name=...  1.9 kB  24.84 s    
conversations?offset=1120limit=20 200  fetch  userscript.html?name=...  1.9 kB  23.73 s    
conversations?offset=1140limit=20 500  fetch  userscript.html?name=...   328 B  30.14 s

@moonmd
Copy link
Author

moonmd commented Dec 3, 2024

Modified to batch size 100

The API max is 100 (101 errors with 422)

Request URL:    https://chatgpt.com/backend-api/conversations?offset=0&limit=101
Request Method: GET
Status Code:    422 Unprocessable Content

FetchAllConversations is failing at the same point, but much faster: 2m40s vs. 10m25s
(This would dramatically speed up Export All.)

Name                             Status  Type   Initiator               Size     Time
conversations?offset=0&limit=100    200  fetch  userscript.html?name=   6.7 kB   1.73 s
conversations?offset=100&limit=100  200  fetch  userscript.html?name=   6.8 kB   2.06 s
conversations?offset=200&limit=100  200  fetch  userscript.html?name=   7.1 kB   2.47 s
rgstr                               202  fetch  lrt789z2rawt9v6n.js:214  908 B    71 ms
conversations?offset=300&limit=100  200  fetch  userscript.html?name=   7.2 kB   4.50 s
conversations?offset=400&limit=100  200  fetch  userscript.html?name=   7.4 kB   5.28 s
conversations?offset=500&limit=100  200  fetch  userscript.html?name=   7.4 kB  10.31 s
conversations?offset=600&limit=100  200  fetch  userscript.html?name=   7.3 kB  13.62 s
conversations?offset=700&limit=100  200  fetch  userscript.html?name=   7.2 kB  22.32 s
conversations?offset=800&limit=100  200  fetch  userscript.html?name=   7.3 kB  20.21 s
conversations?offset=900&limit=100  200  fetch  userscript.html?name=   6.8 kB  23.86 s
conversations?offset=1000&limit=100 200  fetch  userscript.html?name=   7.0 kB  21.75 s
conversations?offset=1100&limit=100 500  fetch  userscript.html?name=    320 B  30.14 s

After a bunch of experimentation, it appears this is an issue with their servers, and I am seeing similar behavior with their conversation interface.

export async function fetchAllConversations(): Promise<ApiConversationItem[]> {
    const conversations: ApiConversationItem[] = []
    const limit = 100
    let offset = 1100
  • GET takes more than 30s at larger offsets and fails:
    • Starting at 1100+100
      • conversations?offset=1100&limit=100 500 fetch -- 319 B 30.24 s
    • Starting at 1200+100
      • conversations?offset=1200&limit=100 500 fetch -- 323 B 30.17 s
    • Starting at 1500+10
      • conversations?offset=1500&limit=10 500 fetch -- 327 B 30.16 s

It might make sense to add an option to set a maximum conversations to fetch, defaulting to 1000.
Or just hardcode it in fetchAllConversations.
The 'Refer' feature in PR #271 allows for the higher offsets to be downloaded and the UI handles this fine. (Tested to 1476)

@pionxzh pionxzh added the enhancement New feature or request label Dec 3, 2024
@pionxzh
Copy link
Owner

pionxzh commented Dec 3, 2024

Wow. Thanks for the detailed testing.

I think it's reasonable to set the limit to 1000 and adjust the batch size to 100.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants