Can not log in when importing

[jonathan-morton]

Describe the bug
I set up a brand new account and added all my credentials. I then tried to add my trakt page, and it seemed to be importing, but then appeared to get stuck on adding a star rating from a movie. I hit the X and it said it would continue. I was able to go to a few more pages and see some newly added watches but then it logged me out. I logged back in, was able to browse a couple more pages and then was logged out. Then I kept getting not authorized errors or user doesn't exist, with an occasional log in letting me into a loading page and being immediately logged out again.

Looking at the docker logs it appears to still be importing, so I'm going to let it run over night, even though its already been over an hour running

watcharr       | 2024/10/28 00:25:49 /server/watched_episodes.go:91 SLOW SQL >= 200ms
watcharr       | [513.537ms] [rows:108] INSERT INTO `watched_episodes` (`created_at`,`updated_at`,`deleted_at`,`user_id`,`watched_id`,`season_number`,`episode_number`,`status`,`rating`,`id`) VALUES ("2024-10-28 00:22:42.459","2024-10-28 00:25:48.653"...

Here is the error when I tried to log in

2024/10/28 00:18:57 [Recovery] 2024/10/28 - 00:18:57 panic recovered:
watcharr       | database is locked
watcharr       | /server/watched.go:91 (0x9771a4)
watcharr       | /server/routes.go:448 (0x956f74)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0x87f30a)
watcharr       | /server/auth.go:246 (0x95763c)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0x8853f9)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0x8853e7)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0x884524)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:249 (0x88450b)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0x883911)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:633 (0x883380)
watcharr       | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:589 (0x883011)
watcharr       | /usr/local/go/src/net/http/server.go:3142 (0x75f5cd)
watcharr       | /usr/local/go/src/net/http/server.go:2044 (0x75bb47)
watcharr       | /usr/local/go/src/runtime/asm_amd64.s:1695 (0x470580)
watcharr       |
watcharr       |
watcharr       | 2024/10/28 00:18:57 /server/auth.go:234 database is locked
watcharr       | [5004.587ms] [rows:0] SELECT * FROM `users` WHERE id = 1 AND `users`.`deleted_at` IS NULL LIMIT 1
watcharr       | time=2024-10-28T00:18:57.045Z level=ERROR msg="AuthRequired: Failed to select user from database" error="database is locked"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 401 |  5.004823085s |      x.x.x.x | GET      "/api/user"
watcharr       |
watcharr       | 2024/10/28 00:18:57 /server/auth.go:234 database is locked
watcharr       | [5004.622ms] [rows:0] SELECT * FROM `users` WHERE id = 1 AND `users`.`deleted_at` IS NULL LIMIT 1
watcharr       | time=2024-10-28T00:18:57.045Z level=ERROR msg="AuthRequired: Failed to select user from database" error="database is locked"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 401 |  5.004745443s |      x.x.x.x | GET      "/api/user/settings"
watcharr       |
watcharr       | 2024/10/28 00:18:57 /server/auth.go:234 database is locked
watcharr       | [5004.284ms] [rows:0] SELECT * FROM `users` WHERE id = 1 AND `users`.`deleted_at` IS NULL LIMIT 1
watcharr       | time=2024-10-28T00:18:57.046Z level=ERROR msg="AuthRequired: Failed to select user from database" error="database is locked"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 401 |  5.004497229s |      x.x.x.x | GET      "/api/features"
watcharr       |
watcharr       | 2024/10/28 00:18:57 /server/auth.go:234 database is locked
watcharr       | [5004.139ms] [rows:0] SELECT * FROM `users` WHERE id = 1 AND `users`.`deleted_at` IS NULL LIMIT 1
watcharr       | time=2024-10-28T00:18:57.047Z level=ERROR msg="AuthRequired: Failed to select user from database" error="database is locked"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 401 |  5.004256586s |      x.x.x.x | GET      "/api/follow"
watcharr       |
watcharr       | 2024/10/28 00:18:57 /server/tags.go:35 database is locked
watcharr       | [5004.333ms] [rows:0] SELECT * FROM `tags` WHERE user_id = 1 AND `tags`.`deleted_at` IS NULL
watcharr       | time=2024-10-28T00:18:57.059Z level=ERROR msg="getTags: Failed getting tags from database" error="database is locked"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 500 |  5.004555201s |      x.x.x.x | GET      "/api/tag"
watcharr       | [GIN] 2024/10/28 - 00:18:57 | 200 |       29.31µs |      x.x.x.x | GET      "/api/auth/available"

Platform (please complete the following information):

• OS: Linux Docker Compose
• Browser (latest version): Firefox
• Watcharr Version: Latest

[IRHM]

Hi @jonathan-morton, thank you for reporting this issue.

I think this might be a limitation of the sqlite database, the trakt importer just spam adds your entire watch list into the watcharr db.

I'm not entirely sure if it's a limitation of the db engine, but I have heard things about it not being able to handle concurrent transactions. I'll have to look into this, it's possible just adding a short delay between importing chunks of data could allow other requests to go through.

I think you encountered another bug, where the popup doesn't keep updating with whats currently being imported (live job status), but its still importing correctly in the background.

Depending on the size of your trakt history, it might just need that much time to import, it has to gather each page of your history/ratings from trakt, then import each, sadly both tasks only work one by one which is why it can be so slow.

[khakers]

Setting sqlite into WAL mode might help alleviate this issue somewhat since you can get read concurrency.

https://www.sqlite.org/wal.html