🐛 Bug Report: Failed to parse page JSON data: expected value at line 1 column 1 (11/18)

[sigaloid]

This issue has been fixed and a new image has been released. Pull and update - do not comment here saying it's not fixed unless your error page says "✅ Instance is up to date."

Update: this situation is rapidly changing, please check bottom of thread.

Update 2: Been a day or so since it's been broken, please pull a new image!

[sigaloid]

@grngxd You mentioned some instances are up and working, which ones?

[tiphae]

My Redlib is giving this error, but so is my backup Libreddit that usually works when Redlib doesn't. I am able to view Reddit directly just fine (from same IP).

[grngxd]

@grngxd You mentioned some instances are up and working, which ones?

https://libreddit.privacydev.net/ was working when i tested it, its dead now. @sigaloid

[sigaloid]

Ah, possibly still being cached.

[Coccocoahelper]

From my 1-minute testing, no instance seems to be functional. Still going to check more, though.

[gigirassy]

Thought I was IP banned, but looks like this applies to all Redlib instances.
Here's my WebDock KVM VPS' Redlib instance (basic auth), displaying this error:

And here's a test container on my regular main Windows PC:

VPS logs:

Windows PC Docker Desktop logs:

[Coccocoahelper]

Thought I was IP banned, but looks like this applies to all Redlib instances. Here's my WebDock KVM VPS' Redlib instance (basic auth), displaying this error: And here's a test container on my regular main Windows PC: VPS logs: Windows PC Docker Desktop logs:

Same thing here. But another frontend, Eddrit (repo) does work, for some reason.

[argium]

Is the process for debugging this documented? I'd like to dump the underlying request/response.

[e455a81e-d3ba-41a2-bc6d-7aafb1d9a5cd]

I just noticed this with my private instance. Interestingly enough, queries to the json endpoints succeed with curl but fail in redlib.

[sigaloid]

Unfortunately there's not much in terms of debugging. There was a PR up a few weeks back to disable TLS verification which would be the first step but we would also need to add proxy support. We're on an old version of hyper so it might be a pain to do that.

It's been on my back burner for a while because it's trivial to issue the two exact requests Redlib sends, and if it works, it's definitely a TLS signature error.

good news is that I'm fairly certain that the issue is another TLS fingerprinting issue because issuing the same requests independent of our TLS stack works just fine. So I might be forced to upgrade that anyway in the course of fixing it. Bad news is, of course, that this means it's officially a cat and mouse game between me and some Reddit SWE. That just hurts - other frontends like previously mentioned aren't being targeted like Redlib is! I guess it's because we're the most popular. :(

Anyway. Working on finding a fix now

[maxexcloo]

This has to be one of the best supported selfhosted apps I use - I always wake up, notice Reddit isn't working, have a coffee and notice it's fixed!

Always appreciate the hard work @sigaloid ☕

[Bumpf]

https://reddit.privacy.com.de/ is throwing this error on the VPN IP I am usually on, and also on my home IP! I realized you said to check a second IP after I had already filled out the info but maybe this is useful anyway (it's for the VPN).

1. Confirmed
2. All requests seem to return Failed to parse yadda yadda [I checked the main page, a subreddit, and a specific post. 3 URLs means I can't access any page, right? ;) ]
3. IPv4
4. AS No. 396253, IP:149.19.41.176

EDIT:
preview links are throwing a 403 e.g. https://reddit.privacy.com.de/preview/pre/6j2r57gbsf0e1.jpeg with "Error 54113
Details: cache-fra-etou8220065-FRA 1731975459 1877502380"
but img links seem to be working fine e.g. https://reddit.privacy.com.de/img/em5uzn5isyod1.jpeg

[argium]

@sigaloid I added some instrumentation and ran it locally.

Are you working by trial and error? Do you run the real reddit app and then compare?

PS. Have we considered filing a support ticket? 😂

[sigaloid]

I test with the two basic requests that tell me where in the stack in the error is; basically just token request then /r/popular with the token auth (matching the same headers and all). If that works in python/curl, I know the issue is within the TLS stack.

That network policy error is pretty common, back when the API was easier to work with it happened when you had certain messages in your user agent. Nowadays it means something in your request is making the edge CDN's and WAF's concerned.

[APT37]

Same thing here. But another frontend, Eddrit (repo) does work, for some reason.

Shows Error 500, so it's probably also blocked.

That just hurts - other frontends like previously mentioned aren't being targeted like Redlib is! I guess it's because we're the most popular. :(

They might just have started with the most popular project.

[argium]

I test with the two basic requests that tell me where in the stack in the error is; basically just token request then /r/popular with the token auth (matching the same headers and all). If that works in python/curl, I know the issue is within the TLS stack.

Is that what's happening today too?

FWIW, they could just be targeting third party use generally. I'm sure they have a lot of bots to contend with that cause them significantly worse headaches.

They may have also pushed out a new app apk a couple days ago and I wonder whether they're making corresponding changes there too.

[lvxnull2]

Even the pre-#305 commits of redlib are broken, they worked previously after frequent restarts

[sigaloid]

Shows Error 500, so it's probably also blocked.

Ah, phew, glad it's not just redlib 😁

Is that what's happening today too?

Yes. I can recreate the requests without failures on another TLS stack in Rust as well. So I know it's a hyper_rustls issue.

They may have also pushed out a new app apk a couple days ago and I wonder whether they're making corresponding changes there too.

I don't know if they'd break new apps that quickly. I hope not, as that'll really increase my maintenance workload.

Anyway, good news all, I've confirmed a fix in a96bebb by switching to hyper-tls which uses native-tls which uses a vendored openssl-dev. A few downsides: not a rustic TLS stack, so it makes cross-compiling and static-compiling a bit harder and less predictable. Also it's slightly larger binary-wise. But I can confirm the latest docker image at quay.io/redlib/redlib:latest does work :)

EDIT: okay, had the fix wrong, lol, header fingerprinting was identified as the culprit

[donslice]

Just pulled and restarted. Can confirm working on my self-hosted instance. Thanks for being so responsive!

[geraldwuhoo]

Thanks for your hard work on this. Unfortunately, it looks like the cross-compilation is broken: https://github.com/redlib-org/redlib/actions/runs/11917620723/job/33213127030. Realized when I pulled the latest on my ARM server and it was still out-of-date. Any chance for a fix?

[halictuz]

Shows Error 500, so it's probably also blocked.

Ah, phew, glad it's not just redlib 😁

Is that what's happening today too?

Yes. I can recreate the requests without failures on another TLS stack in Rust as well. So I know it's a hyper_rustls issue.

They may have also pushed out a new app apk a couple days ago and I wonder whether they're making corresponding changes there too.

I don't know if they'd break new apps that quickly. I hope not, as that'll really increase my maintenance workload.

Anyway, good news all, I've confirmed a fix in a96bebb by switching to hyper-tls which uses native-tls which uses a vendored openssl-dev. A few downsides: not a rustic TLS stack, so it makes cross-compiling and static-compiling a bit harder and less predictable. Also it's slightly larger binary-wise. But I can confirm the latest docker image at quay.io/redlib/redlib:latest does work :)

I pulled from there 5 minutes ago. And it sill says:

⚠️ This instance is not up to date and is 5 commits old. Test and confirm on an up-to-date instance before reporting.

I do it from my Pi4 at home with my public home ip. thats different from my public server ip where the container doesnt even start. (both arm)

[sigaloid]

Yep, arm build is unfortunately broken due to the switch. I'll be looking at this when I get a chance. #326

[3ster]

I am still getting this error on the latest version (sha-96e40e888742):

Failed to parse page JSON data: expected value at line 1 column 1 | /r/popular/hot.json?&raw_json=1&geo_filter=GLOBAL
[Reddit Status](https://www.redditstatus.com/)

✅ Instance is up to date.

Expected something to work? [Report an issue](https://github.com/redlib-org/redlib/issues/new?assignees=&labels=bug&projects=&template=bug_report.md&title=%F0%9F%90%9B+Bug+Report%3A+Failed%20to%20parse%20page%20JSON%20data:%20expected%20value%20at%20line%201%20column%201%20|%20/r/popular/hot.json?&raw_json=1&geo_filter=GLOBAL)

If you're getting a "Failed to parse page JSON data" error, please check [#318](https://github.com/redlib-org/redlib/issues/318)

Head back [home]()?

Instance is running on a residential IP and CURLing reddit / accessing reddit in browser works fine.

[sigaloid]

Can you by chance test on another IP? This sounds like #318

[Kakifrucht]

same issue, tried my local ip, vpn ip and datacenter ip. same error on all, latest commit build

[3ster]

Grabbed a new v4 (and v6 prefix), same issue - don't think this is IP-related, unless reddit decided to ban my entire ISP.

[0x62c734ec]

curl/curl#9362 check this.

[Patriot2407]

I have a one person instance so I dont think its a ban tbh

⚠️ This instance is not up to date and is 12 commits old. Test and confirm on an up-to-date instance before reporting.

I'm just a normie and new around here, but my non-educated guess is that this is because the docker image may be a few commits behind the latest code commit. The docker image is probably not being rebuilt and published every time there is a new commit (reasonably so). Feel free to correct me if I'm off my rocker.

[NTchrist]

Showerthought:

what if we changed the license to something that prohibited companies with more than $100m in revenue from reading any code from the repository without paying $10m for a license, change the fingerprinting again, know that they are violating the license by reading your commits, sue them for said $10m dollars, and during discovery get them to divulge how their fingerprinting works?

( I don't have anything actually constructive to add, so hopefully I at least made someone laugh. We're rooting for you @sigaloid ! )

[sigaloid]

The docker image is probably not being rebuilt and published every time there is a new commit (reasonably so). Feel free to correct me if I'm off my rocker.

It's actually built on every single commit, so if you run into any errors, the first thing you should do is pull a new version (that's why I added that message). Currently the latest version does work, so definitely pull it. And that includes you ARM folk :D

[binamov]

Y'all get this man some caffeine ☕

[donslice]

I dropped him a few bucks here a few hours ago. His dedication is amazing!

[FireMasterK]

good news is that I'm fairly certain that the issue is another TLS fingerprinting issue because issuing the same requests independent of our TLS stack works just fine. So I might be forced to upgrade that anyway in the course of fixing it. Bad news is, of course, that this means it's officially a cat and mouse game between me and some Reddit SWE. That just hurts - other frontends like previously mentioned aren't being targeted like Redlib is! I guess it's because we're the most popular. :(

I think it's most likely a TLS fingerprinting issue too, we should replicate the TLS fingerprint from the Android client. We should copy the TLS extensions using something like Wireshark, and try to replicate it. :)

[WreckingBANG]

Y'all get this man some caffeine ☕

Do it! I did my part! o7

[NTchrist]

Y'all get this man some caffeine ☕

Y'all get this man some caffeine ☕

Do it! I did my part! o7

[alangrafu]

Thanks for all your effort!

[maxexcloo]

Appreciate the work again! Some sleuthing indeed 🕵🏻

[sigaloid]

Thanks for all of the kind words and donations. Very motivational during these times of high pressure :)

[maxexcloo]

Issue seems to be back again on my instance unfortunately...

[sigaloid]

Unable to reproduce, perhaps give it a few minutes?

[maxexcloo]

Unable to reproduce, perhaps give it a few minutes?

Still unavailable for me, it was working fine till I posted the message - is there any way I can debug?

[sigaloid]

Try from another IP - I'm not getting it residentially nor commercially. It may be a widespread issue just taking longer to reach the edge CDN servers I'm nearest. But more than likely #318

[sooox-cc]

its working fine for me after pulling the latest docker image

[maxexcloo]

Try from another IP - I'm not getting it residentially nor commercially. It may be a widespread issue just taking longer to reach the edge CDN servers I'm nearest. But more than likely #318

Yup it was an IP ban, thanks for the tip - have posted a workaround with Cloudflare WARP inside docker compose here: #318 (comment)

[dave0003]

For what it's worth, I tried the latest build: 6be6f89 11/21/2024
And immediately experienced this problem again. I reverted back to: d3ba5f3 11/19/2024
And this slightly older version is still working.

I'm running the binary: redlib-x86_64-unknown-linux-musl

[sigaloid]

That's really weird, nothing in those commits should have changed anything (d3ba5f3...6be6f89).

Removal of X-Reddit-Device-Id perhaps, but that no longer is sent from my tests...

I can't reproduce. I'd say it's #318 but if it's working on an older version...

Is this reproducible consistently? There's some things that are set non-deterministically in the spoofer.

[dave0003]

It's consistent. I just downloaded the latest build again, and immediately encountered the error. Reverted to the older version again and it's fine.

It's a private instance, too. I'm the only one using it.

[sigaloid]

I wouldn't recommend downloading builds, the release system is a little weird and the releases get overwritten by new commit builds (but not the one labeled redlib in the release). Been meaning to work on that for a while.

I would recommend cloning the code and checking out the commits in question (right now would be good to test main), with RUST_LOG=redlib=trace cargo run. That, or using the container images over at https://quay.io/redlib/redlib

[APT37]

Works fine for me on the latest commit, on older versions I'm usually blocked almost immediately. (Private instance with a data center IP.)

[Handrail9]

I'm having this issue on the latest pull. I run the docker container and it looks like my residential IP address might be blocked? I'm being told to sign in if I visit regular Reddit. Let me know if theres any more info I can provide or if Ive done Something wrong

[sigaloid]

Cannot reproduce - unfortunately if your IP is flat-out banned there's not much you can do without some kind of proxying solution. Perhaps try this in a docker-compose?

#318 (comment)

[AyoungDukie]

Encountering this, and it seems like a new push of banning again, as many of the public instances are also going down:

https://stats.uptimerobot.com/mpmqAs1G2Q

[satonotdead]

I'm the only one getting ⚠️ Error checking update status.?

Already updated to last commit and restarted the intance.