Upgrade baseline curl to 8.1.1

[lwthiker]

The new curl version includes bugfixes, new options but mainly websocket support which was long requested in curl-impersonate. This commit attempts to upgrade the baseline curl version to 8.1.1 for the Chrome version.

TODO:

• Fix HTTP2 header and settings handling which was reworked in the upstream curl
• Upgrade the Firefox version as well.
• Test WebSocket handling. (will be postponed to a later PR)

[lwthiker]

This seems to be finally working. Upstream curl reworked their http2 code which made this upgrade way more difficult than I had originally anticipated. Since this is a large patch, I'm going to leave it open for a couple of days and let anyone who might want to test it out before merging.

This also fixes the broken build on mac (#166)

[coderb]

@lwthiker is there any way to access the binaries for linux x64? i'd like to test but i dont have a build environment setup. thanks for your effort, this project is fantastic!

[lwthiker]

@coderb There you go: https://github.com/lwthiker/curl-impersonate/releases/tag/v0.6.0-alpha.1

[coderb]

I still have issue #175 (firefox w/false start breaks) with the build (running up to debian bookworm system, tested on 2 systems).

$ ./curl_ff102 -k -v https://www.bing.com/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 13.107.21.200:443...
* Connected to www.bing.com (13.107.21.200) port 443 (#0)
* Initializing NSS with certpath: none
*  CAfile: none
*  CApath: none
* loaded libnssckbi.so
* ALPN: offers http/1.1,h2
* skipping SSL peer certificate verification
* Trying TLS False Start
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=www.bing.com,O=Microsoft Corporation,L=Redmond,ST=WA,C=US
*  start date: Jul 26 23:57:23 2023 GMT
*  expire date: Jan 22 23:57:23 2024 GMT
*  common name: www.bing.com
*  issuer: CN=Microsoft Azure TLS Issuing CA 05,O=Microsoft Corporation,C=US
* using HTTP/1.x
> GET / HTTP/1.1
> Host: www.bing.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate, br
> Upgrade-Insecure-Requests: 1
> Sec-Fetch-Dest: document
> Sec-Fetch-Mode: navigate
> Sec-Fetch-Site: none
> Sec-Fetch-User: ?1
> TE: Trailers
>
* ALPN: server accepted h2
* Received HTTP/0.9 when not allowed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

[lwthiker]

I can't reproduce it. When I run curl-impersonate on a debian bookworm docker it works fine.

I guess the solution would be to just remove the --false-start flag from the wrapper scripts. I don't think it has any effect over the browser fingerprint anyway. Let's keep it for a separate PR after I merge this.

[coderb]

would it be possible for you to share the docker file or image somehow? i'd like to compare it to mine to track down the issue. thank you.

[coderb]

while doing some additional testing i did a library call using libcurl-impersonate-chrome.so to curl_easy_setopt(h, CURLOPT_SSL_FALSESTART, 1) and it returned the error code CURLE_NOT_BUILT_IN.

the command line scripts for chrome seem to include --false-start but run ok for me so my guess is that the command line handling is either not handling the option properly or ignore the error code from setopt.

running w/o the false start option doesn't seem to cause me any problems, but perhaps for completeness / stealthiness it does seem like there is a problem here if the native browsers actually have this feature enabled in their ssl stacks.

i tested the chrome .so for both the current release version and the alpha you published with the same result.

thanks

[coderb]

ok, i just looked throught the current tip of the source tree for curl and from what i can tell there is very limited support for false start.

https://github.com/search?q=repo%3Acurl%2Fcurl%20false_start&type=code
it looks like maybe it's only supported for macos/ios using sectransp because the other ssl backends seem to be pointing their implementation to Curl_none_false_start.

this would all make sense, but you say it works for you somehow so i am a bit baffled.

[coderb]

oof, should have rtfm...

      --false-start
              (TLS) Tells curl to use false start during  the  TLS  hand‐
              shake.  False start is a mode where a TLS client will start
              sending application data before verifying the server's Fin‐
              ished  message,  thus saving a round trip when performing a
              full handshake.

              This is currently only implemented in the  NSS  and  Secure
              Transport  (on  iOS  7.0  or  later, or OS X 10.9 or later)
              backends.

              Added in 7.42.0.

So chrome shouldn't work....but the convenience scripts include the option.

For some reason my code search didn't pick up the NSS ssl provider code snippet.... will look into it.

[coderb]

i just tried the docker runtime image (latest) and it fails:

$ docker pull lwthiker/curl-impersonate
$ docker run --rm lwthiker/curl-impersonate curl_ff109 https://www.bing.com/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (1) Received HTTP/0.9 when not allowed

[lwthiker]

ok so I managed to reproduce it by doing curl_ff109 -v -k https://13.107.21.200 (i.e. using the exact same IP address as yours). Something in this server makes curl initiate a false start using http/1.1, while the server chooses http/2 based on alpn (as noted in the logs you attached). It happens also with vanilla curl using nss and the --false-start flag, so it's a curl bug:

* Connected to 13.107.21.200 (13.107.21.200) port 443 (#0)
* Initializing NSS with certpath: none
* WARNING: failed to load NSS PEM library libnsspem.so. Using OpenSSL PEM certificates will not work.
*  CAfile: none
*  CApath: none
* loaded libnssckbi.so
* ALPN: offers h2,http/1.1
* skipping SSL peer certificate verification
* Trying TLS False Start
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=www.bing.com,O=Microsoft Corporation,L=Redmond,ST=WA,C=US
*  start date: Jul 26 23:57:23 2023 GMT
*  expire date: Jan 22 23:57:23 2024 GMT
*  common name: www.bing.com
*  issuer: CN=Microsoft Azure TLS Issuing CA 05,O=Microsoft Corporation,C=US
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 13.107.21.200
> User-Agent: curl/8.1.1-DEV
> Accept: */*
> 
* ALPN: server accepted h2
* Received HTTP/0.9 when not allowed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

I think removing this flag altogether would be best for now, especially given that upstream curl does not support nss anymore.