httr2::request() does not retrieve data

[toobiwankenobi]

I have the same issue with both httr and httr2. I described the issue here.
Thank you!

[hadley]

Can you please include the httr2 code you tried directly in this issue?

[toobiwankenobi]

Hi Hadley @hadley

Sure, here is an example. If I open the query URL in my browser, I can see that the body is not empty.

> httr2::request("https://rest.uniprot.org/uniprotkb/search?query=organism_id%3A9606%20OR%20gene_exact%3AA2M&fields=id%2Caccession%2Cgene_primary%2Corganism_name%2Cprotein_name%2Creviewed&format=tsv&size=25")
<httr2_request>
GET
https://rest.uniprot.org/uniprotkb/search?query=organism_id%3A9606%20OR%20gene_exact%3AA2M&fields=id%2Caccession%2Cgene_primary%2Corganism_name%2Cprotein_name%2Creviewed&format=tsv&size=25
Body: empty

[hadley]

I suspect the problem is something to do with escaping the query params since this works for me:

request("https://rest.uniprot.org/uniprotkb/search") |> 
  req_url_query(
    query = "organism_id:9606 OR gene_exact:A2M",
    fields = "id,accession,gene_primary,organism_name,protein_name,reviewed",
    format = "tsv",
    size = "25",
  ) |> 
  req_perform()

[jonthegeek]

@toobiwankenobi It looks like you only set up the request. The Body: empty in the print is because the body of the request is empty. You need to req_perform() it to retrieve results. The "Get started" article might help.

[LiNk-NY]

It seems like the gzip content isn't automatically decoded / decompressed on Mac. The same code works fine on Linux and Windows.

suppressPackageStartupMessages(library(httr2))
request("https://rest.uniprot.org/uniprotkb/search") |> 
    req_url_query(
        query = "organism_id:9606 OR gene_exact:A2M",
        fields = "id,accession,gene_primary,organism_name,protein_name,reviewed",
        format = "tsv",
        size = "25",
    ) |> 
    req_perform() |>
    resp_body_raw() |>
    rawToChar()
#> Error in rawToChar(resp_body_raw(req_perform(req_url_query(request("https://rest.uniprot.org/uniprotkb/search"), : embedded nul in string: '\037\x8b\b\0\0\0\0\0\0\xff\x8dVKs\xa38\020>k\177\x85nc\037\xe4\xb2\xc0\xc6\xc9\xee\t\xbf\xb3\023l\xc6v&\xc9lmM\xc9 \xdb\xda\0\xa2\004$!\xbf~[\xd8\006\x92\xc1\xa9\xb9$\xb8\xd5\xdd\xfa\xba\xf5\xf5c\022\xa5*\xc7\v\026r4џh\xc6#^\xfcNp+V"d*o\xa3\xa5ڳH$!r\x95L\xb9\x88p\xa4\025Њ?\v\xfe\xc2\xfd?lÙ\xfd\x9c\xdf9\xf6\002\xb9]\xda5L\004\0224\x97\xa1\xc4\t\x8b\005\x8f\xc0\xd9<\vY\xd4Fv\020\037\0301H\xc8<%\xf7\x81\xdcf\001\xf8k\x9d\xc5N\033\xb7F&f\x91\x8f\xdd\037.\t\xc4\023Ǭ\xd1ė!\023\021\xf1d\x94\xc2\177\021\xedq|\002\xd7o#\xf5\016\xd9\xca\xde\0.\xcb0\xaf\0W\x88V,M\xb3\004GR=\xf3\xbd\xf0\xe0\xb3\005\xa2߀V\xb9\005l?\x9d\xe5\xddz\x82\xbeY\xf4\n\xfc\xbao1r\xc0S\x98%^\026h\x97\x8e\xcc\022ކ\x8c\xf1}\xc4"/\xc7o\0222{\xc6\xd8j\xbc\xac\xfd.\021\xff\x8c\002Ξ\xb9\x8fE\x94\xca?q3<j\xf5\xf1Ә\xe1$\xdbf\x91H\xff\xba\xa0f\xbe\xd3\xfa\xf7C\x82Ρ̾mh\x91\xa2\xc6P\032=\023\xf7R,\xd5\035\xcer\xb3\036\x9d\xf8aw\xed\xee\xa8?\xec\xcf,\xe4l\xc8j\xb1\xa2\xcd<qD*\xbd\x83\x8c|%X@|\xae\x84

request("https://rest.uniprot.org/uniprotkb/search") |> 
    req_url_query(
        query = "organism_id:9606 OR gene_exact:A2M",
        fields = "id,accession,gene_primary,organism_name,protein_name,reviewed",
        format = "tsv",
        size = "25",
    ) |> 
    req_perform() |>
    resp_header("content-encoding")
#> [1] "gzip"

^{Created on 2025-01-07 with reprex v2.1.1}

The same happens with httr:

suppressPackageStartupMessages(library(httr))
base_url <- "https://rest.uniprot.org/uniprotkb/search"
query_params <- list(
    query = "organism_id:9606 OR gene_exact:A2M",
    fields = "id,accession,gene_primary,organism_name,protein_name,reviewed",
    format = "tsv",
    size = 25
)
response <- GET(url = base_url, query = query_params) 
content(response, "text")
#> No encoding supplied: defaulting to UTF-8.
#> [1] NA
headers(response)$`content-encoding`
#> [1] "gzip"

^{Created on 2025-01-07 with reprex v2.1.1}

session_info()

> sessioninfo::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.2 (2024-10-31)
 os       macOS Monterey 12.7.6
 system   x86_64, darwin20
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2025-01-07
 rstudio  2024.12.0+467 Kousa Dogwood (desktop)
 pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/x86_64/ (via rmarkdown)

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version  date (UTC) lib source
 BiocManager   1.30.25  2024-08-28 [1] CRAN (R 4.4.1)
 callr         3.7.6    2024-03-25 [1] CRAN (R 4.4.0)
 cli           3.6.3    2024-06-21 [1] CRAN (R 4.4.0)
 clipr         0.8.0    2022-02-22 [1] CRAN (R 4.4.0)
 curl          6.0.1    2024-11-14 [1] CRAN (R 4.4.1)
 digest        0.6.37   2024-08-19 [1] CRAN (R 4.4.1)
 evaluate      1.0.1    2024-10-10 [1] CRAN (R 4.4.1)
 fastmap       1.2.0    2024-05-15 [1] CRAN (R 4.4.0)
 fs            1.6.5    2024-10-30 [1] CRAN (R 4.4.1)
 glue          1.8.0    2024-09-30 [1] CRAN (R 4.4.1)
 htmltools     0.5.8.1  2024-04-04 [1] CRAN (R 4.4.0)
 httpuv        1.6.15   2024-03-26 [1] CRAN (R 4.4.0)
 httr        * 1.4.7    2023-08-15 [1] CRAN (R 4.4.0)
 httr2       * 1.0.7    2024-11-26 [1] CRAN (R 4.4.1)
 jsonlite      1.8.9    2024-09-20 [1] CRAN (R 4.4.1)
 knitr         1.49     2024-11-08 [1] CRAN (R 4.4.1)
 later         1.4.1    2024-11-27 [1] CRAN (R 4.4.1)
 lifecycle     1.0.4    2023-11-07 [1] CRAN (R 4.4.0)
 magrittr      2.0.3    2022-03-30 [1] CRAN (R 4.4.0)
 pillar        1.10.1   2025-01-07 [1] CRAN (R 4.4.2)
 pkgconfig     2.0.3    2019-09-22 [1] CRAN (R 4.4.0)
 processx      3.8.4    2024-03-16 [1] CRAN (R 4.4.0)
 promises      1.3.2    2024-11-28 [1] CRAN (R 4.4.1)
 ps            1.8.1    2024-10-28 [1] CRAN (R 4.4.1)
 R6            2.5.1    2021-08-19 [1] CRAN (R 4.4.0)
 rappdirs      0.3.3    2021-01-31 [1] CRAN (R 4.4.0)
 Rcpp          1.0.13-1 2024-11-02 [1] CRAN (R 4.4.1)
 reprex        2.1.1    2024-07-06 [1] CRAN (R 4.4.0)
 rlang         1.1.4    2024-06-04 [1] CRAN (R 4.4.0)
 rmarkdown     2.29     2024-11-04 [1] CRAN (R 4.4.1)
 rstudioapi    0.17.1   2024-10-22 [1] CRAN (R 4.4.1)
 sessioninfo   1.2.2    2021-12-06 [1] CRAN (R 4.4.0)
 tibble        3.2.1    2023-03-20 [1] CRAN (R 4.4.0)
 vctrs         0.6.5    2023-12-01 [1] CRAN (R 4.4.0)
 withr         3.0.2    2024-10-28 [1] CRAN (R 4.4.1)
 xfun          0.49     2024-10-31 [1] CRAN (R 4.4.1)
 yaml          2.3.10   2024-07-26 [1] CRAN (R 4.4.0)

 [1] /Users/mramos/R/bioc-release
 [2] /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library

[hadley]

It works just fine for me, so I suspect you've installed curl in some non-standard way.