feat: IPIP-402 based backpressure #160

aschmahmann · 2023-07-06T15:54:52Z

Relies on ipfs/boxo#369

7/31:

The error rates seem to have shifted (more 500s, fewer 502s, fewer 4xxs) as a result of issues here (i.e. not returning special error types to be picked up by boxo/gateway). However, the number of errors seem to be about the same
Saturn nodes are not properly implementing IPIP-402 byte ranges which makes this PR not usable with Saturn until that's resolved.

Other outstanding (non functionally mandatory) items:

Attempt to reduce code complexity by merging the selector-like pieces together
Add/decide on an error type for the gateway code to indicate that a given path traversal is impossible to satisfy

Note: This change makes the testing very heavily rely on code paths other than kubo sharness tests between

Not using the same gateway backend implementation
Leaning almost entirely on go-unixfsnode and go-ipld-prime rather than go-unixfs and go-ipld-format.

There have been bugs discovered in each, so being eagle eyed and reporting bugs is appreciated.

BigLep · 2023-07-11T16:49:43Z

2023-07-11 conversation:

Focused on TAR support today
Then moving to reducing code complexity or testing
For testing, can use feat: add inverse and version to filter cmd ipld/go-car#457

lidel · 2023-07-25T16:34:46Z

lib/graph_gateway.go

+	escapedPath := url.PathEscape(path.String())
+	escapedPath = strings.ReplaceAll(escapedPath, "%2F", "/")
+	paramsStr := paramsToString(params)
+	urlWithoutHost := fmt.Sprintf("%s?%s", escapedPath, paramsStr)


In the beginning we only had string content paths, and passed that to fetcher.
Now we have URL query params and we append them blindly to content path.

I am afraid mixing URL paths and query params with IPFS content paths will lead to ugly code, special handling like the one above, and even more bugs down the road.

We should clean this up in this PR.

Current state (iiuc)

I think the underlying source of problems is that we have a content path concatenated with URL query params – this is invalid, mixing abstractions, and will lead to more bugs, such as unnecessary escaping of entity-bytes values.

My understanding of "right thing" is the edge case where content path includes characters which look like percent-encoding, and we want to preserve that by double encoding:

In Kubo 0.21 this works:
http://localhost:8080/ipns/en.wikipedia-on-ipfs.org/I/Auditorio_de_Tenerife%252C_Santa_Cruz_de_Tenerife%252C_Espa%C3%B1a%252C_2012-12-15%252C_DD_02.jpg.webp?format=car&dag-scope=entity&entity-bytes=0:* → returns HTTP 301 → http://en.wikipedia-on-ipfs.org.ipns.localhost:8080/I/Auditorio_de_Tenerife%252C_Santa_Cruz_de_Tenerife%252C_Espa%C3%B1a%252C_2012-12-15%252C_DD_02.jpg.webp?format=car&dag-scope=entity&entity-bytes=0:*

We want the same to work in biforst-gateway.

Proposed fix

We need to separate content path from query params, it cna be done in two ways:

(A) pass ImmutablePath + params as url.Values (requires fetcher to construct URL correctly)

(B) create a detached (only path + query, no host) url.URL and pass that to fetcher (imo cleaner and less error prone – we control the source of truth, fetcher won't mangle it as it already has valid url.URL to execute, will only add hostname and protocol)

Rationale: ul.URL already takes care of escaping, we should reuse it as abstraction if possible

JS has dedicated functions for escaping full URI vs URI component:
encodeURIComponent('/ą/ę') → %2F%C4%85%2F%C4%99
encodeURI('/ą/ę') → /%C4%85/%C4%99

in GO we have url.PathEscape and url.QueryEscape instead of encodeURIComponent
and the url.URL type's String() func itself will act as encodeURI already, so no need for extra escaping.

And I think we can reuse it here, as it takes care of all escaping + a detached path is a valid ("relative") URL:

escapedPath := (&url.URL{Path: "/ą/ę"}).String() // → /%C4%85/%C4%99

@aschmahmann @willscott my preference would be to do proper cleanup and do (B) to remove number of moving pieces pased to fetcher – would it be ok?

I can help with code tomorrow if that helps.

Passing a url object rather than a string here seems like a good idea to me. We should be clear about the contract of what's allowed/not in the URL though.

For example, if it's valid/invalid to put format=raw in the params or headers or if the Fetch function implies CAR. While it's somewhat obviously true now it doesn't have to be that way so the docs should clarify either way.

@aschmahmann I've pushed small refactor in 982535a that does not change the fetcher API (we still pass URL as string), but fixes the way we encode content paths. I also added explicit format=car to make it easier to debug and test.

I don't think we should block this PR on the fetcher API cleanup unless we really need to include it, and it does not seem to me like we need to solve the problem at hand: it seems to work as expected already.
Would it be ok to fill issue to do it as follow-up work?

@lidel yeah, I don't think we need to block on a refactor unless it's hindering functionality in some way. If this is enough for now and the refactor is cleanup + reducing headaches here then can definitely be a follow up.

Updating the top-level post with the areas currently under evaluation/investigation before merging.

this implements part of #160 (comment) that fixes the way we encode content paths, without changing the interface of fetcher in caboose.

aschmahmann · 2023-07-31T19:24:04Z

blockstore_proxy.go

@@ -43,14 +43,20 @@ func (ps *proxyBlockStore) Fetch(ctx context.Context, path string, cb lib.DataCa
 		return err
 	}
 	goLog.Debugw("car fetch", "url", req.URL)
-	req.Header.Set("Accept", "application/vnd.ipld.car")
+	req.Header.Set("Accept", "application/vnd.ipld.car;order=dfs;dups=y")


Note: this resolves #177 for non-Saturn backends. Caboose/Saturn will have to change when they're ready.

BigLep · 2023-08-01T17:07:32Z

2023-08-01 conversation:

Impaired because Saturn doesn't support range requests
Should be able to test this against Kubo 0.22 (since it has 402 and 412 support)
We aren't falling back to block requests as often

lidel · 2023-07-27T15:31:12Z

.github/workflows/gateway-conformance.yml

TODO: check if we have tests for when CAR backend returns HTTP 502 or 504.
We want to ensure pass-through works and the same code is returned to end user.

.github/workflows/gateway-conformance.yml

lidel · 2023-08-02T21:19:53Z

lib/gateway_traversal.go

+		case data.Data_Raw, data.Data_Metadata:
+			// TODO: for now, we decided to return nil here. The different implementations are inconsistent
+			// and UnixFS is not properly specified: https://github.com/ipfs/specs/issues/316.
+			// 		- Is Data_Raw different from Data_File?
+			//		- Data_Metadata is handled differently in boxo/ipld/unixfs and go-unixfsnode.
+			return nil


💭 @aschmahmann do we know what was the old behavior in Kubo?
Unsure how popular these are, but if we don't have spec, we may want to derisk and not change behavior that has been in production for years.

We sort of ended up giving this a 🤷 anyhow here ipfs/boxo#303 (comment). We can revisit both, but don't think that needs to happen in this PR.

Co-authored-by: Will Scott <[email protected]>

…nto feat/backpressure

lib/graph_gateway.go

…rectories for dir_index.html rendering

lidel

@aschmahmann I've enabled gateway-conformance tests with GRAPH_BACKEND=true on this PR and main branch (merged both #189 and #190). In the past we've only run conformance against the block-by-block backend, because the gw code paths were the same. With Rhea changes, we've diverged so much we need to run tests twice now.

Main branch is green, but around 200 checks are failing in this PR:

https://github.com/ipfs/bifrost-gateway/actions/runs/5859733390/job/15886335539?pr=160#step:10:700

The number looks big, but it is most likely one or two bugs which impact a lot of tests.

Some quick thoughts (I did not dig too deep, just shallow pass over the log linked above):

seem to be mostly related to 404 / "not found" scenarios and related error handling:
- sometimes returned code is 5XX instead of expected 404
- index.html is not found in a few cases (no link named \"index.html\" under bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i)
- some _redirects file tests are failing because HTTP 502 is returned instead of expected 200|302 etc

aschmahmann · 2023-08-14T23:59:25Z

@lidel thanks for noticing the discrepancies with the how the conformance tests were run 🙏 and getting that fixed up. Will poke around a little more to get to the bottom, but I suspect some of the failures are coming from kubo v0.22.0 (and any released version of boxo) not returning CAR blocks when the path doesn't exist and returning an error instead which .... is not great for the use case of tooling like bifrost-gateway but is being fixed in the version of boxo this PR relies on here.

Alright to swap out the action that pulls a released kubo for custom building a branch?

…hat happens to be for a raw block

…a new dag-order

… CAR fetches for duplicates

This is to handle a bug where if there is a CAR request for a non-existent path an error is returned rather than a CAR that proves that the requested path cannot exist.

lidel mentioned this pull request Jul 6, 2023

meta: GRAPH_BACKEND fixes and latency improvements #88

Closed

aschmahmann force-pushed the feat/backpressure branch from e56df8a to db60039 Compare July 6, 2023 18:38

BigLep assigned aschmahmann Jul 10, 2023

aschmahmann force-pushed the feat/backpressure branch from 5265703 to 4664ac7 Compare July 24, 2023 21:47

aschmahmann marked this pull request as ready for review July 25, 2023 00:45

aschmahmann mentioned this pull request Jul 25, 2023

test backpressure in staging #176

Merged

lidel suggested changes Jul 25, 2023

View reviewed changes

lidel mentioned this pull request Jul 26, 2023

Add test for content paths that have percent-encoded parts ipfs/gateway-conformance#115

Closed

lidel added a commit that referenced this pull request Jul 26, 2023

refactor: add contentPathToCarUrl and tests

982535a

this implements part of #160 (comment) that fixes the way we encode content paths, without changing the interface of fetcher in caboose.

aschmahmann force-pushed the feat/backpressure branch from b0fca23 to 59887ff Compare July 27, 2023 07:50

aschmahmann mentioned this pull request Jul 31, 2023

Use IPIP-412 parameters to request ordered CARs with duplicates rather than defaults #177

Open

aschmahmann commented Jul 31, 2023

View reviewed changes

lidel suggested changes Aug 2, 2023

View reviewed changes

BigLep mentioned this pull request Aug 3, 2023

Release 0.22 ipfs/kubo#9911

Closed

hacdias force-pushed the feat/backpressure branch from b0de13d to 03a7448 Compare August 3, 2023 12:44

aschmahmann added 13 commits August 3, 2023 15:00

chore: update boxo

83eec5d

feat: switch GetCAR to be backpressured

ce24a3f

feat: switch ResolvePath, Head and GetBlock to be backpressured

7ec385a

feat: switch Get to be backpressured (not fully implemented)

468796a

switch fetch to do escaping

db4ecfe

fix detection of go-ipld-prime schema errors and path resolution

2458e3e

fix boxo go.mod

8b2cdc7

fix passing range bytes in CAR requests, and reporting entity ranges

56fce88

chore: remove unused code

965e29d

feat: add support for backpressured tar files

c2fb61a

chore: removed unused testing tools

0691af9

feat: add some partial traversal tests and fixes

7f8eb45

fix: HAMT fetching bug for Get requests

60f2d4a

aarshkshah1992 and others added 5 commits August 7, 2023 16:37

fix memory leak

8c217f9

update to latest caboose

06077bc

Send verification error to Caboose (#184)

0bbc0c4

Co-authored-by: Will Scott <[email protected]>

Merge remote-tracking branch 'origin/main' into feat/backpressure

addb2a1

Merge branch 'feat/backpressure' of github.com:ipfs/bifrost-gateway i…

4958149

…nto feat/backpressure

aschmahmann commented Aug 9, 2023

View reviewed changes

lib/graph_gateway.go Outdated Show resolved Hide resolved

aschmahmann added 2 commits August 10, 2023 07:17

bump go-unixfsnode and enable returning the sizes of items in HAMT di…

c9fd369

…rectories for dir_index.html rendering

return gateway errors on CAR fetching failure

e2be930

aschmahmann force-pushed the feat/backpressure branch from 688d15c to e2be930 Compare August 10, 2023 11:24

add verifcid check

d8485e2

This was referenced Aug 14, 2023

test(main): gateway-conformance with GRAPH_BACKEND=true|false #190

Merged

test: backpressure PR conformance with GRAPH_BACKEND=true #189

Merged

test: run conformance with GRAPH_BACKEND=true

1c54c91

lidel suggested changes Aug 14, 2023

View reviewed changes

aschmahmann added 5 commits August 14, 2023 20:26

fix: correctly decode UnixFS symlinks

a978e34

fix: return starting at the correct offset when doing a Get request t…

4c5e7f9

…hat happens to be for a raw block

fix: add identity-blockstore style behavior to carToLinearBlockGetter

ce2ef89

chore: add guard to GetCar so we don't accidentally forget to handle …

1275db5

…a new dag-order

feat: handle IPIP-412 requests with and without duplicates, backed by…

e605576

… CAR fetches for duplicates

aschmahmann force-pushed the feat/backpressure branch from 53b76b3 to 4f6b224 Compare August 15, 2023 06:55

aschmahmann added 3 commits August 15, 2023 03:01

ci: use a kubo development commit for conformance tests

c23e66f

This is to handle a bug where if there is a CAR request for a non-existent path an error is returned rather than a CAR that proves that the requested path cannot exist.

test: fix TestGetHAMTDirectory to not look for //index.html

983ad96

test: fix CAR retry tests to use CarWriter that allows duplicate blocks

004ea10

aschmahmann force-pushed the feat/backpressure branch from 4f6b224 to 004ea10 Compare August 15, 2023 07:35

Merge branch 'main' into feat/backpressure

5325e53

aschmahmann merged commit 9288910 into main Aug 15, 2023
12 checks passed

aschmahmann deleted the feat/backpressure branch August 15, 2023 17:37

This was referenced Aug 15, 2023

traverse path locally when possible #75

Closed

CAR based Gateway implementation #62

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: IPIP-402 based backpressure #160

feat: IPIP-402 based backpressure #160

aschmahmann commented Jul 6, 2023 •

edited

Loading

BigLep commented Jul 11, 2023

lidel Jul 25, 2023 •

edited

Loading

aschmahmann Jul 25, 2023

lidel Jul 26, 2023 •

edited

Loading

aschmahmann Jul 26, 2023

aschmahmann Jul 31, 2023

BigLep commented Aug 1, 2023

lidel Jul 27, 2023 •

edited

Loading

lidel Aug 2, 2023 •

edited

Loading

aschmahmann Aug 14, 2023

lidel left a comment

aschmahmann commented Aug 14, 2023

feat: IPIP-402 based backpressure #160

feat: IPIP-402 based backpressure #160

Conversation

aschmahmann commented Jul 6, 2023 • edited Loading

BigLep commented Jul 11, 2023

lidel Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

Current state (iiuc)

Proposed fix

Rationale: ul.URL already takes care of escaping, we should reuse it as abstraction if possible

aschmahmann Jul 25, 2023

Choose a reason for hiding this comment

lidel Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

aschmahmann Jul 26, 2023

Choose a reason for hiding this comment

aschmahmann Jul 31, 2023

Choose a reason for hiding this comment

BigLep commented Aug 1, 2023

lidel Jul 27, 2023 • edited Loading

Choose a reason for hiding this comment

lidel Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

aschmahmann Aug 14, 2023

Choose a reason for hiding this comment

lidel left a comment

Choose a reason for hiding this comment

aschmahmann commented Aug 14, 2023

aschmahmann commented Jul 6, 2023 •

edited

Loading

lidel Jul 25, 2023 •

edited

Loading

Rationale: `ul.URL` already takes care of escaping, we should reuse it as abstraction if possible

lidel Jul 26, 2023 •

edited

Loading

lidel Jul 27, 2023 •

edited

Loading

lidel Aug 2, 2023 •

edited

Loading