This repository has been archived by the owner on Jun 20, 2024. It is now read-only.
generated from ipfs/ipfs-repository-template
-
Notifications
You must be signed in to change notification settings - Fork 20
feat: IPIP-402 based backpressure #160
Merged
Merged
Changes from all commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
83eec5d
chore: update boxo
aschmahmann ce24a3f
feat: switch GetCAR to be backpressured
aschmahmann 7ec385a
feat: switch ResolvePath, Head and GetBlock to be backpressured
aschmahmann 468796a
feat: switch Get to be backpressured (not fully implemented)
aschmahmann db4ecfe
switch fetch to do escaping
aschmahmann 2458e3e
fix detection of go-ipld-prime schema errors and path resolution
aschmahmann 8b2cdc7
fix boxo go.mod
aschmahmann 56fce88
fix passing range bytes in CAR requests, and reporting entity ranges
aschmahmann 965e29d
chore: remove unused code
aschmahmann c2fb61a
feat: add support for backpressured tar files
aschmahmann 0691af9
chore: removed unused testing tools
aschmahmann 7f8eb45
feat: add some partial traversal tests and fixes
aschmahmann 60f2d4a
fix: HAMT fetching bug for Get requests
aschmahmann 1c28ef4
test: add tests for Get on HAMT directories
aschmahmann f473fea
feat: fix and add tests for Get on large files. Consolidate some Get …
aschmahmann 5a99a19
feat: allow passing a custom prometheus registry to the graph gateway…
aschmahmann d7383de
fix: remove unneeded block fetching remnants from graph backend. also…
aschmahmann 79523e7
fix: resolve issue with file resumption on error and add test
aschmahmann 5e44660
chore: remove unneeded blocks pubsub implementation
aschmahmann c0f250e
fix: TestGetHAMTDirectory is no longer flaky, although it is more imp…
aschmahmann be81dfc
fix: preserve '/' characters even after escaping to help URL parsers …
aschmahmann c3fb8d8
refactor: add contentPathToCarUrl and tests
lidel 7691b62
bump boxo dependency to handle _redirects better and switch from boxo…
aschmahmann 236009c
update caboose to fix issues draining responses
aschmahmann 53101eb
add more information when erroring on receiving an unexpected cid
aschmahmann 7df26b0
fix: switch use of ErrPartialResponse to not use pointers so they wor…
aschmahmann b1e8074
test: add test for file range requests with errors during download
aschmahmann b3d2909
fix: pass through range parameters for partial file load resumption
aschmahmann 03cba57
fix: perform gateway error conversion for graph backend
aschmahmann 28801f0
feat: add ipip-412 parameters for order and duplicates to non-saturn …
aschmahmann db66484
fix: allow reads with empty car roots (#182)
willscott 8c217f9
fix memory leak
aarshkshah1992 06077bc
update to latest caboose
willscott 0bbc0c4
Send verification error to Caboose (#184)
aarshkshah1992 addb2a1
Merge remote-tracking branch 'origin/main' into feat/backpressure
willscott 4958149
Merge branch 'feat/backpressure' of github.com:ipfs/bifrost-gateway i…
willscott c9fd369
bump go-unixfsnode and enable returning the sizes of items in HAMT di…
aschmahmann e2be930
return gateway errors on CAR fetching failure
aschmahmann d8485e2
add verifcid check
aschmahmann 1c54c91
test: run conformance with GRAPH_BACKEND=true
lidel a978e34
fix: correctly decode UnixFS symlinks
aschmahmann 4c5e7f9
fix: return starting at the correct offset when doing a Get request t…
aschmahmann ce2ef89
fix: add identity-blockstore style behavior to carToLinearBlockGetter
aschmahmann 1275db5
chore: add guard to GetCar so we don't accidentally forget to handle …
aschmahmann e605576
feat: handle IPIP-412 requests with and without duplicates, backed by…
aschmahmann c23e66f
ci: use a kubo development commit for conformance tests
aschmahmann 983ad96
test: fix TestGetHAMTDirectory to not look for //index.html
aschmahmann 004ea10
test: fix CAR retry tests to use CarWriter that allows duplicate blocks
aschmahmann 5325e53
Merge branch 'main' into feat/backpressure
aschmahmann File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -43,14 +43,20 @@ | |
return err | ||
} | ||
goLog.Debugw("car fetch", "url", req.URL) | ||
req.Header.Set("Accept", "application/vnd.ipld.car") | ||
req.Header.Set("Accept", "application/vnd.ipld.car;order=dfs;dups=y") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note: this resolves #177 for non-Saturn backends. Caboose/Saturn will have to change when they're ready. |
||
resp, err := ps.httpClient.Do(req) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
if resp.StatusCode != http.StatusOK { | ||
return fmt.Errorf("http error from car gateway: %s", resp.Status) | ||
errData, err := io.ReadAll(resp.Body) | ||
if err != nil { | ||
err = fmt.Errorf("could not read error message: %w", err) | ||
} else { | ||
err = fmt.Errorf("%q", string(errData)) | ||
} | ||
return fmt.Errorf("http error from car gateway: %s: %w", resp.Status, err) | ||
} | ||
|
||
err = cb(path, resp.Body) | ||
|
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: check if we have tests for when CAR backend returns HTTP 502 or 504.
We want to ensure pass-through works and the same code is returned to end user.