Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

feat: IPIP-402 based backpressure #160

Merged
merged 49 commits into from
Aug 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
83eec5d
chore: update boxo
aschmahmann Jun 9, 2023
ce24a3f
feat: switch GetCAR to be backpressured
aschmahmann Jun 10, 2023
7ec385a
feat: switch ResolvePath, Head and GetBlock to be backpressured
aschmahmann Jun 12, 2023
468796a
feat: switch Get to be backpressured (not fully implemented)
aschmahmann Jun 12, 2023
db4ecfe
switch fetch to do escaping
aschmahmann Jun 29, 2023
2458e3e
fix detection of go-ipld-prime schema errors and path resolution
aschmahmann Jul 6, 2023
8b2cdc7
fix boxo go.mod
aschmahmann Jul 6, 2023
56fce88
fix passing range bytes in CAR requests, and reporting entity ranges
aschmahmann Jul 6, 2023
965e29d
chore: remove unused code
aschmahmann Jul 6, 2023
c2fb61a
feat: add support for backpressured tar files
aschmahmann Jul 13, 2023
0691af9
chore: removed unused testing tools
aschmahmann Jul 13, 2023
7f8eb45
feat: add some partial traversal tests and fixes
aschmahmann Jul 18, 2023
60f2d4a
fix: HAMT fetching bug for Get requests
aschmahmann Jul 21, 2023
1c28ef4
test: add tests for Get on HAMT directories
aschmahmann Jul 21, 2023
f473fea
feat: fix and add tests for Get on large files. Consolidate some Get …
aschmahmann Jul 24, 2023
5a99a19
feat: allow passing a custom prometheus registry to the graph gateway…
aschmahmann Jul 24, 2023
d7383de
fix: remove unneeded block fetching remnants from graph backend. also…
aschmahmann Jul 24, 2023
79523e7
fix: resolve issue with file resumption on error and add test
aschmahmann Jul 24, 2023
5e44660
chore: remove unneeded blocks pubsub implementation
aschmahmann Jul 24, 2023
c0f250e
fix: TestGetHAMTDirectory is no longer flaky, although it is more imp…
aschmahmann Jul 25, 2023
be81dfc
fix: preserve '/' characters even after escaping to help URL parsers …
aschmahmann Jul 25, 2023
c3fb8d8
refactor: add contentPathToCarUrl and tests
lidel Jul 26, 2023
7691b62
bump boxo dependency to handle _redirects better and switch from boxo…
aschmahmann Jul 27, 2023
236009c
update caboose to fix issues draining responses
aschmahmann Jul 27, 2023
53101eb
add more information when erroring on receiving an unexpected cid
aschmahmann Jul 27, 2023
7df26b0
fix: switch use of ErrPartialResponse to not use pointers so they wor…
aschmahmann Jul 27, 2023
b1e8074
test: add test for file range requests with errors during download
aschmahmann Jul 27, 2023
b3d2909
fix: pass through range parameters for partial file load resumption
aschmahmann Jul 27, 2023
03cba57
fix: perform gateway error conversion for graph backend
aschmahmann Jul 31, 2023
28801f0
feat: add ipip-412 parameters for order and duplicates to non-saturn …
aschmahmann Jul 31, 2023
db66484
fix: allow reads with empty car roots (#182)
willscott Aug 3, 2023
8c217f9
fix memory leak
aarshkshah1992 Aug 7, 2023
06077bc
update to latest caboose
willscott Aug 8, 2023
0bbc0c4
Send verification error to Caboose (#184)
aarshkshah1992 Aug 8, 2023
addb2a1
Merge remote-tracking branch 'origin/main' into feat/backpressure
willscott Aug 9, 2023
4958149
Merge branch 'feat/backpressure' of github.com:ipfs/bifrost-gateway i…
willscott Aug 9, 2023
c9fd369
bump go-unixfsnode and enable returning the sizes of items in HAMT di…
aschmahmann Aug 10, 2023
e2be930
return gateway errors on CAR fetching failure
aschmahmann Aug 10, 2023
d8485e2
add verifcid check
aschmahmann Aug 10, 2023
1c54c91
test: run conformance with GRAPH_BACKEND=true
lidel Aug 14, 2023
a978e34
fix: correctly decode UnixFS symlinks
aschmahmann Aug 15, 2023
4c5e7f9
fix: return starting at the correct offset when doing a Get request t…
aschmahmann Aug 15, 2023
ce2ef89
fix: add identity-blockstore style behavior to carToLinearBlockGetter
aschmahmann Aug 15, 2023
1275db5
chore: add guard to GetCar so we don't accidentally forget to handle …
aschmahmann Aug 15, 2023
e605576
feat: handle IPIP-412 requests with and without duplicates, backed by…
aschmahmann Aug 15, 2023
c23e66f
ci: use a kubo development commit for conformance tests
aschmahmann Aug 15, 2023
983ad96
test: fix TestGetHAMTDirectory to not look for //index.html
aschmahmann Aug 15, 2023
004ea10
test: fix CAR retry tests to use CarWriter that allows duplicate blocks
aschmahmann Aug 15, 2023
5325e53
Merge branch 'main' into feat/backpressure
aschmahmann Aug 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions .github/workflows/gateway-conformance.yml
Copy link
Collaborator

@lidel lidel Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: check if we have tests for when CAR backend returns HTTP 502 or 504.
We want to ensure pass-through works and the same code is returned to end user.

Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,16 @@ jobs:
car-env: ["GRAPH_BACKEND=false", "GRAPH_BACKEND=true"]
steps:
# 1. Start the Kubo gateway
- name: Download Kubo gateway
uses: ipfs/download-ipfs-distribution-action@v1
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: 1.19.x
- uses: protocol/cache-go-action@v1

- name: Install Kubo gateway from source
#uses: ipfs/download-ipfs-distribution-action@v1
run: |
go install github.com/ipfs/kubo/cmd/ipfs@73552dfd9c6f73fdab887c4d2e99cf7e848a3c23
- name: Start Kubo gateway
uses: ipfs/start-ipfs-daemon-action@v1

Expand Down Expand Up @@ -46,10 +54,6 @@ jobs:
echo "IPFS_NS_MAP=${IPFS_NS_MAP}" >> $GITHUB_ENV

# 4. Build the bifrost-gateway
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: 1.19.x
- name: Checkout bifrost-gateway
uses: actions/checkout@v3
with:
Expand Down
36 changes: 2 additions & 34 deletions blockstore.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,11 @@
import (
"context"
"errors"
"fmt"
"strings"
"time"

"github.com/filecoin-saturn/caboose"
"github.com/ipfs/bifrost-gateway/lib"
blockstore "github.com/ipfs/boxo/blockstore"
exchange "github.com/ipfs/boxo/exchange"
"github.com/ipfs/boxo/gateway"
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
"go.uber.org/zap/zapcore"
Expand Down Expand Up @@ -38,7 +35,7 @@

blk, err := e.bstore.Get(ctx, c)
if err != nil {
return nil, gatewayError(err)
return nil, lib.GatewayError(err)

Check warning on line 38 in blockstore.go

View check run for this annotation

Codecov / codecov/patch

blockstore.go#L38

Added line #L38 was not covered by tests
}
return blk, nil
}
Expand Down Expand Up @@ -67,33 +64,4 @@
return nil
}

// gatewayError translates underlying blockstore error into one that gateway code will return as HTTP 502 or 504
// it also makes sure Retry-After hint from remote blockstore will be passed to HTTP client, if present.
func gatewayError(err error) error {
if errors.Is(err, &gateway.ErrorStatusCode{}) ||
errors.Is(err, &gateway.ErrorRetryAfter{}) {
// already correct error
return err
}

// All timeouts should produce 504 Gateway Timeout
if errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, caboose.ErrSaturnTimeout) ||
// Unfortunately this is not an exported type so we have to check for the content.
strings.Contains(err.Error(), "Client.Timeout exceeded") {
return fmt.Errorf("%w: %s", gateway.ErrGatewayTimeout, err.Error())
}

// (Saturn) errors that support the RetryAfter interface need to be converted
// to the correct gateway error, such that the HTTP header is set.
for v := err; v != nil; v = errors.Unwrap(v) {
if r, ok := v.(interface{ RetryAfter() time.Duration }); ok {
return gateway.NewErrorRetryAfter(err, r.RetryAfter())
}
}

// everything else returns 502 Bad Gateway
return fmt.Errorf("%w: %s", gateway.ErrBadGateway, err.Error())
}

var _ exchange.Interface = (*exchangeBsWrapper)(nil)
10 changes: 8 additions & 2 deletions blockstore_proxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,20 @@
return err
}
goLog.Debugw("car fetch", "url", req.URL)
req.Header.Set("Accept", "application/vnd.ipld.car")
req.Header.Set("Accept", "application/vnd.ipld.car;order=dfs;dups=y")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this resolves #177 for non-Saturn backends. Caboose/Saturn will have to change when they're ready.

resp, err := ps.httpClient.Do(req)
if err != nil {
return err
}

if resp.StatusCode != http.StatusOK {
return fmt.Errorf("http error from car gateway: %s", resp.Status)
errData, err := io.ReadAll(resp.Body)
if err != nil {
err = fmt.Errorf("could not read error message: %w", err)
} else {
err = fmt.Errorf("%q", string(errData))
}
return fmt.Errorf("http error from car gateway: %s: %w", resp.Status, err)

Check warning on line 59 in blockstore_proxy.go

View check run for this annotation

Codecov / codecov/patch

blockstore_proxy.go#L53-L59

Added lines #L53 - L59 were not covered by tests
}

err = cb(path, resp.Body)
Expand Down
44 changes: 0 additions & 44 deletions blockstore_test.go

This file was deleted.

89 changes: 26 additions & 63 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,20 @@ module github.com/ipfs/bifrost-gateway
go 1.19

require (
github.com/cskr/pubsub v1.0.2
github.com/filecoin-saturn/caboose v0.0.4
github.com/filecoin-saturn/caboose v0.0.5-0.20230808125624-a986b542aa10
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/golang-lru/v2 v2.0.1
github.com/ipfs/boxo v0.11.0
github.com/ipfs/boxo v0.11.1-0.20230811122208-5de8f5f5ca60
github.com/ipfs/go-block-format v0.1.2
github.com/ipfs/go-cid v0.4.1
github.com/ipfs/go-ipld-format v0.5.0
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipfs/go-unixfsnode v1.7.1
github.com/ipld/go-car v0.6.0
github.com/ipfs/go-unixfsnode v1.7.4
github.com/ipld/go-car v0.6.2
github.com/ipld/go-car/v2 v2.10.2-0.20230622090957-499d0c909d33
github.com/ipld/go-codec-dagpb v1.6.0
github.com/ipld/go-ipld-prime v0.20.0
github.com/libp2p/go-libp2p v0.26.3
github.com/libp2p/go-libp2p-kad-dht v0.23.0
github.com/libp2p/go-libp2p-routing-helpers v0.7.0
github.com/mitchellh/go-server-timing v1.0.1
github.com/multiformats/go-multicodec v0.9.0
Expand All @@ -27,123 +26,88 @@ require (
github.com/spf13/cobra v1.6.1
github.com/stretchr/testify v1.8.4
github.com/willscott/go-requestcontext v0.0.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.40.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0
go.opentelemetry.io/contrib/propagators/autoprop v0.40.0
go.opentelemetry.io/otel v1.14.0
go.opentelemetry.io/otel/sdk v1.14.0
go.opentelemetry.io/otel/trace v1.14.0
go.opentelemetry.io/otel v1.16.0
go.opentelemetry.io/otel/sdk v1.16.0
go.opentelemetry.io/otel/trace v1.16.0
go.uber.org/atomic v1.10.0
go.uber.org/multierr v1.9.0
go.uber.org/zap v1.24.0
)

require (
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
github.com/asecurityteam/rolling v0.0.0-20230418204413-b4052899307d // indirect
github.com/benbjohnson/clock v1.3.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/containerd/cgroups v1.0.4 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/elastic/gosigar v0.14.2 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/flynn/noise v1.0.0 // indirect
github.com/francoispqt/gojay v1.2.13 // indirect
github.com/gabriel-vasile/mimetype v1.4.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
github.com/godbus/dbus/v5 v5.1.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/gddo v0.0.0-20180823221919-9d8ff1c67be5 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/gopacket v1.1.19 // indirect
github.com/google/pprof v0.0.0-20221203041831-ce31453925ec // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/huin/goupnp v1.0.3 // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/ipfs/bbloom v0.0.4 // indirect
github.com/ipfs/go-bitfield v1.1.0 // indirect
github.com/ipfs/go-blockservice v0.5.0 // indirect
github.com/ipfs/go-datastore v0.6.0 // indirect
github.com/ipfs/go-ipfs-blockstore v1.3.0 // indirect
github.com/ipfs/go-ipfs-delay v0.0.1 // indirect
github.com/ipfs/go-ipfs-ds-help v1.1.0 // indirect
github.com/ipfs/go-ipfs-exchange-interface v0.2.0 // indirect
github.com/ipfs/go-ipfs-redirects-file v0.1.1 // indirect
github.com/ipfs/go-ipfs-util v0.0.2 // indirect
github.com/ipfs/go-ipld-cbor v0.0.6 // indirect
github.com/ipfs/go-ipld-legacy v0.2.1 // indirect
github.com/ipfs/go-libipfs v0.6.0 // indirect
github.com/ipfs/go-log v1.0.5 // indirect
github.com/ipfs/go-merkledag v0.11.0 // indirect
github.com/ipfs/go-metrics-interface v0.0.1 // indirect
github.com/ipfs/go-verifcid v0.0.2 // indirect
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
github.com/jbenet/goprocess v0.1.4 // indirect
github.com/klauspost/compress v1.15.15 // indirect
github.com/klauspost/cpuid/v2 v2.2.3 // indirect
github.com/koron/go-ssdp v0.0.3 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-cidranger v1.1.0 // indirect
github.com/libp2p/go-doh-resolver v0.4.0 // indirect
github.com/libp2p/go-flow-metrics v0.1.0 // indirect
github.com/libp2p/go-libp2p-asn-util v0.2.0 // indirect
github.com/libp2p/go-libp2p-kad-dht v0.23.0 // indirect
github.com/libp2p/go-libp2p-kbucket v0.5.0 // indirect
github.com/libp2p/go-libp2p-record v0.2.0 // indirect
github.com/libp2p/go-msgio v0.3.0 // indirect
github.com/libp2p/go-nat v0.1.0 // indirect
github.com/libp2p/go-netroute v0.2.1 // indirect
github.com/libp2p/go-reuseport v0.2.0 // indirect
github.com/libp2p/go-yamux/v4 v4.0.0 // indirect
github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd // indirect
github.com/mattn/go-isatty v0.0.17 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/miekg/dns v1.1.50 // indirect
github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b // indirect
github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.8.0 // indirect
github.com/multiformats/go-multiaddr-dns v0.3.1 // indirect
github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
github.com/multiformats/go-multibase v0.2.0 // indirect
github.com/multiformats/go-multistream v0.4.1 // indirect
github.com/multiformats/go-varint v0.0.7 // indirect
github.com/onsi/ginkgo/v2 v2.5.1 // indirect
github.com/opencontainers/runtime-spec v1.0.3-0.20211123151946-c2389c3cb60a // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/openzipkin/zipkin-go v0.4.1 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/polydawn/refmt v0.89.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/quic-go/qpack v0.4.0 // indirect
github.com/quic-go/qtls-go1-19 v0.2.1 // indirect
github.com/quic-go/qtls-go1-20 v0.1.1 // indirect
github.com/quic-go/quic-go v0.33.0 // indirect
github.com/quic-go/webtransport-go v0.5.2 // indirect
github.com/raulk/go-watchdog v1.3.0 // indirect
github.com/serialx/hashring v0.0.0-20200727003509-22c0c7ab6b1b // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
Expand All @@ -154,35 +118,34 @@ require (
github.com/whyrusleeping/cbor-gen v0.0.0-20230126041949-52956bd4c9aa // indirect
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.42.0 // indirect
go.opentelemetry.io/contrib/propagators/aws v1.15.0 // indirect
go.opentelemetry.io/contrib/propagators/b3 v1.15.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.15.0 // indirect
go.opentelemetry.io/contrib/propagators/ot v1.15.0 // indirect
go.opentelemetry.io/otel/exporters/jaeger v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/internal/retry v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.16.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.14.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.14.0 // indirect
go.opentelemetry.io/otel/metric v0.37.0 // indirect
go.opentelemetry.io/otel/metric v1.16.0 // indirect
go.opentelemetry.io/proto/otlp v0.19.0 // indirect
go.uber.org/dig v1.15.0 // indirect
go.uber.org/fx v1.18.2 // indirect
go.uber.org/multierr v1.9.0 // indirect
golang.org/x/crypto v0.6.0 // indirect
golang.org/x/exp v0.0.0-20230213192124-5e25df0256eb // indirect
golang.org/x/mod v0.7.0 // indirect
golang.org/x/net v0.7.0 // indirect
golang.org/x/mod v0.8.0 // indirect
golang.org/x/net v0.8.0 // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/sys v0.6.0 // indirect
golang.org/x/text v0.7.0 // indirect
golang.org/x/tools v0.5.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/text v0.8.0 // indirect
golang.org/x/tools v0.6.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
gonum.org/v1/gonum v0.11.0 // indirect
google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f // indirect
google.golang.org/grpc v1.53.0 // indirect
google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 // indirect
google.golang.org/grpc v1.55.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.7 // indirect
nhooyr.io/websocket v1.8.7 // indirect
)
Loading