Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add s3 batch consumer #43

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
be0d884
add localstack s3 support for local testing
cortze Jan 16, 2025
7508c93
add first working draft of a s3 trace submitter + tests
cortze Jan 16, 2025
ebdec2b
WIP - extesion of hermes cmd to s3 config
cortze Jan 16, 2025
68e8573
update dependencies adding parquet-go and s3
cortze Jan 16, 2025
bd69e76
address feedback from Dennis
cortze Jan 22, 2025
4a0ff35
update root config for s3 extension
cortze Jan 22, 2025
1952a05
add: parquet and s3 benchmarks cmd
cortze Jan 22, 2025
042ae7f
add: localstack s3 instance to CI tests
cortze Jan 22, 2025
e8c8fb2
update: ignore .json files
cortze Jan 22, 2025
e036050
fix: make ci happy
cortze Jan 22, 2025
2b6e786
feat: add s3conf part of the eth-node struct + siplify init
cortze Jan 22, 2025
18fb939
update: s3 config init + loggin on s3 datastream creation
cortze Jan 22, 2025
ccedfc2
fix: spotted bug on logging level flag
cortze Jan 22, 2025
865b786
fix: add non-locking batcher methods + remove possible race-conditions
cortze Jan 22, 2025
9686d6a
ensure error on datastream init failure + apply s3 credentials only w…
cortze Jan 23, 2025
3b03b39
updt: s3key format to only producer
cortze Jan 23, 2025
db2dd04
update Docs
cortze Jan 23, 2025
cfcca1f
update CI test format
cortze Jan 23, 2025
920b266
add s3 tag
cortze Jan 27, 2025
626dfb0
WIP: not done yet
cortze Jan 27, 2025
4609b85
fix: bug on event type
cortze Jan 27, 2025
50f7182
fix: new tag
cortze Jan 27, 2025
1a68153
first working version of the specific event formating (still WIp, cod…
cortze Jan 28, 2025
6411a26
Merge branch 'upgrade-event-parquet-formatting' into add-s3-batch-con…
cortze Jan 28, 2025
35cf7b6
standarize fields on kinesis payloads
cortze Jan 29, 2025
81e19ef
add: parquet format
cortze Jan 29, 2025
a0e350c
add: event specific parquet structs
cortze Jan 29, 2025
afd00ce
adapt: s3-related code to independet parquet formats
cortze Jan 29, 2025
e61a36b
add: snappy compression by default
cortze Jan 29, 2025
672867d
fmt code
cortze Jan 29, 2025
2d05de7
rm: analysis py.file from repo
cortze Jan 29, 2025
7598eb7
correct: wrong pointer reference on connection renderer
cortze Jan 29, 2025
d026f40
make .env functional at docker-compose
cortze Jan 30, 2025
4f0106d
remove sync.Map from the eventStore
cortze Jan 30, 2025
18f4b3c
address comments
cortze Jan 30, 2025
52c1831
add timeout to s3 healthcheck on github actions' workflow
cortze Jan 30, 2025
99dca22
add metrics
cortze Jan 31, 2025
8ea241c
replace: ipdx go-test for actions/setup-go@v5
cortze Jan 31, 2025
23074a9
fix env variables at github actions
cortze Jan 31, 2025
60ff16d
fix env variables
cortze Jan 31, 2025
73a464c
get rid of the health check
cortze Jan 31, 2025
a4589bf
fix: wrong s3 init.py path
cortze Jan 31, 2025
e978511
2nd attempt to fix the path
cortze Jan 31, 2025
15ed940
make sure localstack doesn't copy the file as root
cortze Jan 31, 2025
a05a8ba
fix: change file to read-only
cortze Jan 31, 2025
9fdd276
fix: last chance
cortze Jan 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address feedback from Dennis
  • Loading branch information
cortze committed Jan 22, 2025
commit bd69e761ef570cc19cfd55ddd40555cd5a69794d
10 changes: 9 additions & 1 deletion host/flush_tracer.go
Original file line number Diff line number Diff line change
@@ -69,13 +69,21 @@ func (t *TraceEvent) toParquet() *ParquetTraceEvent {
}

type ParquetTraceEvent struct {
Timestamp int64
Type string
Topic string
PeerID string
Timestamp int64
Payload string
}

func (pte *ParquetTraceEvent) BytesLen() int {
return 8 + // int64
len([]byte(pte.Type)) +
len([]byte(pte.Topic)) +
len([]byte(pte.PeerID)) +
len([]byte(pte.Payload))
}

var _ gk.Record = (*TraceEvent)(nil)

var _ pubsub.RawTracer = (*Host)(nil)
Loading