Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.0.16 #13

Closed
wants to merge 77 commits into from
Closed
Show file tree
Hide file tree
Changes from 74 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
651310c
use yellowstone-proto v1.13.0+solana.1.17.28
grooviegermanikus Apr 10, 2024
c1ca3e2
1.13 with buffer window size
grooviegermanikus Apr 11, 2024
1d2937a
clippy
grooviegermanikus Apr 16, 2024
e9eb3ec
relax yellowstone dependencies
grooviegermanikus Apr 16, 2024
7f115fb
Adding notify channel to stop tasks
godmodegalactus Mar 27, 2024
280f03d
Changing Notify channel to broadcast channel
godmodegalactus Apr 2, 2024
e000a96
bump version to 0.10.5
grooviegermanikus Apr 16, 2024
aad4baf
cargo
grooviegermanikus Apr 16, 2024
fef2e70
cleanup
grooviegermanikus Apr 16, 2024
dd18a20
use dedicated AST
grooviegermanikus Apr 16, 2024
85d04ac
export GeyserGrpcClient, GeyserGrpcClientResult, GeyserGrpcClientError
grooviegermanikus Apr 16, 2024
20f29c1
bump version to 0.10.6
grooviegermanikus Apr 16, 2024
1245d8b
Merge branch 'main' into release/v0.10.x+yellowstone.1.13
grooviegermanikus Apr 18, 2024
dd78804
clarify await_or_exit behavior on channel close
grooviegermanikus Apr 29, 2024
1d1a01c
clarify await_or_exit behavior on channel close
grooviegermanikus Apr 29, 2024
929b20e
setup simple account subscription
grooviegermanikus Apr 29, 2024
f280f90
cleanup
grooviegermanikus Apr 29, 2024
e09e5e2
collectors
grooviegermanikus Apr 29, 2024
1393527
calculate delta
grooviegermanikus Apr 29, 2024
edd3a1a
histogram
grooviegermanikus Apr 29, 2024
eb964e3
log deltas
grooviegermanikus Apr 29, 2024
28c3041
log blocktime
grooviegermanikus Apr 29, 2024
2d65f2d
flip delta time
grooviegermanikus Apr 29, 2024
1e83353
document on slot delta
grooviegermanikus Apr 29, 2024
8e5b1b8
log update span
grooviegermanikus Apr 30, 2024
a6cb6bb
mock correct size distribution
grooviegermanikus May 2, 2024
39199b8
stream token accounts
grooviegermanikus May 3, 2024
2cef5ef
token account
grooviegermanikus May 3, 2024
1b536cb
build map of mint+owner
grooviegermanikus May 3, 2024
e278c14
build simple map
grooviegermanikus May 3, 2024
f60e9f2
timestamp tagging
grooviegermanikus May 6, 2024
e4bb528
configure logger
grooviegermanikus May 6, 2024
09d4700
parse logs
grooviegermanikus May 7, 2024
b64f700
args
grooviegermanikus May 7, 2024
1b5d4f4
provide logs as args
grooviegermanikus May 7, 2024
087625d
track phoenix orderbook
grooviegermanikus May 7, 2024
88f8d71
count data
grooviegermanikus May 7, 2024
193a1b6
make SubscribeRequest forward-compatible
grooviegermanikus May 8, 2024
bd439a3
Merge branch 'release/v0.10.x+yellowstone.1.13' into groovie/performa…
grooviegermanikus May 8, 2024
78b8197
calc delta using two slot sources
grooviegermanikus May 8, 2024
86412cf
define AtomicSlot
grooviegermanikus May 8, 2024
430847b
debounce update log
grooviegermanikus May 8, 2024
61b202a
stop subscribing blocks meta
grooviegermanikus May 8, 2024
6ffc8d0
bench ONLY spl accounts
grooviegermanikus May 8, 2024
32aeb33
try to enable compression
grooviegermanikus May 8, 2024
f620664
fix ordering of slot change
grooviegermanikus May 10, 2024
4157779
subscribe to all accounts
grooviegermanikus May 10, 2024
03b3d14
add x-token for slot1+slot2
grooviegermanikus May 13, 2024
08f17d3
account to CSV
grooviegermanikus May 13, 2024
71d49a5
token accounts logging
grooviegermanikus May 14, 2024
5710402
parse for time tagged log files
grooviegermanikus May 17, 2024
e366b59
read only vote transactions
grooviegermanikus Jun 6, 2024
a57860e
calculate vote timestamp spread
grooviegermanikus Jun 6, 2024
beeca85
fix account bench
grooviegermanikus Jun 7, 2024
759170c
collect hashes
grooviegermanikus Jun 7, 2024
af17800
xor+lz4
grooviegermanikus Jun 7, 2024
460ca4c
data delta
grooviegermanikus Jun 12, 2024
238f44a
fix compile
grooviegermanikus Jun 13, 2024
3fd7e6d
enable slot check
grooviegermanikus Jun 13, 2024
6319612
smarter lag logging
grooviegermanikus Jun 13, 2024
a4a93de
allow for mock reset
grooviegermanikus Jun 13, 2024
c096a04
set nodelay + add env for window+biffer sizes
grooviegermanikus Jun 14, 2024
0604573
log if env config missing
grooviegermanikus Jun 14, 2024
1724047
compare account subscription with confirmation
grooviegermanikus Aug 12, 2024
c239e1b
clarify
grooviegermanikus Aug 12, 2024
18729e8
dump
grooviegermanikus Nov 5, 2024
8574783
wip
grooviegermanikus Nov 5, 2024
c71cbbe
accounts stream from da11 ledger-tool
grooviegermanikus Nov 12, 2024
6e97616
subscripbe accounts
grooviegermanikus Nov 14, 2024
ce03864
Update to 2.0.16
Nov 20, 2024
e10e3f4
Applied cargo fmt
Nov 20, 2024
214886a
Integrates groovies stuff
Nov 25, 2024
e90ecd1
Applies PR feedback and some clippy fixes
Nov 26, 2024
9ce723f
Makes clippy happy
Nov 26, 2024
88ebcea
fix rust+deps version
grooviegermanikus Dec 9, 2024
8acb7d3
comment on loop break
grooviegermanikus Dec 9, 2024
89f27a3
comment on yellowstone version
grooviegermanikus Dec 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,265 changes: 1,320 additions & 945 deletions Cargo.lock

Large diffs are not rendered by default.

25 changes: 18 additions & 7 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "geyser-grpc-connector"
version = "0.10.1+yellowstone.1.12"
version = "0.11.0+yellowstone.2.0.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yellowstone proto is 1.15 not 2.0.0 afaik

edition = "2021"

description = "Multiplexing and Reconnection on Yellowstone gRPC Geyser client streaming"
Expand All @@ -9,16 +9,15 @@ authors = ["GroovieGermanikus <[email protected]>"]
repository = "https://github.com/blockworks-foundation/geyser-grpc-connector"

[dependencies]
yellowstone-grpc-client = { version = "1.13.0+solana.1.17.15", git = "https://github.com/rpcpool/yellowstone-grpc.git", tag = "v1.12.0+solana.1.17.15" }
yellowstone-grpc-proto = { version = "1.12.0+solana.1.17.15", git = "https://github.com/rpcpool/yellowstone-grpc.git", tag = "v1.12.0+solana.1.17.15" }

yellowstone-grpc-client = { version = "2.0.0", git = "https://github.com/rpcpool/yellowstone-grpc.git", tag = "v2.0.0+solana.2.0.16" }
yellowstone-grpc-proto = { version = "2.0.0", git = "https://github.com/rpcpool/yellowstone-grpc.git", tag = "v2.0.0+solana.2.0.16" }

# required for CommitmentConfig
solana-sdk = "~1.17.15"
solana-sdk = "~2.0.16"

url = "2.5.0"
async-stream = "0.3.5"
tokio = { version = "1.28" , features = ["rt"] }
tokio = { version = "1.28" , features = ["rt", "rt-multi-thread"] }
futures = "0.3.28"
merge-streams = "0.1.2"
anyhow = "1.0.70"
Expand All @@ -31,7 +30,19 @@ base64 = "0.21.5"
bincode = "1.3.3"

csv = "1.3.0"
dashmap = "6.1.0"
tonic = { version= "0.12.3", features=["gzip"] }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tonic dependency had some historic version issues. do we need that here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@grooviegermanikus - I think those dependencies cam from your performancetest branch - I just updated tonic to most recent - we need this to enable gzip in yellowstone_grpc_util

        GeyserClient::with_interceptor(channel, interceptor)
            .accept_compressed(CompressionEncoding::Gzip)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets move forward, i found a workaround for the zeroize version issue.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can drop Gzip support if it helps

tonic-health = "0.12.3"
regex = "1.10.4"
clap = { version = "4.2", features = ["derive"] }

lz4_flex = "0.11.3"

[dev-dependencies]
tracing-subscriber = "0.3.16"
solana-logger = "1"
solana-logger = "2.0.16"
solana-account-decoder = "2.0.16"

[patch.crates-io.curve25519-dalek]
git = "https://github.com/anza-xyz/curve25519-dalek.git"
rev = "b500cdc2a920cd5bff9e2dd974d7b97349d61464"
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ These are the currently maintained versions of the library: [see Wiki](https://g
```cargo add geyser-grpc-connector ```


An example how to use the library is provided in `stream_blocks_mainnet.rs`.
An example how to use the library is provided in `stream_blocks_mainnet_stream.rs`.

## Known issues
* Library does not support other data than Blocks/Slots very well.
Expand Down
Empty file added accounts-testnet.csv
Empty file.
4 changes: 2 additions & 2 deletions ci/rust-version.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
if [[ -n $RUST_STABLE_VERSION ]]; then
stable_version="$RUST_STABLE_VERSION"
else
stable_version=1.70.0
stable_version=1.81.0
This conversation was marked as resolved.
Show resolved Hide resolved
fi

if [[ -n $RUST_NIGHTLY_VERSION ]]; then
Expand Down Expand Up @@ -91,7 +91,7 @@ export rust_nightly_docker_image=solanalabs/rust-nightly:"$nightly_version"
if [[ -n $RUST_STABLE_VERSION ]]; then
stable_version="$RUST_STABLE_VERSION"
else
stable_version=1.70.0
stable_version=1.81.0
fi

if [[ -n $RUST_NIGHTLY_VERSION ]]; then
Expand Down
40 changes: 40 additions & 0 deletions examples/accounts_meta_to_csv.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
use itertools::Itertools;
use std::fs::File;
use std::io;
use std::io::BufRead;
use std::path::PathBuf;

pub fn main() {
let accounts_meta_file =
PathBuf::from("/Users/stefan/mango/projects/geyser-misc/ledger-debug-accounts.txt");

let file = File::open(accounts_meta_file).expect("file must exist");
let reader = io::BufReader::new(file);
for blocks in &reader.lines().chunks(9) {
let blocks = blocks.collect_vec();
let account_pk = blocks[0].as_ref().unwrap().replace(':', "");
if account_pk.is_empty() {
break;
}
let owner_pk = blocks[2].as_ref().unwrap();
let ltick = owner_pk.find('\'');
let rtick = owner_pk.rfind('\'');
let owner_pk = &owner_pk[ltick.unwrap() + 1..rtick.unwrap()];

let data_len = blocks[6].as_ref().unwrap().replace(" data_len: ", "");

println!("{};{};{}", account_pk, owner_pk, data_len);
}
}

/*
16FMCmgLzCNNz6eTwGanbyN2ZxvTBSLuQ6DZhgeMshg:
balance: 0.00095352 SOL
owner: 'Feature111111111111111111111111111111111111'
executable: false
slot: 0
rent_epoch: 0
data_len: 9
data: 'AQAAAAAAAAAA'
encoding: "base64"
*/
Loading
Loading