geyser: stream all accounts state outside of startup stage

[fanatid]

Problem

Right now if somebody wants to build an indexer based on geyser plugin they need to create a custom plugin and can't use gRPC plugin because geyser stream all accounts state only on the startup stage and it's not possible to get it later. With streaming all accounts it would be possible to create read layers outside of instance where node work.

Proposed Solution

agave:

• stream all accounts by request (set is_startup to true?)
• provide access to the bank (see Proof of Concept: Add update_bank to Geyser interface solana-labs/solana#32094), with bank it's possible to do much more than just stream all accounts
• ?

outside:

• stream all accounts on startup to some storage and update it with new accounts in the loop (full data duplication)

cc @lijunwangs

[lijunwangs]

This can be achieved by replaying the states from a recent full snapshot and followed by incremental snapshots until the difference is small enough to cut over to the normal geyser plugin. This could have serious performance impact and can be opted in by explicit validator configuration.

[ceciEstErmat]

This would be a really useful useful feature at my company and i'm more than willing to implement it.

Taking a look at the code i see that :
The notify process in reconstruct_accountsdb_from_fields starts once the restore is complete.
https://github.com/anza-xyz/agave/blob/master/runtime/src/serde_snapshot.rs#L1213
But, the notify itself does not really take that into account, it just gets all the slots from the storage, get the entry for each and notify.
https://github.com/anza-xyz/agave/blob/master/accounts-db/src/accounts_db/geyser_plugin_utils.rs#L41

Thus, I'm wondering why we can't do the same here ? Maybe going over "all slots" is not necessary in our case and just starting from the "latest slot" to the last full snapshot slot is enough ?

A Geyser request would then be added to the rpc_handler for the user to notify an willingness to start the stream process.

[diman-io]

You could use ledger-tool for this