[Epic: js-waku] Reliability Protocol for Resource-Restricted Clients

[chair28980]

Our user story:

1. As a developer, I want a simple API to register hooks to process messages on one or several content topics: feat(filter)!: new simpler filter API  #2092
2. As a developer, I do not care about shards or pubsub topics, I am only using autosharding
3. As developer, I want fair reliability when receiving messages (e..g redundant filter nodes + filter ping + store sync)
4. As a developer, I do not want the same incoming message to be processed several times (e.g. filter redundancy should be abstracted)
5. As a developer, I want fair reliability when sending messages.

And our implementation notes:

Store

• implement store-v3: feat!: store v3 #2036

Combination

• Message verification and retry: feat: message verification and retry #2075
• Message retry for LightPush: feat(lightpush): add retries to failing peers #2069
• Continuous discovery: chore(peer-exchange): support updates of previously discovered peer's addresses  #2051

Liveness of a node:

• Node health state - feat: introduce node health as a metric  #2070
• Filter subscription recovery after offline state feat: improve offline recoverability of Filter  #2024
• (questionable) node recovery strategies feat: node recovery strategies #2076

Testing

• unit testing: feat: unit testing @waku/sdk #2078 and feat: unit testing @waku/core #2077
• Test pipeline - test: create scripts for running light-push/filter and measuring ratio of messages sent and received #2007

Filter:

• ensure number of peers to subscribe to is configurable, documented and respected by implementation; 0297682
• dynamic peer management: feat: peer management for protocols (with disconnection management) #2002
  • if filter ping fails on node 1 but not node 2, disconnect from node 1 and replace: feat(filter): peer/subscription renewal with recurring Filter pings #2052
  • introduce a monitor if no messages are coming from peer A but are coming from peer B - drop A: feat: validate messages for individual filter nodes & perform renewals #2057
• automatically connect and subscribe to 2-3 filter nodes: feat: SDK for redundant usage of filter/lightpush #1463;
• seen cache to ensure dupe messages are not bubbled up;
• when subscribicing, starting running filter ping: feat: run filter ping during filter subscription #1995;
• keep alive feature feat: add keep alive to Filter #1970
• filter pings should work per peer, instead of per subscription

LightPush:

• ensure number of peers to use is configurable and documented 0297682
• dynamic peer management: feat: peer management for protocols (with disconnection management) #2002
  • if any node from the pool rejects messages - replace it;
• connect to 2/3 lightpush node when sending messages;

---

First revision

Tasks (obsolete)

• write interop tests to cover non-relay client user stories;
• protocol reliability + exposing additional configs to enable / disable behavior + unit tests:
  • Filter/LightPush uses more than one peer: feat: SDK for redundant usage of filter/lightpush #1463
• orient protocols to be more RFC oriented + unit tests: chore: protocol implementations in @waku/core should be as unopinionated as possible #1886
  • Filter: chore(lightpush)!: move protocol implementation to @waku/sdk (1/n) #1964
  • LightPush: chore(lightpush)!: move protocol implementation opinions to @waku/sdk #1887
  • Store: chore(store)!: move protocol implementation opinions to @waku/sdk #1913

We can de-scope some of these streams in the interest of time.

Second revision

Filter:

• ensure number of peers to subscribe to is configurable, documented and respected by implementation;
• Filter subscription recovery after offline state feat: improve offline recoverability of Filter  #2024

LightPush:

• ensure number of peers to use is
  • configurable
  • documented

Store

• implement store-v3: feat!: store v3 #2036
• peer renewal for store peers;

Liveness of a node:

• feat: introduce node health as a metric  #2070
• [ ]

Testing

• unit testing;
• test: create scripts for running light-push/filter and measuring ratio of messages sent and received #2007

[chair28980]

@waku-org/js-waku-developers please proceed with creating the tasks required in the js-waku repository, please tag them with the E:js-waku Improve Reliability label, and include them in Task list above in the description.

[fryorcraken]

@waku-org/js-waku-developers please note that we now expect an RFC out of this work. Also, need to understand what is done in status-go and align. @chaitanyaprem would be a good person to discuss with as he has some familiarity with usage of req-res protocols in status-go

[weboko]

in status-go

I was planning to schedule knowledge sharing after Status offsite is done

we now expect an RFC out of this work

after knowledge sharing and revisiting completed work we will discuss RFC internally

[fryorcraken]

is there an understanding of what of the change listed above would give you most ROI?

[danisharora099]

is there an understanding of what of the change listed above would give you most ROI?

A lot of high-impact items have been merged, or are in progress. Other than that, imo:

• all protocols: disconnection management (P1)

• filter: validating filter messages from each node against each other to do disconnection (P3)

• filter: running recurring store queries to validate incoming filter messages (dependant on store v3) (P2)

• filter: pinging per peer, instead of per subscription (P3)

• lightpush: seems quite stable once disconnection management is implemented

• store: store v3

• client/node stability:

  • ensure peers are redialed (if not dialed already)
  • recover filter subscriptions
  • running store query upon reconnection to receive missed filter messages

[weboko]

Linking some relevant status-go issues that makes sense to take a look at in the scope of js-waku:

• bug: find out why filter subscriptions are taking time to stabilize when non fleet nodes are used status-im/status-go#5344
• feat: unsubscribe in case invalid push requests are received from server go-waku#1124
• fix: add peer addresses on expiry if peer is discovered again go-waku#1128

[danisharora099]

status-im/status-go#5344

Based on status-im/status-go#5350 and status-im/status-go#5344 (comment). it seems that status-go did not filter peers based on shards when discovered using Peer Exchange, thus attempting to open subscriptions with peers with unsupported pubsub topics.

waku-org/go-waku#1124

Investigating this: waku-org/go-waku#1124 (comment)

waku-org/go-waku#1128

Very interesting. Though we haven't run into a problem like this before (perhaps because of the nature of js-waku node being short lived, thus peer updating its addresses was never observed) -- but this seems to be a very important check to introduce. Opened an issue here: #2051

[chaitanyaprem]

This is the master issue i am using to track all fixes/changes wrt lightclient reliability being done in go-waku/status-go. waku-org/pm#219
@danisharora099 , @weboko you can refer to this to see if something that can be applied to js-waku and also can be added to the rfc.

[weboko]

Another point to consider from @jm-clius comment here - waku-org/specs#18 (review):

• verify delivered messages by tracking them by Filter subscription to some node that is different from the one used during LightPush.

Let's discuss at js-waku PM meeting, @danisharora099

[danisharora099]

Strictly based on the newly merged Reliability RFC, derived features and their current state in the js-waku codebase are:

Node Health Management

• Implement node health monitoring for connection states: feat: introduce node health as a metric  #2070
  • Define conditions for Unhealthy, Minimally Healthy, and Sufficiently Healthy states.

Peers and Connection Management

• Maintain a pool of reliable service nodes for
  • LightPush: feat(lightpush): peer management for protocols #2003
  • Filter: feat(filter): use protocol peer management #2047
• Replace failing service nodes upon detection: feat: peer management for protocols (with disconnection management) #2002
  • Filter: feat(filter): peer/subscription renewal with recurring Filter pings #2052, feat: validate messages for individual filter nodes & perform renewals #2057
  • LightPush: feat(lightpush): peer management for protocols #2003
• Implement preferences for selecting service nodes based on multiaddresses, protocols, and shard participation: feat!: protocols filter peers per shard #1756, feat!: set peer-exchange with default bootstrap #1469
• Continuously discover and update service node information: chore(peer-exchange): support updates of previously discovered peer's addresses  #2051
  • Ensure up-to-date Peer Store with ENR and Libp2p multiaddresses.

LightPush Protocol Enhancements

• Implement redundant message sending to multiple service nodes: feat: SDK for redundant usage of filter/lightpush #1463
• Implement retry mechanism for failed LightPush requests with increasing intervals: feat(lightpush): add retries to failing peers #2069
• Implement verification of message delivery using and retry feat: message verification and retry #2075
  • Store (v3) protocol: feat!: store v3 #2036
  • Filter protocol

Filter Protocol Enhancements

• Implement regular pings to maintain active subscriptions: feat: add keep alive to Filter #1970
• Implement redundant Filter subscriptions for message loss mitigation: feat: SDK for redundant usage of filter/lightpush #1463
• Handle duplicate messages using deterministic message hashing: feat: SDK for redundant usage of filter/lightpush #1463
• Implement offline recoverability mechanisms for network disconnections: feat: offline state recovery for Filter subscription #2049
• Initiate Filter pings and replace failing nodes upon network reconnection: feat: run filter ping during filter subscription #1995

[weboko]

Nice, good job at summarizing!
Let's discuss it at our PM meeting tomorrow as I believe some of the features we might want to make optional such as Store / Filter message verification and retry.

Also to note: we are waiting for update to Light Push to implement some of the strategies in the RFC such as renewing peers only in some cases and doing retries in others.

Upd#1: additional task #2074
Upd#2: one more #2076

[danisharora099]

This epic's deliverables are now all merged in, and can be closed! (minus the unit tests issues)
cc @weboko @chair28980

[weboko]

Unit testing is decoupled into debt and will be addressed independently.
Other things are described in waku-org/pm#186 (comment) and will be a fast follow up in dogfooding work stream.