[Deliverable] Status usage of Waku scaling and bandwidth optimization recommendation

[chair28980]

Task breakdown

This deliverable aims to review the Status app protocol stack to improve how well it scales within a Waku context, specifically by optimising bandwidth usage.

The following initial steps are proposed:

Step 1: Minimal Community Specification and Implementation

Specify community procedures, including the encryption, reliability and functional scope of each. This forum post serves as starting point. A basic spec will allow us to move community control + content messages to separate shards, reintroduce protected shards, and extract Community functionality from the convoluted 1:1, user, open shards, etc. During this step, obvious or pending improvements (the "low-hanging fruits" for Communities) can be addressed in parallel.

Step 2: Minimal telemetry

With Community Control + Content on separate shards, telemetry will likely already give us useful information by just considering per-shard bandwidth. E.g. we may notice that the actual community traffic is just a small percentage of overall bandwidth usage, a clear indication that we need to disable/limit 1:1 and user features significantly.

Step 3: Minimal solution for user (self-addressed) messages

Specify and implement a workaround solution (lightpush + store) that removes self-addressed procedures (backup/sync) from the relay network.

Epics

• nwaku
• go-waku
• js-waku

[fryorcraken]

The initial deliverable was meant to be an overall recommendation to then plan and tackle the work.
A different approach is proposed here. The benefit of the new approach is to implement some temporary solutions for a quick gain.

Note that the initial deliverable output was defined as follows:

The output of this deliverable is to compile a list of recommendations, for both Waku and Status Communities and chat protocols. This should include potential benefits of changes and enable scheduling the work between Status and Waku teams. Decision on the work to be done and planning it should be part of the output of this milestone.

However, we believe there are improvements we can do know. I suggest to define new deliverables to track those improvements.

Here is a new proposal for the bandwidth optimization and protocol review milestone, in order of priority:

Milestone: bandwidth optimization and protocol review

Once completed, further specification and analyse of Status chat protocol will be available; as well as a recommendation on how Status Chat protocol should use Waku in a scalable and bandwidth efficient manner.

Moreover, low hanging-fruits and temporary solutions to improve bandwidth consumption would have been implemented.

Deliverable: Minimal Community Specification and Implementation

1. Specify community procedures, including the encryption, reliability and functional scope of each. This forum post serves as starting point.

2. Using spec from (1) as a basis, define message flows to be moved to separate shards (e.g. community control + content messages). Extracting community messages off the common open shard (also used by 1:1 chats) should be considered.
   As well as other low-hanging fruits improvements previously identified.
   Breaking changes and migration plans, if necessary, should be specified as part of this output.

Finally, proceed with the implementation of (2).

Deliverable: Telemetry review

With Minimal Community Specification and Implementation

Using the output of Telemetry: Measure Bandwidth and Minimal Community Specification and Implementation (traffic separated by shards), analyse the distribution of traffic on relay and data in store. Providing a report in terms of Status Chat functionality.

Deliverable: Minimal solution for greedy messages

Based on telemetry review, specify and implement a workaround solution (light push + store) that removes most data greedy functionality from the relay network.
At the time of writing, we assume that self-addressed messages (backup/sync) and community description messages would be affected by this change.

Deliverable: Define long-term solution

The output of this deliverable is to compile a list of recommendations, for both Waku and Status Communities and chat protocols. This should include potential benefits of changes and enable scheduling the work between Status and Waku teams.
Decision on the work to be done and planning it should be part of the output of this deliverable.

The recommendation should be grounded in the output of the previous deliverables of this milestone.

The recommendation is unlikely to encompass the entire chat protocol, and all status message types due to the amount of work it would entailed. Instead, an empirical approach should be taken where changes are prioritised based on the user impact, known limitations and functionalities (e.g. profile backup and device pairing usage of Waku), and telemetry metrics.

This would impact any current usage of Waku by the Status app. Which does include communities, 1:1 chat but profile back and device pairing.

This could include review of discv5 implementation in go-waku and nwaku if bandwidth usage is excessive.

Deliverable: Review usage of content topics in Status Chat and Communities protocol

(unchanged)

The usage of content topics in Status is aligned with Wakuv1. Waku v2 comes with a new recommended format that enables auto-sharding.
Moreover, single Status users currently use a high number of content topics, which may have an impact on performance of req-res protocols such as store and filter.
Such impact is to be measured in a previous milestone by Vac DST.
This work should also include an understanding of the Status messages types, as well as their requirements in terms of priority, retention and reliability. As well as understanding how partial subscription to a given community can be done.
The output of this deliverable should be an RFC update on how content topics should be used, backed with simulations when performance improvement is expected.
It should include migration strategy and potential impact on the product.

[fryorcraken]

See https://roadmap.logos.co/waku/milestones/open/2024-bandwidth-optimization-and-protocol-review for new definition.