feat: Network and NodeMetadata Protobuf

[thomas-swirlds-labs]

Description:

• Defines a protobuf-based network object to replace the legacy config.txt file, fully encapsulating the information required to bootstrap a network with Threshold Signature Scheme (TSS) enabled and with the legacy address book completely replaced by Roster

Design and Motivations:

• Eliminates the requirement for separate overrides or genesis configurations, providing a standardized format for storing metadata about all nodes in a network, including TssEncryptionKeys, Roster entries, and ledgerIds.
• Note: this is a format for data that exists on disk, not in state or block streams, and can be regenerated during state freeze operations for network resets.
• Each node in the network is represented by NodeMetadata, which includes fields such as the TssEncryptionKey (now removed from the Roster) and other information previously found in the Address Book.
• This ensures compatibility for both Genesis startup (networks starting without state) and Network Transplant (starting from a previously saved state from another network).
• tssEncryptionKey is required here because it has been removed from the Roster and will need to be create here with Tss Key Material
• LedgerId is required, because it needs to be validated against when you start the new network

Related issue(s):

Fixes #16549

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.36%. Comparing base (65af0d1) to head (5ca3c13).
Report is 4 commits behind head on develop.

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #16581      +/-   ##
=============================================
- Coverage      63.37%   63.36%   -0.02%     
+ Complexity     20170    20151      -19     
=============================================
  Files           2532     2533       +1     
  Lines          94122    94125       +3     
  Branches        9848     9862      +14     
=============================================
- Hits           59650    59639      -11     
- Misses         30855    30863       +8     
- Partials        3617     3623       +6     

see 22 files with indirect coverage changes

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.02% (target: -1.00%)	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (65af0d1)	96941	63129	65.12%
Head commit (5ca3c13)	96865 (-76)	63062 (-67)	65.10% (-0.02%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#16581)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

[jsync-swirlds]

It isn't entirely clear (even after reading the linked issue and further linked Notion page) what these messages are for.

General suggestions:

1. Detail in the PR description all of "what", "why", and "how" of the PR.
   • What is being done here, design/feature wise
   • Why is this needed, and why is it designed in this way
   • How is this to be used and how does it meet future needs
2. Make sure that proto file specification always explains all of the relevant requirements for each message and field.
3. In a proto file comment, add any relevant additional detail, possibly including the "what/why/how" from above, as applied to that particular file.
4. Writing too much description is, in my experience, almost never a problem. The common problem is PRs, specifications, design documentation (i.e. code files), and related text that is woefully short and imprecise.

[thomas-swirlds-labs]

It isn't entirely clear (even after reading the linked issue and further linked Notion page) what these messages are for.

General suggestions:

1. Detail in the PR description all of "what", "why", and "how" of the PR.

   • What is being done here, design/feature wise
   • Why is this needed, and why is it designed in this way
   • How is this to be used and how does it meet future needs

2. Make sure that proto file specification always explains all of the relevant requirements for each message and field.

3. In a proto file comment, add any relevant additional detail, possibly including the "what/why/how" from above, as applied to that particular file.

4. Writing too much description is, in my experience, almost never a problem. The common problem is PRs, specifications, design documentation (i.e. code files), and related text that is woefully short and imprecise.

@jsync-swirlds. Agreed. I've updated the description of the PR to include more detailed description of the motivation and design and give some more background on the what/why/how. Hopefully that should provide a little more clarity now

[jsync-swirlds]

A few suggestions:

1. If this is purely for on-disk data, it doesn't belong in the state folder. Placing it in state implies it will be stored in state; and it is not (cannot be) designed properly for both uses.
2. Why are we even using protobuf for this? On-disk startup files should be human readable and human editable to support operations teams, and protobuf is not a good format for that purpose. I would think a YAML file would be the best option, closely followed by JSON.
3. It should be clearly documented in these files that this is an on-disk configuration file definition.
4. The HAPI documentation guidelines do not apply to on-disk data formats; but that requires this definition to be well separated from the HAPI files (not in the hapi module).

[edward-swirldslabs]

Can we specify the structure of the file in protobuf and still have a JSON representation on disk? Could it even be YAML, translated to JSON, then parsed by protobuf?

[jsync-swirlds]

It is possible to output JSON from a protobuf specification, but it's not a production option; that format is intended for debugging and development. In particular it's not guaranteed to round-trip deterministically.

It might be much easier to just create a Java record class and have that output and input as YAML, using whichever YAML processor we prefer.

Protobuf-JSON format might work; I cannot say for certain either way, but it's definitely not what I would recommend.

One other thought. This is an internal private detail of the consensus node that should not be published in the HAPI API or presented to SDKs in the hedera-protobufs repo. As such (and as noted above) this definition probably belongs in a different module (hedera-app, or perhaps the network service?).

[edward-swirldslabs]

If we are producing this file by hand, that may be problematic. It's meant to be an automated export (probably at startup) and consumed as an override-network in the transplant and reset use case. With the TSS information, there's a lot that can go wrong. It's all binary data and doesn't go well for by-hand assembly.

[jsync-swirlds]

The TSS data is binary, and probably should be retained as such (e.g. base64 encoded).
The Node data, however, very likely needs to be edited every time (because the destination network won't have the same nodes), so the overall file format still needs to be editable.

[Neeharika-Sompalli]

Yeah the reason I remember, we added a protobuf object was to be able to easily parse TSS key data. @tinker-michaelj thoughts ?