Balances: Arbitrary number of genesis accounts

[ggwpez]

We have the issue for benchmarking that we need at least 1M accounts in the genesis state, which does not work well with JSON files and can cause some parts to allocate high memory.

Additionally to having a Vec here, we should add a dev_accounts: (u32, T::Balance) field that makes the runtime generate an arbitrary number of derived dev accounts when building the genesis state:

polkadot-sdk/substrate/frame/balances/src/lib.rs

Line 507 in c0ddfba

pub balances: Vec<(T::AccountId, T::Balance)>,

Should probably be feature-flagged for benchmarks.

[xlc]

why JSON genesis for such use case? wouldn't it be faster to just operate on the raw storage?

[runcomet]

relevant other links would be helpful to look closely on this :)

[ggwpez]

why JSON genesis for such use case? wouldn't it be faster to just operate on the raw storage?

@xlc our omni tooling works with genesis presets. Relying on specific storage addresses would probably work in practice, but it is a bit hacky IMHO. What downside do you see in having this?

relevant other links would be helpful to look closely on this :)

I added the link to the exact line of code in the issue. Not sure what else i can explain about it, but you can check the build function where the genesis state is build:

polkadot-sdk/substrate/frame/balances/src/lib.rs

Line 545 in c0ddfba

assert!(T::AccountStore::insert(who, AccountData { free, ..Default::default() })

[michalkucharczyk]

Would be nice if the accounts were compatible with polkadot-stps and ttxt tools.

They use this derivation pattern:
https://github.com/paritytech/polkadot-stps/blob/8de8d4af332b278bda4f99e7ed619caa6b581de0/utils/src/lib.rs#L37
https://github.com/paritytech/polkadot-stps/blob/8de8d4af332b278bda4f99e7ed619caa6b581de0/utils/sender/src/pre/mod.rs#L19

Or even better, we could pass the derivation string (actually address uri string) that has a placeholder for the index: "//Sender/{}"

That would allow some flexibility.

[xlc]

it is just there is big overhead of creating such big json and parse it in the node and convert to raw compare to just supply the raw format. I am worried there is possibility of some steps may not work well with such big file. but maybe it will just works.

[michalkucharczyk]

it is just there is big overhead of creating such big json and parse it in the node and convert to raw compare to just supply the raw format. I am worried there is possibility of some steps may not work well with such big file. but maybe it will just works.

With this proposal chain spec JSON file will be simple and small. Here is example (1000000 accounts each with 500 balance):

{
  "name": "Custom",
  "id": "custom",
  "chainType": "Live",
  "bootNodes": [],
  "telemetryEndpoints": null,
  "protocolId": null,
  "properties": null,
  "codeSubstitutes": {},
  "genesis": {
    "runtimeGenesis": {
      "code": "0x..",
      "patch": {
        "balances": {
          "dev_accounts": [1000000, 500]
        },
      }
    }
  }
}

It would be:

...
 "dev_accounts": [1000000, 500, "//Sender/{}"]
...

if my proposal with address uri sounds ok.

[xlc]

oh I see. I misunderstood the issue.

[runcomet]

would like to work on this

Assign if possible

[kianenigma]

One of the only downsides of adding a new field to RuntimeGenesisConfig would be that it might will a breaking change, at least when compiling a node with hardcoded chainspecs.

[michalkucharczyk]

One of the only downsides of adding a new field to RuntimeGenesisConfig would be that it might will a breaking change, at least when compiling a node with hardcoded chainspecs.

That would be the case only for hardcoded RuntimeGenesisConfig (meaning native runtime) or for the nodes that are using full genesis config (which I don't think is the case as we advocate patches).

[bkchr]

That would be the case only for hardcoded RuntimeGenesisConfig (meaning native runtime) or for the nodes that are using full genesis config (which I don't think is the case as we advocate patches).

Also most of the code probably already used ..Default::default() syntax for this since quite some time.

Generally I like the proposed solution here! Also means that not every tooling needs to replicate a way to generate these accounts and can instead just set this field.

[ggwpez]

@runcomet I assigned you. We will need this for: #5891
LMK if you can pick it up soon, otherwise i will do so.

[kianenigma]

@gpestana you might want to add a similar mechanism for staking: just receive two numbers, and generate that many stakers.

At least it removes one issue with creating testnets with large number of stakers.

[runcomet]

@runcomet I assigned you. We will need this for: #5891

LMK if you can pick it up soon, otherwise i will do so.

🙏

Hacking on it