v3.0.19

[wpank]

Candidates Config and DB Changes:

• updates candidate files to have a slotId and kyc field (set to false if they have not been set yet) https://github.com/w3f/1k-validators-be-PRIVATE/issues/16
  • unique on candidates not set yet - there needs to be separations of non-candidates telemetry nodes (which will otherwise have null slotIds to get implemented first)
• check for duplicates of stash or telemetry name at startup (will log list if duplicates are found) https://github.com/w3f/1k-validators-be-PRIVATE/issues/21
• Updates CI check to make sure all candidates in the config file have a slotId and kyc field
• adds scripts for helping generate a config json with those fields
• After this is deployed, any candidate with the same slotId field can be merged / changed without any need to adjust things manually in the db. There may still need to be an editing of existing duplicates, though. https://github.com/w3f/1k-validators-be-PRIVATE/issues/3

Misc Refactoring:

• refactors scorekeeper to multiple files https://github.com/w3f/1k-validators-be-PRIVATE/issues/30
• removes council and democracy score weight in configs, legacy democracy schema and endpoints, democracy / delegation jobs Opengov cleaning leftovers #2564 https://github.com/w3f/1k-validators-be-PRIVATE/issues/27
• removes dry run in scorekeeper / nominations, as well as prod configs Dry run and Test deprecation cleaning #2571
• cleans up unused fields returned from candidates endpoint in gateway Opengov cleaning leftovers #2564
• removes unused Bootstrap.ts job (was intended to be run from local instance to query prod instance and set data to what's returned, but isn't used).
• Updates the Config.ts to be able to get a candidates list either from a url (prod version is a github url), or a local path (for when running locally)
• Updates weight defaults in constants to match prod / staging values
• Updates constraints to fetch location data from the latest updated location db value instead of what was previously a field on the candidate schema
• location and infrastructureData fields get removed from each candidate db entry on startup in the cleaning hook (as they are instead fetched fresh from the location db records)
• fix add nominator startup error https://github.com/w3f/1k-validators-be-PRIVATE/issues/37
• renames references to controller to bondedAddress in nominator.ts in core, as controllers are deprecated https://github.com/w3f/1k-validators-be-PRIVATE/issues/36
• refactors scorekeeper into multiple files implement Factory and Template patterns #2607
  • moves cron.ts and jobs.ts (both only run as monolith, as functions imported from the separate Worker package, as cron jobs) -> into new /scorekeeper/jobs/cron/ folder
• refactors chaindata into multiple files (removes a bunch of unused functions)
• refactors constraints.ts into multiple files (in core/src/constraints) https://github.com/w3f/1k-validators-be-PRIVATE/issues/23
• removes most unused functions in common and core

Consistency FIxes:

• updates gateway cache to be shorter https://github.com/w3f/1k-validators-be-PRIVATE/issues/22
  • gateway cache is read from config, or defaults to a new value defined in constants https://github.com/w3f/1k-validators-be-PRIVATE/issues/40
• removes location data from candidates schema - all location related fields / queries are queried fresh from the location schema db items
• makes ranks deterministic - EraPointsJob has been updated to query for all eras until the beginning of the chain - a candidates rank is the max of the number of eras there exists era points for of any identity address that is apart of a validators sub/super identity. This has removed rank updates from scorekeeper rounds https://github.com/w3f/1k-validators-be-PRIVATE/issues/33
• moves Nomination db records to be created when a non-proxy nomination tx happens, or when a proxy tx is executed rather than when the proxy announcement is made https://github.com/w3f/1k-validators-be-PRIVATE/issues/34

Scoring Changes:

• changes nominatedAt score from 30 -> 80 https://github.com/w3f/1k-validators-be-PRIVATE/issues/24
• updates location score max/min distribution bounds. This makes the location / provider score a lot more evenly distributed https://github.com/w3f/1k-validators-be-PRIVATE/issues/22 https://github.com/w3f/1k-validators-be-PRIVATE/issues/28

API Changes:

• adds /candidates/rank endpoint, that returns the list of all candidates (valid and invalid), ordered by rank https://github.com/w3f/1k-validators-be-PRIVATE/issues/33
• adds /locationstats/valid endpoint, which returns the list of locations from only valid nodes, along with the associated score (this should map 1:1 with a candidates score - so long as the score is also updated around the same time as the endpoint is fetched from. The endpoint calculates it on the fly, so this is a more recent source of 'truth' - when the score of a candidate updates it should be the same as what's gotten from this endpoint when queried from) https://github.com/w3f/1k-validators-be-PRIVATE/issues/32
• adjust the /nominators endpoint to list the validators of the last Nomination record - this reflects the validators that are currently on-chain, rather than the validators that will be nominated in the next round. https://github.com/w3f/1k-validators-be-PRIVATE/issues/34

Testing:

• refactor some of the existing tests that were in test/api to be Jest tests - these are a mix of integration tests and mocks. These are first just to get something working as well as a part of enabling it as a CircleCI check (these tests may be flakey, so if those fail that shouldn't be a blocker yet). So far the tests there pass. This also adds testCore as a check to CircleCI https://github.com/w3f/1k-validators-be-PRIVATE/issues/2

Note: since this adds new CI checks for the candidates file - #2597 should be merged before this so that CI checks for the candidates file will pass (as this introduces both the checks and the updated candidates files)