Make the explorer persistent

[ffakenz]

Why

Once we start hosting the explorer, it becomes necessary to upgrade the backend with every release. Maintaining persistence eliminates the need to restart the explorer from genesis during each upgrade.

What

The explorer now incrementally persists on-chain transactions and recovers its state based on them.

• CHANGELOG updated or not needed
• Documentation updated or not needed
• Haddocks updated or not needed
• No new TODOs introduced or explained herafter

[github-actions]

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-02-02 15:02:50.163310544 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	985245919fcc6c0c5cd116023cd2c947c43e80dcbb5075fe12433fbb	4072
νCommit	7cb20fa71eb4c563ca283566ebe0aa65859d96c3f8cba35c52c181fd	2043
νHead	7a36661f5c15e9f1783aeaab890812c59b7286cbbc6de762d3110772	8816
μHead	8b111ac12274e46314769295a1c5dcab1d260096fc469fd698065463*	3851

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4375	10.46	4.04	0.46
2	4573	12.74	4.91	0.49
3	4776	14.94	5.74	0.52
5	5177	19.39	7.43	0.59
10	6184	30.20	11.51	0.75
41	12418	98.92	37.55	1.77

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	532	11.37	4.44	0.30
2	723	15.04	6.07	0.35
3	909	18.85	7.75	0.40
5	1285	26.90	11.27	0.51
10	2217	49.55	20.97	0.80
19	3901	99.43	41.75	1.43

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	480	21.78	8.51	0.41
2	113	590	32.21	12.73	0.53
3	170	704	45.18	18.02	0.68
4	225	810	64.72	25.79	0.89
5	283	920	77.07	31.08	1.04
6	338	1035	93.29	37.84	1.22

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	540	16.68	7.67	0.36
2	773	18.45	9.34	0.40
3	899	20.30	10.97	0.43
5	1234	23.84	14.20	0.50
10	2147	32.08	21.90	0.66
50	8933	98.51	83.51	1.94

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	570	20.27	8.97	0.40
2	769	22.30	10.71	0.44
3	895	23.78	12.04	0.47
5	1353	28.15	15.82	0.55
10	2196	37.71	24.02	0.72
42	7617	96.77	75.17	1.81

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4325	18.85	8.12	0.55
2	4468	31.97	13.92	0.71
3	4642	48.15	21.10	0.90
4	4840	65.99	29.06	1.11
5	5021	87.58	38.64	1.36

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4205	8.26	3.46	0.43
5	1	57	4239	9.49	4.22	0.44
5	5	286	4376	14.19	7.18	0.51
5	10	570	4546	20.55	11.08	0.60
5	20	1139	4885	32.85	18.71	0.77
5	30	1708	5225	45.15	26.34	0.93
5	40	2276	5563	57.33	33.92	1.10
5	50	2845	5902	69.56	41.52	1.27
5	74	4214	6719	99.13	59.87	1.68

End-To-End Benchmark Results

This page is intended to collect the latest end-to-end benchmarks results produced by Hydra's Continuous Integration system from the latest master code.

Please take those results with a grain of salt as they are currently produced from very limited cloud VMs and not controlled hardware. Instead of focusing on the absolute results, the emphasis should be on relative results, eg. how the timings for a scenario evolve as the code changes.

Generated at 2024-02-02 15:05:56.046757007 UTC

Baseline Scenario

Number of nodes	3
Number of txs	9000
Avg. Confirmation Time (ms)	22.600302914
P99	64.59003430000023ms
P95	32.408841899999985ms
P50	20.0022655ms
Number of Invalid txs	0

Baseline Scenario

Number of nodes	1
Number of txs	3000
Avg. Confirmation Time (ms)	3.996533692
P99	5.276622409999989ms
P95	4.721905499999999ms
P50	3.9081349999999997ms
Number of Invalid txs	0

[ch1bo]

IMO this creates more problems than it helps the way it is currently implemented (see my comment about explorer-state).

Furthermore, I don't even agree to the Why of this PR (although I like that you have spelled it out). Would have been better if we discuss this before you spend time on this (although it seemed to be doable quickly)

[ch1bo]

Should: not re-use hydra-node library, but create new trace data types.

This makes it harder to decouple this later from it (and hydra-chain-observer). Also, we are not logging from a hydra-node here, but from the explorer.

[ch1bo]

Could: make this consistent

Either use all the toArg functions or none.

Eventually we need to get rid of using them anyway as we can not depend on hydra-node or hydra-chain-observer when we do #1282

[ch1bo]

Must: include chain information in persisted data to avoid inconsistencies by design

This explorer-state seems not to include any information about which point on the chain was last seen. Hence, we can easily "miss heads" if we start the explorer, stop it and then start it again after a few minutes (e.g. when doing maintenance)

This must be avoided! Having an explorer synchronizing everything from a fixed point (--start-chain-from or even genesis) is strictly more correct than this half-way persisting. Also, it is not a big problem if starting the explorer takes a few minutes before it is fully live (can be controlled by the --start-chain-from option).

I would like to discuss the premise of this whole PR please.

[ffakenz]

Closed as rejected.