[caplin] processBlock: failed to process block body operations: outstanding deposits do not match maximum deposits, parentRoot: e0791f9d3108b4a2 1489efb48fbde37a06f07cf3f904ffa9b0fcbc530cf00ccf

[wmitsuda]

14/01/25 main commit. I started to see this on my mainnet node this morning.

log keeps repeating, so it looks like it entered an unrecoverable state.

[INFO] [01-16|08:22:54.475] head updated                             age=1m55s head=0x87b26246f9c8128165ce6a8053636b745ac8a510742692731dfed044
64c9c853 hash=0x87b26246f9c8128165ce6a8053636b745ac8a510742692731dfed04464c9c853 commit=29.275178ms alloc=8.1GB sys=25.6GB
[INFO] [01-16|08:22:54.818] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=253ms
[INFO] [01-16|08:23:09.516] P2P                                      app=caplin peers=22
[INFO] [01-16|08:23:43.284] Pre-cached shuffled set                  epoch=339042 len=1059877 mix=0x44272b2a2e6cea8df4afc8952eb1d1806656ac09fb
c39148d6d275af72415828
[INFO] [01-16|08:24:00.719] [4/6 Execution] Commit every block       blk=21635820 blks=5 blk/s=0.2 txs=555 tx/s=24 gas/s=2.59M buf=22.1MB/512.
0MB stepsInDB=0.56 step=1728.6 inMem=false alloc=13.9GB sys=25.6GB
[INFO] [01-16|08:24:09.516] P2P                                      app=caplin peers=17
[INFO] [01-16|08:24:27.863] [4/6 Execution] Commit every block       blk=21635824 blks=5 blk/s=0.2 txs=673 tx/s=24 gas/s=2.38M buf=39.9MB/512.
0MB stepsInDB=0.56 step=1728.6 inMem=false alloc=14.3GB sys=25.6GB
[INFO] [01-16|08:24:40.151] [4/6 Execution] Commit every block       blk=21635826 blks=3 blk/s=0.2 txs=302 tx/s=24 gas/s=2.15M buf=46.8MB/512.
0MB stepsInDB=0.56 step=1728.6 inMem=false alloc=8.2GB sys=25.6GB
[INFO] [01-16|08:24:46.105] [p2p] GoodPeers                          eth68=26 eth67=24
[INFO] [01-16|08:24:46.736] [mem] memory stats                       Rss=16.5GB Size=0B Pss=16.5GB SharedClean=2.5MB SharedDirty=0B PrivateCle
an=1.2GB PrivateDirty=15.3GB Referenced=15.8GB Anonymous=15.3GB Swap=0B alloc=8.2GB sys=25.6GB
[INFO] [01-16|08:24:47.182] [txpool] stat                            pending=2286 baseFee=3613 queued=4140 alloc=8.2GB sys=25.6GB
[INFO] [01-16|08:25:09.516] P2P                                      app=caplin peers=15
[INFO] [01-16|08:26:05.730] [4/6 Execution] Done Commit every block  blk=21635827 blks=12 blk/s=0.1 txs=1795 tx/s=12 gas/s=1.21M buf=53.1MB/51
2.0MB stepsInDB=0.00 step=1728.6 inMem=false alloc=8.9GB sys=25.6GB
[INFO] [01-16|08:26:05.733] [4/6 Execution] DONE                     in=2m27.355564402s block=21635816
[INFO] [01-16|08:26:09.515] P2P                                      app=caplin peers=17
[INFO] [01-16|08:26:10.810] head updated                             age=2m59s head=0x5629afffeb71607cc87a5c964c0e76ccfa609a4fc797c8bb4d7e097e
d1adb480 hash=0x5629afffeb71607cc87a5c964c0e76ccfa609a4fc797c8bb4d7e097ed1adb480 commit=119.711317ms alloc=8.9GB sys=25.6GB
[INFO] [01-16|08:26:11.284] Timings: Post-Forkchoice (slower than 50ms) Prune TxLookup=361ms Prune Execution=61ms
[INFO] [01-16|08:27:09.516] P2P                                      app=caplin peers=15
[INFO] [01-16|08:27:46.235] [p2p] GoodPeers                          eth68=27 eth67=23
[INFO] [01-16|08:27:46.999] [mem] memory stats                       Rss=20.4GB Size=0B Pss=20.4GB SharedClean=2.5MB SharedDirty=0B PrivateCle
an=2.3GB PrivateDirty=18.1GB Referenced=19.5GB Anonymous=18.1GB Swap=0B alloc=10.3GB sys=25.6GB
[INFO] [01-16|08:27:47.182] [txpool] stat                            pending=2213 baseFee=3888 queued=4193 alloc=10.6GB sys=25.6GB
[INFO] [01-16|08:28:09.516] P2P                                      app=caplin peers=15
[INFO] [01-16|08:29:09.515] P2P                                      app=caplin peers=13
[INFO] [01-16|08:30:09.516] P2P                                      app=caplin peers=11
[INFO] [01-16|08:30:46.100] [p2p] GoodPeers                          eth68=28 eth67=24
[INFO] [01-16|08:30:46.813] [mem] memory stats                       Rss=21.9GB Size=0B Pss=21.9GB SharedClean=2.5MB SharedDirty=0B PrivateCle
an=3.4GB PrivateDirty=18.5GB Referenced=21.3GB Anonymous=18.5GB Swap=0B alloc=11.8GB sys=25.6GB
[INFO] [01-16|08:30:47.180] [txpool] stat                            pending=3103 baseFee=4103 queued=4439 alloc=11.8GB sys=25.6GB
[INFO] [01-16|08:31:09.515] P2P                                      app=caplin peers=9
[INFO] [01-16|08:32:09.515] P2P                                      app=caplin peers=10
[INFO] [01-16|08:33:09.518] P2P                                      app=caplin peers=13
[INFO] [01-16|08:33:46.100] [p2p] GoodPeers                          eth68=28 eth67=25
[INFO] [01-16|08:33:46.862] [mem] memory stats                       Rss=21.2GB Size=0B Pss=21.2GB SharedClean=2.5MB SharedDirty=0B PrivateCle
an=3.6GB PrivateDirty=17.6GB Referenced=20.7GB Anonymous=17.6GB Swap=0B alloc=11.9GB sys=25.6GB
[INFO] [01-16|08:33:47.181] [txpool] stat                            pending=3884 baseFee=4256 queued=4569 alloc=12.0GB sys=25.6GB
[INFO] [01-16|08:34:09.515] P2P                                      app=caplin peers=14
[INFO] [01-16|08:35:09.515] P2P                                      app=caplin peers=14
[INFO] [01-16|08:35:53.600] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=10849248 to=10849377
[WARN] [01-16|08:36:05.286] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: AddChainSe
gment: processBlock: failed to process block body operations: outstanding deposits do not match maximum deposits, parentRoot: e0791f9d3108b4a2
1489efb48fbde37a06f07cf3f904ffa9b0fcbc530cf00ccf"
[WARN] [01-16|08:36:08.343] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: AddChainSe
gment: processBlock: failed to process block body operations: outstanding deposits do not match maximum deposits, parentRoot: e0791f9d3108b4a2
1489efb48fbde37a06f07cf3f904ffa9b0fcbc530cf00ccf"
[INFO] [01-16|08:36:09.515] P2P                                      app=caplin peers=11
[WARN] [01-16|08:36:12.418] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: AddChainSe
gment: processBlock: failed to process block body operations: outstanding deposits do not match maximum deposits, parentRoot: e0791f9d3108b4a2
1489efb48fbde37a06f07cf3f904ffa9b0fcbc530cf00ccf"

[wmitsuda]

restarting unstuck the node, but anyway there is an issue here where some scenario leads to that situation.