Failed to prune, and get error "no snapshot paired state".

[hubv]

System information

Bor client version: v1.2.0 & v1.2.0-beta2
Heimdall client version: v1.0.3

OS & Version: Linux

Environment: Polygon Mainnet

Type of node: Sentry

Overview of the problem

Bor client can't prune the database because of "no snapshot paired state" after we upgrading to v1.2.0.
There is no "Unclean shutdown" and we stopped the node gracefully before the pruning.

Logs / Traces / Output / Error Messages

Error Log

bin/bor  snapshot  prune-state  -datadir.ancient   /data01/bor/.bor/bor/chaindata/ancient -datadir  /data01/bor/.bor/  
ERROR[12-10|06:38:18.699] Failed to prune state                    err="no snapshot paired state"

Logs of Node

[klestrong]

I have the same issue also. I even downloaded the latest Bor snapshot (https://snapshots.polygon.technology/ | 2023-12-02 14:01) and tried to prune after syncing back up. I resolved the issue by walking back the code past Bor v1.1.0 until I was able to prune Bor successfully with Bor v1.0.6. Bor v1.1.0 also failed to prune with the same error.

[Cognacy]

@klestrong Are you still experiencing this same issue?

[klestrong]

@Cognacy
Yes. Just earlier today Bor prune failed with v1.2.0 and v1.1.0. I'm currently able to prune with v1.0.6. I've repeatedly sopped the docker to sync for hours, then try to prune, but fail with the same message. I'm sure I'm doing graceful shutdown as I wait for Bor log to indicate the application stopped and my log session ends the docker container before I stop (delete) it.

When Bor prune fails, I get the same message as the original poster, @hubv .

My docker run is below:

docker run --rm -p 30303:30303 -p 8545:8545 -v /mnt/bor-pv:/bor:rw --net polygon --name bor -d <my bor image> server --datadir /bor --port 30303 --bor.heimdall 'http://heimdallr:1317' --http --http.addr '0.0.0.0' --http.vhosts '*' --http.corsdomain '*' --http.port 8545 --ipcpath /bor/bor.ipc --http.api 'eth,net,web3,txpool,bor' --syncmode 'full' --cache '4096' --miner.gasprice '30000000000' --miner.gaslimit '20000000' --txpool.nolocals --txpool.accountslots 16 --txpool.globalslots 32768 --txpool.accountqueue 16 --txpool.globalqueue 32768 --txpool.pricelimit '30000000000' --maxpeers 200 --bootnodes "enode://0cb82b395094ee4a2915e9714894627de9ed8498fb881cec6db7c65e8b9a5bd7f2f25cc84e71e89d0947e51c76e85d0847de848c7782b13c0255247a6758178c@44.232.55.71:30303,enode://88116f4295f5a31538ae409e4d44ad40d22e44ee9342869e7d68bdec55b0f83c1530355ce8b41fbec0928a7d75a5745d528450d30aec92066ab6ba1ee351d710@159.203.9.164:30303"

My prune command is below:

docker run --rm -it -v /mnt/bor-pv:/datadir 0xpolygon/bor:1.2.0 snapshot prune-state --datadir=/datadir

[klestrong]

@Cognacy
Pruning with previous version v1.0.6 did not resolve the issue. I'm still unable to prune Bor. Pruning with the mentioned version took abnormally long and resulted in disabling Bor as it will no longer sync. Please see captured errors. I'm pulling from a backup, but am still unable to prune. Do you guys have a resolution for this?

[drcliche]

Same issue exists on 1.2.0 for us.

Workaround: Downgrade to 1.1.0, let the node run for a while as usual, then prune with 1.1.0.

[klestrong]

@drcliche
I executed as you described, prune using v1.1.0 after letting it snapshot. The prune was successful and Bor synced up afterwards. Thanks for the assist, much appreciated.

[pratikspatil024]

Thanks for reporting, guys.
We have identified the issue and it has been fixed in PR#1110.

[odannis]

I tried the version 1.2.1, but it didn't solve the problem. I still got the error "no snapshot paired state"

[temaniarpit27]

We will be soon releasing a fix for this in early first week of January.

[odannis]

Ok nice, but downgrading to v1.1.0 works !
Thank you a lot