Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node stays behind approximately distance to snapshot. #339

Open
trufa opened this issue Oct 26, 2024 · 5 comments
Open

Node stays behind approximately distance to snapshot. #339

trufa opened this issue Oct 26, 2024 · 5 comments

Comments

@trufa
Copy link

trufa commented Oct 26, 2024

I have installed I'm running a base client pointed towards my up to date, synced L1.

eth_blockNumber of my L2 stays behind for approximately the same distance to the end of the snapshots, and then keeps the pace but doesn't' advance.

For example if I point HOST_DATA_DIR to the /reth-data (empty) it starts syncing very fast but from the very beginning

If I point it to my snapshot it starts also syncing very fast

node-1 | t=2024-10-26T01:58:38+0000 lvl=info msg="Advancing bq origin" origin=0x42bccb52f4bcdfd7cb77145632f14f98e3c276c8b370d0322c5a7ec68d683231:21015340 originBehind=false

but until certain point and they kind of keeps up but always lagging behind around 3-4 days which is the distance to the end of my snapshotted data.

I've tried running it --l1.trustrpc but doesn't seem to make any difference.

The following command as suggested by the docs:

echo Latest synced block behind by: $((($( date +%s )-
$( curl -s -d '{"id":0,"jsonrpc":"2.0","method":"optimism_syncStatus"}' -H "Content-Type: application/json" http://localhost:7545 |
jq -r .result.unsafe_l2.timestamp))/60)) minutes

Always keeps falling behind.

Any pointers would be greatly appreciated.

The errors I see in the log is one that I don't think is related

execution-1 | ts=2024-10-26T01:08:00.38459394Z level=error target=engine::jwt-validator message="Invalid JWT: Authorization header is missing or invalid"

And one that might be

node-1 | t=2024-10-26T01:51:23+0000 lvl=warn msg="Engine temporary error" err="temp: failed to sync forkchoice with engine: context deadline exceeded"

The peer count is in general around 8 and goes up to 11

@jun0tpyrc
Copy link

jun0tpyrc commented Oct 29, 2024

i was/am still hitting same thing , can you check log of reth if synchronising in batch mode?!

OP_NODE_SYNCMODE=execution-layer
OP_NODE_L2_ENGINE_KIND=reth <== seem without this  https://docs.optimism.io/builders/node-operators/configuration/consensus-config#l2enginekind reth doesn't go into batch mode for me, and it is missing in env example

and i also needed to tune this to be 10x larger https://reth.rs/run/config.html#merkle

then i can go to in-sync from similar scenario of few days, but still some performance issues unsolved occasionally(random beteween few seconds - 1hour delay some times ) on reth (already on nvme & not seeing obvious cpu bottleneck)

@trufa
Copy link
Author

trufa commented Oct 29, 2024

@jun0tpyrc I honestly gave up trying to make reth work and now switched to geth, it seems to be syncing reasonably fast now.

@lacotWin
Copy link

I had quite hard times here. I experienced static 12h drag and struggled a lot to fix it.
OP_NODE_L2_ENGINE_KIND=reth
fixes the problem for me. I believe it should be documented in this guide as there is no mention of it, not even in .env files

@jun0tpyrc
Copy link

I had quite hard times here. I experienced static 12h drag and struggled a lot to fix it. OP_NODE_L2_ENGINE_KIND=reth fixes the problem for me. I believe it should be documented in this guide as there is no mention of it, not even in .env files

wonder what spec you on &/ able to stay within minute of delay on reth for base-mainnet?! 🙏

@azeonmarket
Copy link

peer count 8-11 is quite low for an optimized performance, but if you managed to fix the issue great to hear ! true that OP_NODE_L2_ENGINE_KIND=reth is something you might have to add on configs, but its something you dont necessarily need but can be useful to know if you encounter some problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants