Besu doesn't terminate with 0 exit code

[MqllR]

Steps to Reproduce

1. Run besu locally

$ docker run -it --rm --name besu --entrypoint sh hyperledger/besu:latest
$ besu
2025-01-29 08:48:45.092+00:00 | main | INFO  | Besu | Starting Besu
2025-01-29 08:48:45.319+00:00 | main | WARN  | Besu | --rpc-http-host has been ignored because --rpc-http-enabled was not defined on the command line.
...

2. Exec into the container and send a TERM signal to besu

$ docker exec -it besu sh
$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
besu           1  0.0  0.0   2776  1664 pts/0    Ss   08:48   0:00 sh
besu          14  0.2  0.0   2776  1536 pts/1    Ss   08:48   0:00 sh
besu          20  284  3.5 6273984 566328 pts/0  Sl+  08:48   0:13 java -Dvertx.disableFileCPResolving=t
besu         140  0.0  0.0   8280  4224 pts/1    R+   08:48   0:00 ps aux
$ kill -TERM 20

3. Observe besu termination

...
2025-01-29 08:48:53.780+00:00 | BesuCommand-Shutdown-Hook | INFO  | SnapSyncDownloader | Stopping sync
2025-01-29 08:48:53.781+00:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Stopping Network.
2025-01-29 08:48:53.790+00:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | Stopping eth Subprotocol.
2025-01-29 08:48:53.791+00:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | eth Subprotocol stopped.
2025-01-29 08:48:53.792+00:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Network stopped.
2025-01-29 08:48:53.792+00:00 | BesuCommand-Shutdown-Hook | INFO  | AutoTransactionLogBloomCachingService | Shutting down Auto transaction logs caching service.
2025-01-29 08:48:53.793+00:00 | BesuCommand-Shutdown-Hook | INFO  | DockerNatManager | Stopping docker NAT manager.
$ echo $?
143

Expected behavior: If the besu node is gracefully shutdown, it should exit with 0.

Actual behavior: Graceful shutdown terminates with a non 0 exit code (interrupt exit with 130, termination with 143)

Frequency: 100%

[Gabriel-Trintinalia]

My nodes are having some issues to stop sometimes: it may be related:

^C2025-02-04 20:50:25.307+11:00 | vert.x-eventloop-thread-1 | DEBUG | RecursivePeerRefreshState | Neighbours round timed out
2025-02-04 20:50:25.307+11:00 | vert.x-eventloop-thread-1 | DEBUG | RecursivePeerRefreshState | Skipping bonding round because no candidates are available
2025-02-04 20:50:25.307+11:00 | vert.x-eventloop-thread-1 | DEBUG | RecursivePeerRefreshState | Iterative peer search complete.  19 peers processed over 8 rounds.
2025-02-04 20:50:32.877+11:00 | BesuCommand-Shutdown-Hook | ERROR | DefaultP2PNetwork | DefaultP2PNetwork did not shutdown cleanly: some internal services failed to fully terminate.

[macfarla]

this behaviour is the same outside of docker so it's nothing to do with docker.

The 130 exit code comes from the process being killed.

Added a log line for what besu thinks the exit code is.

besu -h exits gracefully

besu -h; echo $?
...
More info and other profiles at https://besu.hyperledger.org

Besu process exiting with exit code 0
0

besu with ctrl-c to kill the process

besu; echo $? 
...
2025-02-06 13:11:37.215+10:00 | main | INFO  | Runner | Ethereum main loop is up.
^C2025-02-06 13:12:24.607+10:00 | BesuCommand-Shutdown-Hook | INFO  | DefaultSynchronizer | Stopping synchronizer
2025-02-06 13:12:24.608+10:00 | BesuCommand-Shutdown-Hook | INFO  | SnapSyncDownloader | Stopping sync
2025-02-06 13:12:24.609+10:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Stopping Network.
2025-02-06 13:12:24.609+10:00 | BesuCommand-Shutdown-Hook | INFO  | DNSDaemon | Stopping DNSDaemon for enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net
2025-02-06 13:12:24.612+10:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | Stopping eth Subprotocol.
2025-02-06 13:12:24.613+10:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | eth Subprotocol stopped.
2025-02-06 13:12:24.613+10:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Network stopped.
2025-02-06 13:12:24.613+10:00 | BesuCommand-Shutdown-Hook | INFO  | AutoTransactionLogBloomCachingService | Shutting down Auto transaction logs caching service.
2025-02-06 13:12:24.613+10:00 | BesuCommand-Shutdown-Hook | INFO  | NatService | No NAT environment detected so no service could be stopped
2025-02-06 13:12:24.615+10:00 | vert.x-virtual-thread-3002 | INFO  | DNSDaemon | Stopping DNSDaemon for enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net
Besu process exiting with exit code 0
130

[macfarla]

exit codes explained

If you're running a program or script and press Ctrl-C to stop it, your exit status is 130.

https://www.redhat.com/en/blog/exit-codes-demystified

[macfarla]

Additional info here - you can't change the exit code within the shutdown hook
https://systemd-devel.freedesktop.narkive.com/V032lLrE/cannot-make-java-exit-0-on-sigterm

[macfarla]

@MqllR do you get closer to the behaviour you want with running besu within a service? don't know how that fits with docker though

[Unit]
Description=Besu Enterprise Ethereum java client
After=syslog.target network.target

[Service]
User=besu
Group=besu
Environment=HOME=/home/besu
Environment=LOG4J_CONFIGURATION_FILE=/opt/log4j/besu-log-config.xml
Type=simple
ExecStart=/bin/sh -c "/opt/besu/current/bin/besu --config-file=/etc/besu/config.toml >> /var/log/besu/besu.log 2>&1"
SuccessExitStatus=143
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

[macfarla]

TL;DR is if you're killing the java process - TERM, kill or ctrl-c - the exit code is 130.

[macfarla]

I notice in the systemd file we use, we set SuccessExitStatus=143
Not sure if there's a similar config for docker?

[MqllR]

Thanks for taking a look. The systemd config you shared is a good workaround but exiting with a 128+signal code doesn't seem right. You can check this doc https://tldp.org/LDP/abs/html/exitcodes.html and the sysexit.h about exit codes, but if besu exited gracefully, I'd expect it to return a 0 code.