Graceful shutdown of OrientDB running as docker container

[suneelkumarch]

I am running OrientDB v3.2.2-tp3 as a pod in k8s

When pod is restarted, I get this WARNING
Storage 'db' was not closed properly. Will try to recover from write ahead log

How to ensure that OrientDB shutdown gracefully when a pod/container is terminated/restated.?

[tglman]

Hi,

OrientDB should already handle the SIGTERM signal properly and soft terminate, I'm not sure though if docker/kubernates do send termination signal, also be aware that is suggested to use volumes in containerized environments, the container file system usually is not good for databases, that may be one of the reason why you may get a "wrong shutdown" error.

[ptercero]

Hi,

I'm also running OrientDB in k8s environment and this is a common behaviour when a POD restarts. I use external volumes to allocate the databases.

Do you know if is there any command to schedule a graceful shutdown when a restart is planned?

thx

[tglman]

Hi,

yes in the distribution there is a sh to shutdown the server, here some example docs the shutdown can be run also from outside using the same orientdb distribution just with ./shutdown.sh -h host -u rootUser -p rootPassword .

In any case this behaviour seems a bit strange, do you have any documentation that explain how signals are sent to the container by kubernates ? maybe something is missing there.

Regards

[tglman]

Here it is: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace so the SIGTERM seems to be sent, maybe the reason why there is no a clean shutdown is because the grace period is to short, so try to increase that.

as well I would suggest to verify in the log that event are arriving, when OrientDB receive a shutdown will print something like:

WARNI Received signal: SIGINT [OSignalHandler]
INFO OrientDB Server is shutting down... [OServer]
.......
.......
INFO OrientDB Server shutdown complete [OServer]

[suneelkumarch]

@tglman Thank you!

In my case, I was running the "OrientDB server" process with PID other than 1. That was resulting NOT receving SIGTERM when pod/container shutdown. Correcting this, it worked and received the SIGTERM and initiated shutdown process when pod/container termintates.

[ptercero]

That was also my case too. I wasn't running the server with PID 1, thus, not receiving SIGTERM signal.
Changing this, the shutdown process completes correctly.
Thx @suneelkumarch @tglman