Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Max log size not applied when container restarts #266

Open
pascapascapascapasca opened this issue Jun 1, 2021 · 0 comments
Open

Max log size not applied when container restarts #266

pascapascapascapasca opened this issue Jun 1, 2021 · 0 comments

Comments

@pascapascapascapasca
Copy link

pascapascapascapasca commented Jun 1, 2021

The max log size option is set to 200MB through log_size_max = 209715200 in /etc/containers/containers.conf (no other option is uncommented in it), but the container logs exceed 200MB. This seems to happen when a container restarts and thus the conmon process is restarted. In my case, podman restarts the container every few minutes and the bytes written by the previous conmon process do not seem to be taken in count when applying the max log size.

Versions
CentOS 7

conmon --version
conmon version 2.0.21 commit: efe288263ebb1bab4673d2cfbe1a352f3c7d27c1-dirty

runc --version
runc version 1.0.0-rc10 commit: 4791ca3d31174576ddd8f4ff562283339196efa9 spec: 1.0.1-dev

rpm -qa | grep containers
containers-common-1.2.0-3.el7.x86_64

podman version
Version: 2.1.1 API Version: 2.0.0 Go Version: go1.15.2 Built: Wed Oct 28 07:37:02 2020 OS/Arch: linux/amd64

Results obtained
This container's log file is over 800MB:
ls -lah /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
-rw-------. 1 root root 839M Jun 1 14:05 /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log

I can see that the --log-size-max option is applied with the right value by conmon.
ps -ef | grep conmon | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log root 23953 1 0 14:04 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -u 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata -p /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/pidfile -n fluent-sd-pod-fluent-sd --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log --log-size-max 209715200 --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7

Container has been restarted recently:
date
Tue Jun 1 14:36:06 CDT 2021

ps -eo pid,lstart,cmd | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
17386 Tue Jun 1 14:32:21 2021 <...>

Results expected
Container log file size not to exceed 200MB.

@pascapascapascapasca pascapascapascapasca changed the title Max log size not applied Max log size not applied when container restarts Jun 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant