You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The max log size option is set to 200MB through log_size_max = 209715200 in /etc/containers/containers.conf (no other option is uncommented in it), but the container logs exceed 200MB. This seems to happen when a container restarts and thus the conmon process is restarted. In my case, podman restarts the container every few minutes and the bytes written by the previous conmon process do not seem to be taken in count when applying the max log size.
Versions CentOS 7
conmon --version conmon version 2.0.21 commit: efe288263ebb1bab4673d2cfbe1a352f3c7d27c1-dirty
runc --version runc version 1.0.0-rc10 commit: 4791ca3d31174576ddd8f4ff562283339196efa9 spec: 1.0.1-dev
podman version Version: 2.1.1 API Version: 2.0.0 Go Version: go1.15.2 Built: Wed Oct 28 07:37:02 2020 OS/Arch: linux/amd64
Results obtained
This container's log file is over 800MB: ls -lah /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log -rw-------. 1 root root 839M Jun 1 14:05 /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
I can see that the --log-size-max option is applied with the right value by conmon. ps -ef | grep conmon | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log root 23953 1 0 14:04 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -u 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata -p /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/pidfile -n fluent-sd-pod-fluent-sd --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log --log-size-max 209715200 --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7
Container has been restarted recently: date Tue Jun 1 14:36:06 CDT 2021
The max log size option is set to 200MB through
log_size_max = 209715200
in /etc/containers/containers.conf (no other option is uncommented in it), but the container logs exceed 200MB. This seems to happen when a container restarts and thus the conmon process is restarted. In my case, podman restarts the container every few minutes and the bytes written by the previous conmon process do not seem to be taken in count when applying the max log size.Versions
CentOS 7
conmon --version
conmon version 2.0.21 commit: efe288263ebb1bab4673d2cfbe1a352f3c7d27c1-dirty
runc --version
runc version 1.0.0-rc10 commit: 4791ca3d31174576ddd8f4ff562283339196efa9 spec: 1.0.1-dev
rpm -qa | grep containers
containers-common-1.2.0-3.el7.x86_64
podman version
Version: 2.1.1 API Version: 2.0.0 Go Version: go1.15.2 Built: Wed Oct 28 07:37:02 2020 OS/Arch: linux/amd64
Results obtained
This container's log file is over 800MB:
ls -lah /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
-rw-------. 1 root root 839M Jun 1 14:05 /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
I can see that the --log-size-max option is applied with the right value by conmon.
ps -ef | grep conmon | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log root 23953 1 0 14:04 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -u 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata -p /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/pidfile -n fluent-sd-pod-fluent-sd --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log --log-size-max 209715200 --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7
Container has been restarted recently:
date
Tue Jun 1 14:36:06 CDT 2021
ps -eo pid,lstart,cmd | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
17386 Tue Jun 1 14:32:21 2021 <...>
Results expected
Container log file size not to exceed 200MB.
The text was updated successfully, but these errors were encountered: