Failed to start container: Volume atlantis-disk-0: Filesystem check failed

[tpolekhin]

Trying to deploy Atlantis on GCE using IAP example.
Container is failing to start because of the filesystem error:

[   70.188299] konlet-startup[1686]: 2023/03/13 14:05:19 Attempting to unmount device /dev/sdb at /mnt/disks/gce-containers-mounts/gce-persistent-disks/atlantis-disk-0.
[   70.190423] konlet-startup[1686]: 2023/03/13 14:05:19 Unmounted /mnt/disks/gce-containers-mounts/gce-persistent-disks/atlantis-disk-0
[   70.190530] konlet-startup[1686]: 2023/03/13 14:05:19 Found 1 volume mounts in container  declaration.
[   70.197085] konlet-startup[1686]: 2023/03/13 14:05:19 Running filesystem checker on device /dev/disk/by-id/google-atlantis-disk-0...
[   70.199060] konlet-startup[1686]: 2023/03/13 14:05:19 Error: Failed to start container: Volume atlantis-disk-0: Filesystem check failed: Failed to execute command [fsck.ext4 -p /dev/disk/by-id/google-atlantis-disk-0]: exit status 8, details: /dev/disk/by-id/google-atlantis-disk-0 is mounted.
[   70.199171] konlet-startup[1686]: e2fsck: Cannot continue, aborting.

Also noticed that chown command fails as well:

...
[   17.150286] systemd-networkd[334]: vethdf3fdbb: Gained carrier
[   17.156971] systemd-networkd[334]: docker0: Gained carrier
[   18.323133] systemd-networkd[334]: docker0: Gained IPv6LL
[   18.962881] systemd-networkd[334]: vethdf3fdbb: Gained IPv6LL
[   39.220362] chown[929]: chown: cannot access '/mnt/disks/gce-containers-mounts/gce-persistent-disks/atlantis-disk-0': No such file or directory
[   55.846631] konlet-startup[627]: 2023/03/13 14:05:04 Received ImagePull response: ({"status":"Pulling from runatlantis/atlantis","id":"latest"}
...

[tpolekhin]

After instance restarted the issue is now gone.
I guess it could be a first startup racing condition or something similar.

[bschaatsbergen]

I've encountered this issue before and it seems like that it's a race condition between COS (the konlet startup service that COS uses and the init config we use to mount the persistent disk).

I would recommend to ignore this. Is everything working as expected?

[tpolekhin]

Is everything working as expected?

Yes, Atlantis container started and im able to browse the UI through IAP.
Thanks a lot for this module!

[bschaatsbergen]

Amazing! With pleasure @tpolekhin