UDM 1.11.0-14 (EA) update broke everything #232

vansummeren · 2021-08-25T08:52:15Z

vansummeren
Aug 25, 2021

Last night, a new early access version of the Unifi-OS for my Dream Machine Pro was released and automatically installed.
(I know, my bad. Turned the auto update off now.)

(The only info about the release I found on reddit: https://www.reddit.com/r/Ubiquiti/comments/pat57d/unifi_os_for_dream_machines_111014_early_access/)

Unfortunately, this release broke the container execution on the DMPro.
My pihole container was no running anymore and when I sshed into the DMPro, I was not even able to start the "unifi-os shell", see below.

I was able to revert back to the 1.10.0 release, everything started working again immediately.
ubnt-upgrade https://fw-download.ubnt.com/data/udm/210e-udmpro-1.10.0-6cc49445340f4896959ab66f6f1b8da3.bin

# unifi-os shell
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus: open /sys/fs/cgroup/libpod_parent/conmon/cpuset.cpus.effective: no such file or directory
Error: OCI runtime error: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02"

goroutine 1 [running]:
github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x40001f7050, 0x5558c44900, 0x0)
        github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001a9dc0, 0x40001f7050, 0x4a, 0x7fd64b5edf)
        github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x400017a090, 0x7fd64b5edf, 0x40, 0x0, 0x0, 0x1, 0x8)
        github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
main.getContainer(0x400012c6e0, 0x5558c5cb14, 0x8, 0x1, 0x400019b3bd)
        github.com/opencontainers/runc/utils_linux.go:89 +0x9c
main.execProcess(0x400012c6e0, 0x0, 0x0, 0x0)
        github.com/opencontainers/runc/exec.go:114 +0x30
main.glob..func5(0x400012c6e0, 0x5559068d80, 0x400019b518)
        github.com/opencontainers/runc/exec.go:104 +0x6c
github.com/urfave/cli.HandleAction(0x5558d5b560, 0x5558dfe070, 0x400012c6e0, 0x400012c6e0, 0x0)
        github.com/urfave/[email protected]/app.go:523 +0x124
github.com/urfave/cli.Command.Run(0x5558c58c4c, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5558c6fb5b, 0x28, 0x0, ...)
        github.com/urfave/[email protected]/command.go:174 +0x408
github.com/urfave/cli.(*App).Run(0x4000140540, 0x40000200e0, 0xe, 0xe, 0x0, 0x0)
        github.com/urfave/[email protected]/app.go:276 +0x588
main.main()
        github.com/opencontainers/runc/main.go:165 +0xa3c

# unifi-os restart
unifi-os: Stopping unifi-os
unifi-os: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02"
unifi-os: goroutine 1 [running]:
unifi-os: github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x40002528a0, 0x55685df900, 0x0)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
unifi-os: github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40000e17a0, 0x40002528a0, 0x4a, 0x7ff6ac7f95)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
unifi-os: github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x4000208000, 0x7ff6ac7f95, 0x40, 0x0, 0x0, 0x55687aab00, 0x5568a03d80)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
unifi-os: main.getContainer(0x4000206160, 0x2, 0x2, 0x0, 0x0)
unifi-os:       github.com/opencontainers/runc/utils_linux.go:89 +0x9c
unifi-os: main.glob..func7(0x4000206160, 0x5568a03d80, 0x40000d3518)
unifi-os:       github.com/opencontainers/runc/kill.go:40 +0x78
unifi-os: github.com/urfave/cli.HandleAction(0x55686f6560, 0x5568799080, 0x4000206160, 0x4000206160, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:523 +0x124
unifi-os: github.com/urfave/cli.Command.Run(0x55685f3d38, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x556861217e, 0x52, 0x0, ...)
unifi-os:       github.com/urfave/[email protected]/command.go:174 +0x408
unifi-os: github.com/urfave/cli.(*App).Run(0x40000bc000, 0x4000132000, 0x4, 0x4, 0x0, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:276 +0x588
unifi-os: main.main()
unifi-os:       github.com/opencontainers/runc/main.go:165 +0xa3c
unifi-os: Error: timed out waiting for file /var/run/libpod/exits/18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02: internal libpod error

# unifi-os start
unifi-os: Starting unifi-os
unifi-os: Stopping unifi-os SSH daemon... OK
unifi-os: Starting unifi-os SSH daemon... OK
unifi-os: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02"
unifi-os: goroutine 1 [running]:
unifi-os: github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x4000240ae0, 0x5594062900, 0x0)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
unifi-os: github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001d57a0, 0x4000240ae0, 0x4a, 0x7fc9ef4f98)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
unifi-os: github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x40001ae000, 0x7fc9ef4f98, 0x40, 0x0, 0x0, 0x5594076f80, 0x4)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
unifi-os: main.getContainer(0x40001fe160, 0x5, 0x4000188900, 0x1, 0x0)
unifi-os:       github.com/opencontainers/runc/utils_linux.go:89 +0x9c
unifi-os: main.glob..func3(0x40001fe160, 0x5594486d80, 0x40001c5518)
unifi-os:       github.com/opencontainers/runc/delete.go:56 +0xa8
unifi-os: github.com/urfave/cli.HandleAction(0x5594179560, 0x559421c050, 0x40001fe160, 0x40001fe160, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:523 +0x124
unifi-os: github.com/urfave/cli.Command.Run(0x55940784bd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5594094be6, 0x4d, 0x0, ...)
unifi-os:       github.com/urfave/[email protected]/command.go:174 +0x408
unifi-os: github.com/urfave/cli.(*App).Run(0x40001f8000, 0x4000020080, 0x4, 0x4, 0x0, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:276 +0x588
unifi-os: main.main()
unifi-os:       github.com/opencontainers/runc/main.go:165 +0xa3c
unifi-os: panic: expected "name=systemd" path to be unified path "/sys/fs/cgroup/devices/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02", got "/sys/fs/cgroup/systemd/libpod_parent/libpod-18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02"
unifi-os: goroutine 1 [running]:
unifi-os: github.com/opencontainers/runc/libcontainer.getUnifiedPath(0x400024cc60, 0x555b50d900, 0x0)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:59 +0x2cc
unifi-os: github.com/opencontainers/runc/libcontainer.cgroupfs2.func1(0x40001e1730, 0x400024cc60, 0x4a, 0x7fe5af6f98)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:111 +0x30
unifi-os: github.com/opencontainers/runc/libcontainer.(*LinuxFactory).Load(0x4000208000, 0x7fe5af6f98, 0x40, 0x0, 0x0, 0x555b521f80, 0x4)
unifi-os:       github.com/opencontainers/runc/libcontainer/factory_linux.go:321 +0x14c
unifi-os: main.getContainer(0x4000206160, 0x5, 0x400018e9c0, 0x1, 0x0)
unifi-os:       github.com/opencontainers/runc/utils_linux.go:89 +0x9c
unifi-os: main.glob..func3(0x4000206160, 0x555b931d80, 0x40001d3518)
unifi-os:       github.com/opencontainers/runc/delete.go:56 +0xa8
unifi-os: github.com/urfave/cli.HandleAction(0x555b624560, 0x555b6c7050, 0x4000206160, 0x4000206160, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:523 +0x124
unifi-os: github.com/urfave/cli.Command.Run(0x555b5234bd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x555b53fbe6, 0x4d, 0x0, ...)
unifi-os:       github.com/urfave/[email protected]/command.go:174 +0x408
unifi-os: github.com/urfave/cli.(*App).Run(0x40001bc000, 0x40000b8000, 0x4, 0x4, 0x0, 0x0)
unifi-os:       github.com/urfave/[email protected]/app.go:276 +0x588
unifi-os: main.main()
unifi-os:       github.com/opencontainers/runc/main.go:165 +0xa3c
unifi-os: time="2021-08-25T10:09:35+02:00" level=error msg="error cleaning up container 18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02: error removing container 18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02 from runtime: `/usr/bin/runc delete --force 18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02` failed: exit status 2"
unifi-os: Error: unable to start container "18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02": error removing container 18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02 from runtime: `/usr/bin/runc delete --force 18e2583686d3a108fd54730a8a4b45a4e3116ffe3b0d2e8b71f9ff83efbf7d02` failed: exit status 2
unifi-os: Starting unifi-os attempt 1 failed
unifi-os: podman commit unifi-os unifi-os:current
unifi-os: Getting image source signatures
unifi-os: Copying blob sha256:32225456de31ccc6f123cbf8ae23de1f7be8f882b750eb5e27dbf999a7961b65
unifi-os: Copying blob sha256:52442be05b6c71fe8a82eeaf42ebc94a6024860893f095a2f61be16d48071c53
unifi-os: Copying config sha256:eb0850329bca12b175d845c0fe87af3040e45b805259b65c8f54e602b4607f06
unifi-os: Writing manifest to image destination
unifi-os: Storing signatures
unifi-os: eb0850329bca12b175d845c0fe87af3040e45b805259b65c8f54e602b4607f06
unifi-os: podman create -d -h ubnt --add-host ubnt:127.0.1.1 --name unifi-os --label version=v1.11.16 --network=host --privileged --stop-timeout=60 --dns=127.0.0.1 --mount type=bind,source=/usr/lib/version,target=/usr/lib/version,ro=true --mount type=bind,source=/etc,target=/etc_host,ro=true --mount type=bind,source=/dev/disk,target=/dev/disk,ro=true --mount type=bind,source=/mnt/data/ssh/id_rsa,target=/root/.ssh/id_rsa,ro=true --mount type=bind,source=/var/run/ssh_proxy_port,target=/etc/unifi-os/ssh_proxy_port,ro=true --mount type=bind,source=/mnt/data/unifi-os,target=/data --mount type=bind,source=/tmp/ucore-dropin,target=/etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf,ro=true --mount type=bind,source=/mnt/persistent,target=/mnt/persistent unifi-os:current
unifi-os: Error: unknown shorthand flag: 'd' in -d
unifi-os: Starting container from unifi-os:current failed, falling back to default
unifi-os: podman create -d -h ubnt --add-host ubnt:127.0.1.1 --name unifi-os --label version=v1.11.16 --network=host --privileged --stop-timeout=60 --dns=127.0.0.1 --mount type=bind,source=/usr/lib/version,target=/usr/lib/version,ro=true --mount type=bind,source=/etc,target=/etc_host,ro=true --mount type=bind,source=/dev/disk,target=/dev/disk,ro=true --mount type=bind,source=/mnt/data/ssh/id_rsa,target=/root/.ssh/id_rsa,ro=true --mount type=bind,source=/var/run/ssh_proxy_port,target=/etc/unifi-os/ssh_proxy_port,ro=true --mount type=bind,source=/mnt/data/unifi-os,target=/data --mount type=bind,source=/tmp/ucore-dropin,target=/etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf,ro=true --mount type=bind,source=/mnt/persistent,target=/mnt/persistent unifi-os:default
unifi-os: Error: unknown shorthand flag: 'd' in -d
unifi-os: Starting container from unifi-os:default failed
unifi-os: Error: no container with name or ID "unifi-os" found: no such container
unifi-os: Starting unifi-os attempt 2 failed

risdenk · 2021-08-25T14:46:38Z

risdenk
Aug 25, 2021

FWIW commenting out the podman upgrade script on_boot fixed this for me. I don't know why the new podman has issues. I also noticed that cgroups and memory seemed to not work when upgrading. I had to remove --memory from a few podman run scripts.

The error about memory and cgroups when using the default podman version 1.6.1 is:

Error: could not find cgroup mount in "/proc/self/cgroup"

0 replies

risdenk · 2021-08-25T19:11:05Z

risdenk
Aug 25, 2021

Also reported in #233

1 reply

gtrabanco Aug 26, 2021

I published in #232 report, a temporary workaround but it not solves the fix, just avoid the update to have all containers working.

Maybe this note is useful for anyone who just want to keep newer version and working containers in the UDM.

boostchicken · 2021-08-26T10:58:41Z

boostchicken
Aug 26, 2021
Maintainer

On 11 they did some fucked up shit. I am trying to unravel it, but if you want Podman Updates DO NOT move to 1.11

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UniFi Utilities

UDM 1.11.0-14 (EA) update broke everything #232

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

UniFi Utilities

UDM 1.11.0-14 (EA) update broke everything #232

vansummeren Aug 25, 2021

Replies: 3 comments · 1 reply

risdenk Aug 25, 2021

risdenk Aug 25, 2021

gtrabanco Aug 26, 2021

boostchicken Aug 26, 2021 Maintainer

vansummeren
Aug 25, 2021

Replies: 3 comments 1 reply

risdenk
Aug 25, 2021

risdenk
Aug 25, 2021

boostchicken
Aug 26, 2021
Maintainer