Ubuntu systems showing lots of core packages as non-compliant.

[ppanon2022]

I've noticed that a lot of our Ubuntu client systems are now showing a lot of core packages as non-compliant.
On one client system, running apt list --upgradeable (and afterwards apt update && apt upgrade) showed many outstanding updates from the "Non-compliant" list. Forcing a package list update from the Uyuni console doesn't change the result. Applying the updates with apt upgrade on the client made those packages disappear from the non-compliant list. Is anybody else experiencing the same problem?

In our case, I'm not exactly sure what caused it. We skipped upgrading to 2024.01 and went straight from 2023.12 to 2024.02. We also had an issue with some package syncs failing due to needing to increase available storage. So I can't know which of those changes/issues caused the change in behaviour. However I'm not seeing the same problems with rpm based distros like CentOS, OpenSUSE, or Redhat on two different servers. So I'm leaning to thinking it's likely due to a change in the package comparison/matching algorithm for the on-compliant list. Was there any change affecting that in 2024.01 or 2024.02?

This is very concerning because it means needed Ubuntu updates are not being deployed.

[ppanon2022]

I ran a reindex database with Uyuni shut down, Then when I bought it back up, I ran a package list refresh, however that seemed to make no difference in the long list of packages showing up as non-compliant.

[akupreev]

@ppanon2022 Hello. Tell me, did you manage to fix this problem?

[ppanon2022]

Well, we updated to 2024.05 and also have done a couple of patching cycles since, and that seems to have helped. Also, during the upgrade to 2024.05, we noticed (for a different reason) that some spacewalk-backend* packages weren't getting updated and we had to do some work to resolve that (some package dependencies and old deprecated packages that needed cleaning up), so perhaps a version mismatch between the front and back ends was a factor. Or perhaps the schema had changed but some of the queries were from the older backend packages and no longer worked properly after the schema change.

[akupreev]

Good afternoon, my friend.
Do I understand correctly that:

1. You have installed patches on Uyuni Server (zypper patch)
2. Updated all spacewalk-backend* packages - Tell me, what version of the packages are they now?

My Version of Uyuni Server:
Repository : uyuni-server-stable
Name : Uyuni-Server-release
Version : 2024.05-230900.217.1.uyuni3
Arch : x86_64
Vendor : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level : Level 3
Installed Size : 1.4 KiB
Installed : Yes
Status : up-to-date
Source package : Uyuni-Server-release-2024.05-230900.217.1.uyuni3.src
Summary : Uyuni Server

[ppanon2022]

The thing is that Uyuni-Server-release isn't what you need to look at because it said 2024.05 was installed before as well. What you need to check is specifically the version of the spacewalk-backend* packages. They should all be the 5.0.6 version.

# rpm -q -a | fgrep spacewalk-backend | sort
spacewalk-backend-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-app-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-applet-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-iss-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-iss-export-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-package-push-server-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-server-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-sql-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-sql-postgresql-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-tools-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-xml-export-libs-5.0.6-230900.1.4.uyuni3.noarch
spacewalk-backend-xmlrpc-5.0.6-230900.1.4.uyuni3.noarch

If they aren't that release then you need to do a zypper refresh && zypper update and see what packages are being held back. Then you need to zypper install any held back spacewalk-backend packages and resolve any dependency conflicts. See #8937

[ppanon2022]

I think one other thing is that the non-compliant list used to only show packages which did not have newer packages with the same name in any of the assigned channels (i.e. orphaned packages which were either manually copied and installed, or which were installed by adding a PPA on the system without any equivalent in an Uyuni channel/repo to replace it). Perhaps it also included packages which were in an assigned channel but had a newer version than the latest available in the channel - for example, when using LCM projects with a cutoff date filter to ensure package consistency between groups of systems that are used to test updates in dev/sandbox, then QA/staging, prior to production deployment, new systems that are patched as part of an installation that occurred after the last version cutoff date for the channel could have a newer version of some packages issued after the cutoff date. That situation could also be aggravated if you're only applying security updates, but the Ubuntu installer applied all updates, including many bug-fix updates that aren't in the channel. I expect that means what probably helped the most in our case was running a patching cycle after that update to bring systems into consistency with the LCM channels.

Now the non-compliant list also seems to include all the packages which need to be updated. I understand why you would want to highlight down-level (update pending) versions as non-compliant, but it makes it much harder to identify which packages are from a 3rd party PPA with no channel equivalent because that info can be a needle buried in the haystack of packages needing updating. Perhaps there should be an additional tab (perhaps Unsupported?) with contents closer to the older Non-Compliant tab, listing packages for which no subscribed channel has a package of the same name but different version?

[ppanon2022]

It looks like for the exact Name, Versio, Release, Epoc (if exist) of the packages that are installed and the packages that are available o the channels assigned in to the system. In your last example, if the installed package is in a version higher then the one provided in the CLM channel, and the new version is not in the channel, it will show as non-compliant since the exact version is not available.
Correct. That's the previous behaviour and what I expected (although technically, if the package version is newer than what's available in the subscribed channels - because of an install-time general update that included more than security updates - it's arguable it should still be compliant). What I didn't expect is that packages in the Updates list would also show in the Non-Compliant list as well, which appears to be the case now and wasn't a few months ago.

Regarding the package bind9 (for example bind9-dnsutils). From the vendor channel that contains all the packages, can you check all the versions that exists? For base channel and each child channels go to its detail, click on packages, and then filter by bind9-dnsutils, then send back the list for each channel.

From information you have sent, looks like the versions are different:

• bind9-dnsutils-9.18.18-0ubuntu0.22.04.2
• bind9-dnsutils-9.18.24-0ubuntu0.22.04.1:1.amd64-deb

version 9.18.18 <> 9.18.24.

I think that's just because I pulled it from different pages and they display the packages differently but using the Software Package Search results show they're the same arch/naming scheme
Currently the LCM projects only include the security updates channel
bind9-dnsutils-9.18.18-0ubuntu0.22.04.2:1.amd64-deb | ubuntu22-prod-Ubuntu 22.04 LTS AMD64 Main Security for Uyuni
bind9-dnsutils-9.18.1-1ubuntu1:1.amd64-deb | ubuntu22-prod-Ubuntu 22.04 LTS AMD64 Main for Uyuni
bind9-dnsutils-9.18.18-0ubuntu0.22.04.2:1.amd64-deb | ubuntu22-it_baseline-Ubuntu 22.04 LTS AMD64 Main Security for Uyuni
bind9-dnsutils-9.18.1-1ubuntu1:1.amd64-deb | ubuntu22-it_baseline-Ubuntu 22.04 LTS AMD64 Main for Uyuni
bind9-dnsutils-9.18.18-0ubuntu0.22.04.2:1.amd64-deb | ubuntu22-devtest-Ubuntu 22.04 LTS AMD64 Main Security for Uyuni
bind9-dnsutils-9.18.1-1ubuntu1:1.amd64-deb | ubuntu22-devtest-Ubuntu 22.04 LTS AMD64 Main for Uyuni

But the actual sync channels include both the updates and security channels and you can see the naming scheme is the same
bind9-dnsutils-9.18.24-0ubuntu0.22.04.1:1.amd64-deb | Ubuntu 22.04 LTS AMD64 Main Updates for Uyuni
bind9-dnsutils-9.18.18-0ubuntu0.22.04.2:1.amd64-deb | Ubuntu 22.04 LTS AMD64 Main Security for Uyuni
bind9-dnsutils-9.18.1-1ubuntu1:1.amd64-deb | Ubuntu 22.04 LTS AMD64 Main for Uyuni

I did a quick search locally, and it looks like version 9.18.18 is only on channel main-security, and the version 9.18.24 is only on channel main-updates. When you assigned the non-LCM channels did you assign both main and security channels?
Yes, I assigned the updates channel (which implicitly includes all security updates in addition to bug fixes). The point is that update channel includes all the updates that are applied by the Ubuntu installer at update time, to avoid confusion in the Non-Compliant list from newer non-security updates applied at install-time.
Another point is that some vendor only keep the latest version on the channel, and delete older version, this can lead packages to be non compliant if the exact version is not found.

Well, that might be a problem after you mainline the GSOC project to only download metadata and packages that are actually requested for installation. But historically I have a little trouble understanding how that would be an issue. Currently any packages in the metadata get downloaded/synced, and vendors who prune their package repo also prune the metadata. Uyuni also seems to keep packages depending on your sync settings, so even if the vendor purges a package from their repo, it doesn't mean Uyuni would (otherwise it shouldn't take three separate steps - deleting channels, packages and running spacewalk-data-fsck - to clean out an obsolete/EOL OS release).

I suppose if you built a system, then waited a few days or weeks before joining it to Uyuni and in the meantime a new version was released and the older version was deleted, then you could conceivably have that happen. But you would have to have been unlucky enough to have multiple packages released between two daily sync jobs, and get one during installation that got deleted before the next daily sync. Even then I would think the package name/arch/packaging-format would still match even if that specific version wasn't in that repo. It would be a very unlikely use case that could be handled by matching on package name/arch/deb-vs.-rpm whereas that behaviour breaks much more common use cases.

The other thing we're seeing now that we weren't seeing before is that the update list now lists all versions of packages that are newer than the installed version. So, if 3 updates to a package have come out since the last time updates were applied, then all 3 updates now show up in the updates list. I guess the idea may have been so that if there was an intermediate version that included a security patch/CVE fix, you wouldn't miss seeing that implicitly included patch because only the latest version without that association was being displayed. The problem is that when you display all those versions, the admin can no longer easily choose to apply all those updates (and perhaps decide to hold back one or two in exceptional cases) because the client complains about duplicate packages. Instead, what should happen is that the latest package version should be associated (through a subquery) with all the CVE patches for versions that were released after the currently installed version. OK, that is somewhat more complicated since the selected systems may have a range of installed versions (you could still make it work by using the oldest version among the selected systems as your boundary condition), and you have to figure out some way to coalesce the associated patch results if there are multiple newer patches. More simply if there are multiple CVE patches from the different newer versions, you could just list the latest one. But it's still better than the current results where the patches will show up almost randomly associated with different versions for different packages when you have multiple versions of binary packages built from a single source package. The same patch can get listed in the update list with one version of a package and a different version of a fraternal package. It's a mess.

I suppose that the new scheduled auto-update feature is supposed alleviate the extra admin overhead and risk of human error in selecting packages, but that doesn't look like it would work on scenarios when you're using LCM projects in conjunction with more complicated maintenance schedules. The current update listing with multiple newer versions of a package are a step backwards as far as I'm concerned. Anything you select other than the latest version is just going to show as still needing to be updated after it's installed, so it doesn't make sense to show them as an update option to the administrator if they don't resolve the entry as a package needing an update.

[rjmateus]

We will look into this, since as far as I know the package upgrade list should show only the latest version available on the channels.

[ppanon2022]

Here's the list of held back packages (haven't done a zypper refresh in a while). Most of the held back packages appear to have newer versions in the OS repos than in the Uyuni repo so I figured it made more sense to stick with the Uyuni repo versions. This is still running 2024.05. We haven't upgraded to 2024.07 yet.

ca07uyuni1:/srv/www/htdocs/pub/repositories/el/8/repodata # zypper update
Loading repository data...
Warning: Repository 'Update repository of openSUSE Backports' metadata expired since 2024-07-18 13:01:53 EDT.
Warning: Repository 'Main Update Repository' metadata expired since 2024-07-03 04:46:10 EDT.

    Warning: Repository metadata expired: Check if 'autorefresh' is turned on (zypper lr), otherwise
    manually refresh the repository (zypper ref). If this does not solve the issue, it could be that
    you are using a broken mirror or the server has actually discontinued to support the repository.

Reading installed packages...

The following 17 package updates will NOT be installed:
  apache-commons-csv apache-commons-ognl byte-buddy classmate cobbler concurrentlinkedhashmap-lru glassfish-fastinfoset glassfish-jaxb-runtime
  glassfish-jaxb-txw2 istack-commons-runtime jackson-module-jaxb-annotations jandex mybatis reprepro stax-ex xmlstreambuffer xsom

Hmm, I see that with 2024.07 repo-sync has changed to be strict sync and clear out old packages that are no longer in the Internet vendor masters. That is presumably why you wrote about packages disappearing when vendors only keep the latest version. Since we're still running 2024.05, that hasn't been a factor. That does bring up a question though, when the new repo-sync removes a package from the channel because it's been removed from the vendor's master repo, does the file get removed as if spacewalk-data-fsck had been run?

[ppanon2022]

Hmm. I thought that a 2024.08 release wasn't listed in the July community hours roadmap for releases and migration.

[ppanon2022]

Well, this whole discussion is now moot due to the switch to containerized Uyuni