Add support for configuration files (and multi local repositories)

[Noeljunior]

This PR adds a global configuration file for: run user, database name, database path, makepkg.conf path, pacman.conf path, if to chroot, chroot path and if to update.
It also adds the possibility to configure multiple repositories with all of these configuration per repository except the run user which is global.

I came with this idea mainly to cross-compile to i686 and aarch64 (alarm) on the same machine without a too messy setup. For that, each repo had to have its own makepkg.conf and pacman.conf as well different database names and paths and chroot paths, hence all these new configurations.

The idea turned out to be way more complex than what I thought when I started this. I mean, I modified a lot more of what I thought it would be needed and that may break the idea you have for aurto.

This PR is not 100% ready for a release, in my opinion. I'm more than happy yo make it ready, regarding my ideas, after you accept this new paradigm. If you believe this is out of aurto purpose, then I just keep a fork for myself (and whoever wants to use it).

--

Note: I have an updated version of aurto.install to be included with this which handles all the modifications I did.

More details about what I did:

• added /etc/aurto/aurto.conf as main configuration file, ini-like format:
  • non-sectioned: global settings loaded as bash variables
  • each section: a repo-sepecific settings loaded as bash arrays (CFG0[], CFG1[], ...)
• /usr/lib/aurto/user and /usr/lib/aurto/conf-disable-chroot are now part of aurto.conf settings
• aurto usage is now: aurto add|addpkg|remove REPO PACKAGES...
• added $lib_dir to local PATH (to help readability)
• the repo is initialized during run time (aurto add) instead of on aurto.install script
• aurto asks the user about adding or not the repo to /etc/pacman.d/aurto
  • you may want run this on a server and you don't need all the repos on the build machine
  • OR you may want to cross compile to aarch64 and is useless (and even erratic) to add it on x86_64 machine
• renamed makepkg-aurto.conf to makepkg-x86_64-aurto.conf
• added pacman-extra-aurto.conf copied from devtools' files

Still needs to be done before a new release:

• some code clean up and comments, mainly on aurto.conf
• shells' auto-completes
• have a look at update-aurto VCS specific code
  • it looks to me you are using old commands and I can't get info about what they did so I couldn't update it:
    • aur sync <pkgs> --no-ver-shallow --print and aur vercmp-devel
• update aurto.install in AUR package

Things I hope to do soon but probably out of this PR:

• signing packages settings
• if to setarch or not on aurutils calls
  • this will be helpfull for cross-compilations
• per-repo update interval
  • and in that way, one could easily setup two repos (one being for VCS pkgs) and customize its interval
• if to delete package or not if not found in AUR anymore
• verify loaded config file for bad settings (and normalize paths)

Suggestions/discuss regarding these modifications:

• I did not modified much of your verbose nor added more because I didn't get how much you want to verbose
  • maybe a verbose option would be great
• ask the user to initialize a new repo? I think no
• [PKGBUILD] make devtools as optional dependency (as is on aurtutils)

Suggestions independent of this PR I might implement:

• make summerize-build a function on shared-functions
• make this package true archicheture independent
  • now it defaults to the provided makepkg.conf which will only work on a x86_64 machine
  • we can provide more archs and select the default the one during installation OR make the makepkg.conf arch-independent (somehow)
• no root / full root:
  • run aurto all userland (no root needed) except for special settings as add a repo to local pacman.conf (and abort aurto if run as root)
  • run update-aurto as user set by systemd service like [email protected]
  • however, this would be mean (for me) that all settings should be saved on that user's home
  • OR keep the settings as root, run all as root, let user be used to run what needs to be run without root but don't use any user's home space
    • this is good if one wants to create a no-home user as the user for building packages

--

There's a lot here non relevant to this PR itself, I know. But I got into a point where either I implement things with the merge in mind, or if you don't like this I just don't need to care about much of the work. I would prefer to merge, though, and keep contributing.

I'll implement all of these things eventually. I just would like to know if I keep the merge in mind or not ;)

Feel free to discuss anything I said here.

And, of course, thanks for your work so far!

cheers

[Noeljunior]

Here: I don't understand what you are trying to do. Can you please help me make this clear?

[alexheretic]

This is checking *-git packages for updates, which happens once daily triggered by the check-vcs file generated by check-aurto-git-trigger.timer.

The "init vsc sync cache" bit ensures the .cache/aurutils/sync/ cache exists for the vcs packages (aurto generally kills this cache regularly to save space).

aur vercmp-devel will then list outdated VCS packages which are rebuild into the aurto repo by removing and re-adding them. It looks like this will change in aurutils 3.0, I'll figure something out then I guess.

[alexheretic]

Thanks for the pr! There are some good ideas you're working towards here.

However, I don't really consider multiple repos or archs in scope for aurto. That's because my vision for aurto is a simple and opinionated utility that doesn't try to solve 100% of use cases.

If extra functionality was optional, so didn't affect 90% of users at all that would be more acceptable.

But then, even if that could be designed, I'd start wondering if the code complexity is really too much to be written in untested bash. If aurto was to take on more complexity I'd probably look to moving those bits to a more appropriate language, like I did with trust-check. Maybe it's worth moving the whole project into rust?

I can sympathise & imagine that you may not agree with me here. In which case, of course, you are very welcome to fork the current code and morph it to fit your needs.

For my branch I'd currently say

• No to multi repo / arch for simplicity.
• Yes to a config file, but maybe not parsed in custom bash, related to Migrate away from bash #38.
• Yes to replacing the install script, see Replace install script with aurto init #35.
• Yes to improving the root / user placement of aurto, Improve root & user separation #37.
• Yes to 'signing packages settings', if it was optional config that didn't affect simple usage.
• Probably no to 'if to delete package or not if not found in AUR anymore' as it only makes sense to delete it from the aurto repo to me. The aurto repo is meant to be an auto-maintained view on AUR, I want pacman -Qm to work here. Raise an issue if you want to talk about that one more.

[Noeljunior]

Thanks for reading this!

Let me try one more time to show you how this could be usefull yet keeping the simplicity you want for aurto.

I think aur lacks good enough software to manage from simple use cases to more complex ones. I liked aurto because it uses aurutils which seems to be solid enough regarding aur building proccess but, again, not simple enough as aurto so one can easily install a program. So, I get the idea of simplicity of aurto. I would like to explore this a bit more with you so we can keep both your simplicity ideal and my needed features.

Actually I tried to keep the same code style as you did to keep things consistent. I never used rust before but if you believe it is better for this I can have a look. If if had to choose another language, I would go for python. However I prefer bash for this kind of tools and for better integration with aurutils. In meanwhile I came into a point where I imagine this could be either integrated (merged) with aurutils or a kind of a plugin, hence the bash.

I totally agree to keep every new features seamless to the current user base. I think the only thing not in that direction is the new usage. I suggest --repo=repo and if no repo provided, take the first, or a global defined one, as default

So, every new "complex" feature would be off by default.

If you still prefer to opt out these new concepts, it's fine. I have some ideas for your current problems which may help you and I'm happy to share them with you.

[alexheretic]

Ok sure, I wouldn't mind --repo args with defaults.

I'll convert the project into a rust binary (I'd favour rust) so we can take a little more complexity with tests etc.

Before that we can still think about the design, moving stuff into the home folder etc.

[Noeljunior]

Ok, thanks.
I'll have a look at your issues and comment what I think I can contribute after the conversion to rust.

[Titaniumtown]

I'm looking forward to fixes being made and this PR being pushed to the master branch!

[Titaniumtown]

I've been doing some testing of this PR on my repo for aarch64 and raspi4 related things on my raspi4. And I have some feedback.

1. The repo name HAS to be aurto when you run a command:
   1a. aurto add: my repo is called "raspi_aarch64", but say I run aurto add raspi_aarch64 gzip-git, it fails: aurto: Repo not found: raspi_aarch64. But, if I run aurto add aurto gzip-git, it runs fine.
   1b. aurto status: if I run either aurto status aurto or aurto status aurto add raspi_aarch64 gzip-git the "pacman -Sl" part always looks for aurto.
2. Cross compiling: I am using bedrock linux on pretty much everything I have (check them out: https://bedrocklinux.org) and one thing they let you do is install multiple distros on the same pc, and switch between them when ever. Another thing that the devs added is cross-arch support, like I can have a aarch64 stratum (that's what a distro under bedrock linux is called) on a x86_64 pc, or a x86_64 stratum on a raspberry pi, it uses qemu-user-static for that. So could support be added to not use a cross compiler, and use a stratum of a different architecture to compile for a different architecture? This could even just replace using a chroot in the first place.

As I can't think of anymore feedback I can give you, I think that's about it. Keep up the good work! :)

[Noeljunior]

Thanks Titaniumtown for your feedback! Please keep in my this PR is just an idea and not a ready-to-go implementation of whatever.

1a. As I did not care for documentation at this point, it's you cannot run aurto add repo_name pkg_name out of the box. I did a proper PKGBUILD and install script for this to upgrade from the current structure to the one I purpose. You can find them here
Assuming you did a proper install, you should now have a /etc/aurto/aurto.conf file. If you keep everything untouched, you should have the exactly configs and behaviour of the current aurto, except for the aurto command **REPO** pkg. You must now edit /etc/aurto/aurto.conf and add another repo, say raspi_aarch64, your whatever settings you want. You can now use the repo. If the chroot does not exits it will create one. If the database does not exists it will create one.

1b. I did not even touch in aurto status code, so don't count on it.

2. Cross compiling to aarch64 was the trigger that made me do this so I do have interest on your feedback here. I'm afraid I have no time to test this out of a standard arch linux distro so please either test it on a arch linux installation (or chroot) or assume everything works on your bedrock.
   I'm not sure if I understood what is you idea. With this PR you should be able to prepare your clean chroot, whatever the architecture it is, as soon as you arch-nspawn into them.
   Does this help? Please let me know.

[Titaniumtown]

@Noeljunior My idea was to use a stratum in bedrock linux, that already implements running everything (non-native architecture wise) through qemu, and use that to manage the different build environments, it would be much simpler. Another benefit of this route is being able to easily install new ones, for instance, say I wanted to install a aarch64 arch linux arm install: sudo brl fetch -a aarch64 arch-arm -n arch-arm-aarch64 creating a stratum named arch-arm-aarch64. This could be expanded to more than one statum.

I was thinking a section in the config file could look like this:

#Uncomment to enable using bedrock linux's stratum with cross compiling:
#bedrock=yes

Arch: aarch64
Statum name: arch-arm-aarch64

Arch: x86_64
Statum name: arch-x86_64

Sorry if I didn't make my idea super clear.

[Noeljunior]

@Titaniumtown as I told you:

• make sure you have a proper installation as in PKGBUILD I shared before
• set chroot path to your aarch64, statum or not
• make sure you can arch-nspawn into it (qemu static is fine)
• IF the chroot is not empty, get a proper pacman.conf with mirrors pointing to aarch64 packages
• get a proper makepkg.conf with whatever you need.

With this approach, aurto is architecture agnostic so no need to do custom code.

Keep in mind that devtools (the arch linux scripts that build the packages inside a chroot) expects a clean chroot, will update the chroot and will copy it every time you build a package.

If you find any problem with this approach let me know.

[Titaniumtown]

Will there be further work on this? this PR has been inactive for a while now.

[Noeljunior]

alexheretic prefers to consider the merge after aurto is rewritten in rust. I have no special interest in rust language and, as such, I don't find myself time to contribute on the translation.
However, if the translation happens, for sure I'll be ready to adapt this PR to rust.
I might consider a fork of the aurto with this and more as a feature development and/or testing new things. I doubt I will ever publish it on AUR or encourage anyone to use on a daily basis as I will not implemente any tests. Again, if aurto may move to a more robust language, there's no point of keep adding features in another language.

[Titaniumtown]

Is this ever going to get updated? I would really like it to! It's been ages lmao

[Noeljunior]

Is this ever going to get updated? I would really like it to! It's been ages lmao

Again, @Titaniumtown. @alexheretic has no interest in these features if they are written in bash so no point further development.

If you, @Titaniumtown, are up to maintaining an up to date fork of aurto I'm happy to adapt this to the current state of aurto. That should also help to debug and, when time comes, translate it to rust.