architecture detection broken with recent »file« releases

[Optiligence]

Unable to guess the architecture of the AppDir source directory "."

AppImageKit/src/appimagetool.c

Line 357 in 5b5ca1a

extract_arch_from_text(g_strsplit_set(line, ",", -1)[1], archfile, archs);

breaks because there is no comma before the arch

~> file -L -N -b /usr/lib/libc-2.27.so
ELF 64-bit LSB pie executable x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /usr/lib/ld-linux-x86-64.so.2, BuildID[sha1]=0c1ecb7b0e67918d1d1991e0e7793f814bfa1ec2, for GNU/Linux 3.2.0, not stripped

using file-5.33

[probonopd]

I am getting

me@host:~$  file -L -N -b /usr/lib/liblilv-0.so.0
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=544dc36f9301a8430783950fcb13334feee83488, stripped

I don't think we can handle ELF 64-bit LSB pie executable x86-64, whatever that is. Probably the oldest still-supported target systems (distributions) can't run PIE executables anyway.

[Optiligence]

it’s the default output for libraries on Arch Linux
may be a file issue too
what’s your file version?

[probonopd]

me@host:~$ file --version
file-5.32
magic file from /etc/magic:/usr/share/misc/magic

[probonopd]

Please be aware of https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages#creating-appimages-that-are-compatible-with-many-systems.

[Optiligence]

i’m aware

it seems to be a change in file interpretation/output. same library with 5.32:

~> file -L -N -b /usr/lib/libc-2.27.so
ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /usr/lib/ld-linux-x86-64.so.2, BuildID[sha1]=0c1ecb7b0e67918d1d1991e0e7793f814bfa1ec2, for GNU/Linux 3.2.0, not stripped

vs the 1. post

fyi an executable looks like this with 5.33:

~> file -L -N -b /usr/bin/g++
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=b9713beeb112ce4a90382a69926cbe1cf2f8c41f, stripped

[probonopd]

Uh, that's interesting. So possibly a bug should be reported against file since sudden changes in its output format will upset many tools that are relying on it.

[TheAssassin]

@probonopd as I found out recently the file command and libmagic can be highly annoying to use when dealing with "newer" distros during the development of linuxdeploy (e.g., Ubuntu bionic).

First of all, it seems like older libmagic binaries can't read newer magic files properly, which means libmagic returns the "random binary file" type for everything (like ELF files). This renders libmagic completely unusable for my purposes.

Next, the filecommand's output is based on whatever the magic database contains. That means that you can't rely on its output, even if you bundle the file binary.

I think we have to bundle the magic database in order to work around these limitations. This guarantees the correct output on every system. I already thought about developing a linuxdeploy plugin for the purpose. This plugin should ideally patch the libmagic binary to make sure the magic database is loaded relative to the library, as otherwise, one would have to use e.g., environment variables, to load the bundled database by default.

[TheAssassin]

@Optiligence can you try file --mime, please? This should yield a MIME type instead of a textual description, which is probably a more consistent output, and easier to parse, too.

[Optiligence]

file-5.33

~> file --mime /usr/lib/libc-2.27.so
/usr/lib/libc-2.27.so: application/x-pie-executable; charset=binary

[probonopd]

Uh, that's not even telling us the architecture at all. Maybe we should not use the file command nor the magic database at all but determine the architecture directly from the ELF header? I mean, we do our fair share of ELF header parsing already in this project ;-)

[TheAssassin]

That'd be better, yes. See https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/elf.c#L135.

[Optiligence]

the relevant file commits, for reference:
introduction: file/file@6876eba
fix: file/file@defcf7e
so it’s 5.33 only

[TheAssassin]

@Optiligence the best solution would be to get rid of file or any sort of magic database for detecting ELF files. We can just replicate this in our source code, the ELF format is very well defined, there's magic bytes, e.g.,

> xxd /bin/bash | head -n1
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............

Furthermore, there's a flag defining the architecture which can simply be read, our ELF code is even capable of this. See https://github.com/AppImage/AppImageKit/blob/appimagetool/master/src/elf.c#L135

[Optiligence]

Sure. but since it also seemed like a file bug, i wanted to check that out, but their bug tracker and mailing list links were down, so i sifted through the commits.

It’s just a confirmation that it is, indeed, a bug, and that it’s also fixed already.

For the time being i’ll set the environment variable ARCH=x86_64as the appimage output helpfully suggested.

[TheAssassin]

Adding this issue to my current TODO list.

[probonopd]

introduction: file/file@6876eba
fix: file/file@defcf7e
so it’s 5.33 only

So a bug in file has been introduced, and has already been fixed. Unless 5.33 happens to land in, say, a Ubuntu LTS release I'd say we can safely ignore this.

[TheAssassin]

But why bother using file at all? This subprocessing stuff can be kicked out entirely, making appimagetool faster and less error prone.

[probonopd]

But why bother using file at all?

Because it is working (besides one bug in one version of it that already has been fixed by them)? I prefer not to touch what we already have working, and focus on the things that we either don't have yet or are not working properly yet (e.g., appimaged is still placing desktop files into the filesystem one after each other rather than all at once, and so on - there our time makes a difference).

[TheAssassin]

Please close this issue as wontfix then.