make check

[MarcusGDaniels]

Hi,

I'm getting some problems when I build flux-core from github.

PASS: test_plugin.t 106 - flux_plugin_call (test.foo) success
FAIL: test_plugin.t 107 - flux_plugin_args_unpack result
FAIL: test_plugin.t 108 - call of test.foo set result foo
PASS: test_plugin.t 109 - flux_plugin_arg_out works
PASS: test_plugin.t 110 - flux_plugin_call (test.bar) success
FAIL: test_plugin.t 111 - flux_plugin_args_unpack result
FAIL: test_plugin.t 112 - call of test.bar set result bar
PASS: test_plugin.t 113 - flux_plugin_set_flags (p, RTLD_NOW) == 0
PASS: test_plugin.t 114 - load of plugin with invalid symbol fails immediately
PASS: test_plugin.t 115 - got expected error message: dlopen: test/.libs/plugin_bar.so: undefined symbol: my_invalid_sym
PASS: test_plugin.t 116 - flux_plugin_get_uuid works
PASS: test_plugin.t 117 - second plugin instance has different uuid
FAIL: test_plugin.t 118 - flux_plugin_load fails if plugin init callback fails
FAIL: test_plugin.t 119 - flux_plugin_strerror() notes that plugin init failed
ERROR: test_plugin.t - exited with status 1

I wonder if they suggest an issue with my environment, or if they are known problems.

Marcus

[garlick]

No those tests normally pass. All of them should.

I don't know if we'll get anything useful in the output but you could try running the test directly rather than through the make check framework, e.g.

$ cd src/common/libflux
$ ./test_plugin.t

[MarcusGDaniels]

A little more info that way.. It looked like I had to use GCC 14 to build parts of flux-sched. Is there a preferred compiler? ok 104 - loaded dso registered its own name # test/.libs/plugin_foo.so ok 105 - flux_plugin_get_path() on loaded dso works ok 106 - flux_plugin_call (test.foo) success not ok 107 - flux_plugin_args_unpack result # Failed test 'flux_plugin_args_unpack result' # at test/plugin.c line 454. not ok 108 - call of test.foo set result foo # Failed test 'call of test.foo set result foo' # at test/plugin.c line 456. # got: 'test/.libs/plugin_foo.so' # expected: 'foo' ok 109 - flux_plugin_arg_out works # out = {"result":"foo"} ok 110 - flux_plugin_call (test.bar) success not ok 111 - flux_plugin_args_unpack result # Failed test 'flux_plugin_args_unpack result' # at test/plugin.c line 467. not ok 112 - call of test.bar set result bar # Failed test 'call of test.bar set result bar' # at test/plugin.c line 469. # got: '(null)' # expected: 'bar' ok 113 - flux_plugin_set_flags (p, RTLD_NOW) == 0 ok 114 - load of plugin with invalid symbol fails immediately ok 115 - got expected error message: dlopen: test/.libs/plugin_bar.so: undefined symbol: my_invalid_sym ok 116 - flux_plugin_get_uuid works ok 117 - second plugin instance has different uuid not ok 118 - flux_plugin_load fails if plugin init callback fails # Failed test 'flux_plugin_load fails if plugin init callback fails' # at test/plugin.c line 522. not ok 119 - flux_plugin_strerror() notes that plugin init failed # Failed test 'flux_plugin_strerror() notes that plugin init failed' # at test/plugin.c line 525. # '' # doesn't match: 'flux_plugin_init failed' 1..119 # Looks like you failed 6 tests of 119 run. From: Jim Garlick ***@***.***> Sent: Wednesday, July 31, 2024 10:39 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) No those tests normally pass. All of them should. I don't know if we'll get anything useful in the output but you could try running the test directly rather than through the make check framework, e.g. $ cd src/common/libflux $ ./test_plugin.t — Reply to this email directly, view it on GitHub <#6160 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVP47WXV4AIHDV4MTOKTZPEOKZAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQYDKOA> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVP3NRV32Z5SVSGVMP2DZPEOKZA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATOZZU.gif> Message ID: ***@***.*** ***@***.***> >

[garlick]

It looked like I had to use GCC 14 to build parts of flux-sched. Is there a preferred compiler?

The newest gcc we are using in CI for flux-core appears to be 13.2.
In flux-sched, one of the CI builders uses 12 so 14 should not be required.

If it were me I would probably try a fresh build of flux-core with the system default compiler and see how that goes.

[garlick]

Mentioning @trws here in case he has any insight on this.

[trws]

We just started building with 14.1 in CI this week, and I’ve been using it a bit so it shouldn’t be that. Given the shared library fixing it below, I’m betting on either:

1. something inside libjansson relying on being a singleton for correctness across TUs, or
2. Ending up with two incompatible Janssons in different components somehow.

The second is more likely, but I’m not sure how that could happen unless mixing components from different builds or jansson packages.

[trws]

Waaaait a minute. I mis-parsed a piece of this. Mixing gcc versions in sched could cause all kinds of issues if you built them yourself because of incompatible libstdc++ versions. Not sure why that would cause this failure in core, but using a self-built g++ to link with c++ code built with a system one is a recipe for extremely hard to diagnose errors. If you want the newer compilers on Alma/rocky/centos/rhel use the devtoolset packages and this entire class of problem evaporates. May or may not be the cause here, but seriously it was the single biggest source of toolchain issues for Livermore as a whole until we did away with the custom built GCCs.

[chu11]

If you run the test with ./test_plugin.t -v -d perhaps we can get more info.

[garlick]

This one's a libtap unit test not a sharness test from t so I don't think those options would do anything.

[MarcusGDaniels]

Appears to be the same ***@***.*** libflux]$ ./test_plugin.t -v -d [..] not ok 107 - flux_plugin_args_unpack result # Failed test 'flux_plugin_args_unpack result' # at test/plugin.c line 454. not ok 108 - call of test.foo set result foo # Failed test 'call of test.foo set result foo' # at test/plugin.c line 456. # got: 'test/.libs/plugin_foo.so' # expected: 'foo' ok 109 - flux_plugin_arg_out works # out = {"result":"foo"} ok 110 - flux_plugin_call (test.bar) success not ok 111 - flux_plugin_args_unpack result # Failed test 'flux_plugin_args_unpack result' # at test/plugin.c line 467. not ok 112 - call of test.bar set result bar # Failed test 'call of test.bar set result bar' # at test/plugin.c line 469. # got: '(null)' # expected: 'bar' ok 113 - flux_plugin_set_flags (p, RTLD_NOW) == 0 ok 114 - load of plugin with invalid symbol fails immediately ok 115 - got expected error message: dlopen: test/.libs/plugin_bar.so: undefined symbol: my_invalid_sym ok 116 - flux_plugin_get_uuid works ok 117 - second plugin instance has different uuid not ok 118 - flux_plugin_load fails if plugin init callback fails # Failed test 'flux_plugin_load fails if plugin init callback fails' # at test/plugin.c line 522. not ok 119 - flux_plugin_strerror() notes that plugin init failed # Failed test 'flux_plugin_strerror() notes that plugin init failed' # at test/plugin.c line 525. # '' # doesn't match: 'flux_plugin_init failed' 1..119 # Looks like you failed 6 tests of 119 run. From: Al Chu ***@***.***> Sent: Wednesday, July 31, 2024 10:45 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) If you run the test with ./test_plugin.t -v -d perhaps we can get more info. — Reply to this email directly, view it on GitHub <#6160 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVPZ5OGG6THBMZ3NL4I3ZPEPADAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQYDSMI> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVPZY3M5AHQIRN4EOBX3ZPEPADA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATOZ3W.gif> Message ID: ***@***.*** ***@***.***> >

[MarcusGDaniels]

I tried both the system compiler (8.5.0 on Rocky 8) and my own GCC 13.2.0 build. Same outcome. ***@***.*** libflux]$ ldd test_plugin.t linux-vdso.so.1 (0x00007ffc53bc8000) libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007fad9fea3000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fad9fc83000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fad9fa7f000) libm.so.6 => /usr/lib64/libm.so.6 (0x00007fad9f6fd000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007fad9f327000) /lib64/ld-linux-x86-6 From: Jim Garlick ***@***.***> Sent: Wednesday, July 31, 2024 11:02 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) It looked like I had to use GCC 14 to build parts of flux-sched. Is there a preferred compiler? The newest gcc we are using in CI for flux-core appears to be 13.2. In flux-sched, one of the CI builders uses 12 so 14 should not be required. If it were me I would probably try a fresh build of flux-core with the system default compiler and see how that goes. — Reply to this email directly, view it on GitHub <#6160 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVP2PZBW6UQGGQA5S553ZPERBRAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQZDAMQ> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVP2EF5BI5IBQBWWX5RDZPERBRA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO2CU.gif> Message ID: ***@***.*** ***@***.***> >

[garlick]

Thanks. We do test rocky 8 with the default compiler in CI so this must be something environmental. I'm really puzzled what it might be though.

[MarcusGDaniels]

It doesn’t appear that jansson and libarchive include files are correctly located if they are in non-standard locations. From: Jim Garlick ***@***.***> Sent: Wednesday, July 31, 2024 11:28 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) Thanks. We do test rocky 8 with the default compiler in CI so this must be something environmental. I'm really puzzled what it might be though. — Reply to this email directly, view it on GitHub <#6160 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVPZGS52GNV6XIHAQFH3ZPEUENAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQZTSMQ> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVP4KFAHC5LTIMU7ZE4DZPEUENA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO2OQ.gif> Message ID: ***@***.*** ***@***.***> >

[garlick]

It doesn’t appear that jansson and libarchive include files are correctly located if they are in non-standard locations.

That should cause a configure failure though?

We find those with pkg-config so setting PKG_CONFIG_PATH to include the directory containing their .pc file would be needed at configure time.

[MarcusGDaniels]

I believe I have that set correctly: ***@***.*** build]$ pkg-config --cflags jansson

…

-I/home/4553vn/packages/jansson/include ***@***.*** build]$ pkg-config --cflags libarchive

-I/home/4553vn/packages/libarchive/include -I/home/4553vn/packages/openssl/include ***@***.*** build]$ From: Jim Garlick ***@***.***> Sent: Wednesday, July 31, 2024 11:39 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) It doesn’t appear that jansson and libarchive include files are correctly located if they are in non-standard locations. That should cause a configure failure though? We find those with pkg-config so setting PKG_CONFIG_PATH to include the directory containing their .pc file would be needed at configure time. — Reply to this email directly, view it on GitHub <#6160 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVPZ6EQFJUC3FHMYZANTZPEVNXAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQ2DIOA> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVP4GHUESIDREOBGO2ITZPEVNXA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO2SA.gif> Message ID: ***@***.*** ***@***.***> >

[MarcusGDaniels]

Switching to a shared library build of jansson fixed the plugin check, for whatever reason. From: Jim Garlick ***@***.***> Sent: Wednesday, July 31, 2024 11:39 AM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) It doesn’t appear that jansson and libarchive include files are correctly located if they are in non-standard locations. That should cause a configure failure though? We find those with pkg-config so setting PKG_CONFIG_PATH to include the directory containing their .pc file would be needed at configure time. — Reply to this email directly, view it on GitHub <#6160 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVPZ6EQFJUC3FHMYZANTZPEVNXAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQ2DIOA> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVP4GHUESIDREOBGO2ITZPEVNXA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO2SA.gif> Message ID: ***@***.*** ***@***.***> >

[MarcusGDaniels]

A common problem where the system has the .so’s but not the -devel bits. So one is forced to install from source. From: Tom Scogland ***@***.***> Sent: Wednesday, July 31, 2024 12:28 PM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) We just started building with 14.1 in CI this week, and I’ve been using it a bit so it shouldn’t be that. Given the shared library fixing it below, I’m betting on either: 1. something inside libjansson relying on being a singleton for correctness across TUs, or 2. Ending up with two incompatible Janssons in different components somehow. The second is more likely, but I’m not sure how that could happen unless mixing components from different builds or jansson packages. — Reply to this email directly, view it on GitHub <#6160 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVP7WIVMKUGH4S5SCZGDZPE3DRAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQ3TGOA> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVPZUOGRGCX2V4KII6E3ZPE3DRA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO3EE.gif> Message ID: ***@***.*** ***@***.***> >

[MarcusGDaniels]

Store a file in flux-core that records the compiler used. If an attempt is made to override it for flux-sched, and flux-sched needs a newer one, explain that to the user. From: Tom Scogland ***@***.***> Sent: Wednesday, July 31, 2024 12:32 PM To: flux-framework/flux-core ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [flux-framework/flux-core] make check (Discussion #6160) Waaaait a minute. I mis-parsed a piece of this. Mixing gcc versions in sched could cause all kinds of issues if you built them yourself because of incompatible libstdc++ versions. Not sure why that would cause this failure in core, but using a self-built g++ to link with c++ code built with a system one is a recipe for extremely hard to diagnose errors. If you want the newer compilers on Alma/rocky/centos/rhel use the devtoolset packages and this entire class of problem evaporates. May or may not be the cause here, but seriously it was the single biggest source of toolchain issues for Livermore as a whole until we did away with the custom built GCCs. — Reply to this email directly, view it on GitHub <#6160 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACVVP5IXB2OCMOHUPTINK3ZPE3UBAVCNFSM6AAAAABLY5P536VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRQGQ3TMMA> . You are receiving this because you authored the thread. <https://github.com/notifications/beacon/AACVVPZVVCA57CINQ7HHECLZPE3UBA5CNFSM6AAAAABLY5P536WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQATO3FQ.gif> Message ID: ***@***.*** ***@***.***> >

[trws]

That's not what I mean, for what it's worth. Different compilers between core and sched is fine, different compilers used to compile sched and the system boost library, on the other hand, can create a completely broken application. It's mixing different copies of the C++ standard library that causes the issue.