Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround for freetype build error with external libz and spack-built pkg-config #496

Merged

Conversation

climbfuji
Copy link
Collaborator

@climbfuji climbfuji commented Dec 27, 2024

Description

This is a specific workaround for more general problem that I logged as an issue the spack upstream repo: spack#48293

In order to build freetype with a spack-built pkg-config and an external libz, this workaround tells spack where to look for zlib.pc by finding the file in the prefix of the external zlib, and then adding it to PKG_CONFIG_PATH.

We need this solution until the general problem has been addressed (or we can push this solution upstream if the spack develop are ok with it).

Testing

See JCSDA/spack-stack#1439.

Temporarily deactivated license header check because it's 2025 now and our spack code still looks for 2024. The next update from spack develop will resolve the problem, and an issue in our spack fork will remind us to reenable the license header check.

@climbfuji climbfuji force-pushed the bugfix/extzlib_pkgconfig_freetype branch from ec0f142 to cd576f2 Compare December 27, 2024 13:31
…os/builtin/packages/freetype/package.py: find zlib.pc and add directory to PKG_CONFIG_PATH
@climbfuji climbfuji force-pushed the bugfix/extzlib_pkgconfig_freetype branch from cd576f2 to ecc987c Compare December 27, 2024 14:37
@climbfuji climbfuji changed the title DRAFT: Bugfix/extzlib pkgconfig freetype Workaround for freetype build error with external libz and spack-built pkg-config Jan 2, 2025
@climbfuji climbfuji marked this pull request as ready for review January 2, 2025 21:47
@climbfuji climbfuji self-assigned this Jan 3, 2025
Copy link
Collaborator

@RatkoVasic-NOAA RatkoVasic-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@AlexanderRichert-NOAA
Copy link
Collaborator

If it's okay I'd prefer to fix the style check, depending how much trouble you think it would be.

As far as the change itself, I don't mind it while a permanent solution is found. Would it work though to add '/usr/lib64/pkg-config' or whatever to PKG_CONFIG_PATH in compilers.yaml? I agree that a general solution should be found that lets a Spack-installed pkg-config find external libraries...

@climbfuji
Copy link
Collaborator Author

If it's okay I'd prefer to fix the style check, depending how much trouble you think it would be.

I can try to pull in spack#48352 in a separate PR and merge that first. Hopefully, not too many conflicts (since we are already a bit behind spack develop).

As far as the change itself, I don't mind it while a permanent solution is found. Would it work though to add '/usr/lib64/pkg-config' or whatever to PKG_CONFIG_PATH in compilers.yaml? I agree that a general solution should be found that lets a Spack-installed pkg-config find external libraries...

I didn't want to do that, because I don't want all *.pc files from the OS be found.

@AlexanderRichert-NOAA
Copy link
Collaborator

Isn't that what this is doing though? If it's an external zlib, then wouldn't os.path.dirname(find_first(self.spec["zlib-api"].prefix, "zlib.pc", bfs_depth=10)) resolve to '/usr/lib64/pkg-config' or whatever? Or do you mean for other packages? Anyway, I'm fine with this as-is as long as you're happy with it, so I'll go ahead and approve.

@climbfuji
Copy link
Collaborator Author

Isn't that what this is doing though? If it's an external zlib, then wouldn't os.path.dirname(find_first(self.spec["zlib-api"].prefix, "zlib.pc", bfs_depth=10)) resolve to '/usr/lib64/pkg-config' or whatever? Or do you mean for other packages? Anyway, I'm fine with this as-is as long as you're happy with it, so I'll go ahead and approve.

Correct, for the purpose of finding dependencies of freetype only, I am adding the OS pkg-config path (or wherever the external zlib is installed) to the build environment. But only for this package. I will try to remember looking at the build log on Nautilus where it is picking up the other dependencies from (i.e. whether I need to append_path or prepend_path to make sure the spack-built libpng, bzip2 are found). Until then, I'll hold off merging this.

The update of the license file headers from spack develop is unfortunately more complicated. I think an alternative would be to temporarily modify the license header check to look for 2024 and leave a note in there. There is going to be a merge conflict next time we pull in spack develop (which removes the year range entirely)

@climbfuji
Copy link
Collaborator Author

@AlexanderRichert-NOAA For now, I commented out the license header check in the style CI tests. I added a fat note there that we need to reactivate it, and I will also create an issue for it.

@climbfuji climbfuji mentioned this pull request Jan 3, 2025
3 tasks
@climbfuji
Copy link
Collaborator Author

climbfuji commented Jan 3, 2025

@AlexanderRichert-NOAA I can confirm that the spack build picks up the correct dependencies. libpng and bzip2 are coming from the spack environment, and also the external libz is found and linked correctly:

/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/builds/unix/libtool
--mode=compile /p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/spack/lib/spack/env/oneapi/icx -pedantic -std=c99
-I/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/builds/unix
-I/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/builds/unix
-I/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/include
-c -Wall -fPIC -fvisibility=hidden
-I/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/envs/ne-oneapi-2024.2.1/install/oneapi/2024.2.1/bzip2-1.0.8-5ckiapz/include
-I/p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/envs/ne-oneapi-2024.2.1/install/oneapi/2024.2.1/libpng-1.6.37-totzlr6/include/libpng16
-pthread -DFT_CONFIG_CONFIG_H="<ftconfig.h>" -DFT_CONFIG_MODULES_H="<ftmodule.h>" -DFT_CONFIG_OPTIONS_H="<ftoption.h>" -DFT2_BUILD_LIBRARY -o /p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/builds/unix/ftsystem.lo /p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/cache/build_stage/heinzell/spack-stage-freetype-2.13.2-4xbeksh7tsvmrmn4ghjpreiqnzyyl3eh/spack-src/builds/unix/ftsystem.c
$ ldd envs/ne-oneapi-2024.2.1/install/oneapi/2024.2.1/freetype-2.13.2-4xbeksh/lib/libfreetype.so
        linux-vdso.so.1 (0x00007ffcbdfea000)
        libbz2.so.1.0 => /p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/envs/ne-oneapi-2024.2.1/install/oneapi/2024.2.1/bzip2-1.0.8-5ckiapz/lib/libbz2.so.1.0 (0x00007f60c5d94000)
        libpng16.so.16 => /p/app/projects/NEPTUNE/spack-stack/spack-stack-dev-20250103/envs/ne-oneapi-2024.2.1/install/oneapi/2024.2.1/libpng-1.6.37-totzlr6/lib64/libpng16.so.16 (0x00007f60c5d48000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f60c5a36000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f60c5816000)
        libsvml.so => /p/app/projects/NEPTUNE/spack-stack/oneapi-2024.2.1/compiler/2024.2/lib/libsvml.so (0x00007f60c41d2000)
        libirng.so => /p/app/projects/NEPTUNE/spack-stack/oneapi-2024.2.1/compiler/2024.2/lib/libirng.so (0x00007f60c40d9000)
        libimf.so => /p/app/projects/NEPTUNE/spack-stack/oneapi-2024.2.1/compiler/2024.2/lib/libimf.so (0x00007f60c3ccd000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f60c394b000)
        libintlc.so.5 => /p/app/projects/NEPTUNE/spack-stack/oneapi-2024.2.1/compiler/2024.2/lib/libintlc.so.5 (0x00007f60c5cd7000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f60c3747000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f60c3382000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f60c5c4e000)

@climbfuji climbfuji merged commit 57960d2 into JCSDA:spack-stack-dev Jan 3, 2025
15 checks passed
@climbfuji climbfuji deleted the bugfix/extzlib_pkgconfig_freetype branch January 3, 2025 14:38
climbfuji added a commit to JCSDA/spack-stack that referenced this pull request Jan 7, 2025
This PR updates all NRL sites to use the OS libz.so instead of the spack-built zlib-ng version. This is to address numerous problems, especially on Cray, where after loading spack-stack environments, basic tools like less and more sophisticated tools like ddt would throw many warnings or even stop working. For consistency, we do this on all NRL systems, regardless of being a Cray or not.

Included are also:

Additional, minor updates to the NRL site configs, most notably the update of cray-mpich on Narwhal to the latest version 8.1.26.
In template neptune-dev, exclude neptune-python-env with Intel Classic compilers. Yes, the time has come to say goodbye. The new versions of Python packages py-numpy and py-scipy don't build with these compilers, and it's not worth fixing those builds given that these compilers are deprecated and the replacement (%oneapi with icx, icpx, ifort) works fine. Add same conflict for LLVM (clang) compilers.
Add AMD AOCC compiler to tier2 site config Bounty.
Turn off gssapi variant for package libtircp because of a build error with latest OneAPI compilers (libtircp only needed by hdf@4, and the variant isn't needed)
This PR requires a bugfix/workaround in spack for package freetype, see JCSDA/spack#496
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

3 participants