pkgconfig support

[jayvdb]

It would be handy to have pkg-config support, as a way to get the flags needed to build.

[mdavidsaver]

What do you have in mind? Either in terms of use case, or syntax/API?

[jayvdb]

My recent use-case is Joungkyun/python-chardet#5 , where there are two other existing users of pkg-config.

A stale project I was involved in where this would be useful was https://github.com/pywikibot-catfiles/file-metadata/blob/master/setupdeps.py / https://github.com/pywikibot-catfiles/file-metadata/blob/master/setup.py

https://pypi.org/project/imagecodecs/ is another which would benefit immensely from it, but I believe the author works offline except for publishing to PyPI.

But I can use a setuptools hack like this even if the authors dont like it, as I am often packaging python packages for openSUSE, and patching their setup.py , so a tool like this would let me more easily use pkg-config when patching python library builds which bundle their own source for static linkage to a library, which is against distro packaging policy.

At the same time, I created lpsinger/setuptools_pkgconfig#1 , which looks like it might be sufficient for some use cases. https://github.com/lpsinger/setuptools_pkgconfig/blob/master/setuptools_pkgconfig.py

[mdavidsaver]

I may be missing something, but I'm not sure I see the point of involving pkg-config as this seems to me to imply building against externally managed 3rd party libraries. setuptools-dso is intended to avoid needing to do this.

Maybe an example of what you would want a setup.py to look like would help to clarify?

As background

The situation which motivates this package is a python binding for a large external package consisting of multiple libraries, some of which uses c++, an all or nothing idea of dllimport/export, a special set of Makefiles, and a legacy install tree layout. This all led me to the idea of throwing out the upstream Makefiles entirely, and instead use distutils to build its libraries. This lets me use/abuse pip to manage this non-python build dependency. An added benefit is more seamless fallback to source build when pre-compiled wheel binaries aren't available, or don't work. The drawback of course is having to replace the Makefiles.

The resulting package epicscorelibs isn't so much a python module, as a collection of non-python libraries with associated headers. There is some python code as well, mainly to hold compiler flags (same role as pkg-config). These can either by directly loaded with ctypes, or used as a build dependency of eg. an extension.

I've been able to make this work on Linux, Windows, and (mostly) OSX. In no small part because of recent additions to pip enabling expression of build time dependences in pyproject.toml. It's been a year so far. It remains to be seen whether I can keep this scheme working.

[jayvdb]

There are almost always externally managed 3rd party libraries, even if they are libc/libm, the C++ stdlib, or the STL. pkg-config is a way for the host to provide definitions of those. Often I see Python wrappers look for an external version of a library, shared or static, and fall back to building their own when it is missing. setuptools_dso appears to be focused on the "fallback to building" scenario, and it looks quite good at that.

I mentioned two setup.py at Joungkyun/python-chardet#5 . Here is another example I can find quickly
https://github.com/zeromq/pyzmq/blob/master/setup.py

[navytux]

Another use-case for pkg-config support from setuptools_dso user: my pygolang is building libgolang dso as part of it:

https://lab.nexedi.com/kirr/pygolang/blob/4ca65816/setup.py#L198-203

The day when libgolang starts to depend on a host-provided library, pkg-config will be needed to find cflags/ldflags for that library.

[jayvdb]

@navytux , just fyi, I tried building from the pygolang sdist , failure at https://build.opensuse.org/package/show/home:jayvdb:py-bridge/python-pygolang . Probably missing files, in which case check-manifest is your friend.

[navytux]

@jayvdb, thanks for trying. You are probably reffering to:

[   51s] gcc: error: golang/_golang.cpp: No such file or directory
[   51s] gcc: fatal error: no input files

I agree that the error message tempts one to think that golang/_golang.cpp is missing from sdist. However it should be like this because golang/_golang.cpp should be built from golang/_golang.pyx by Cython and Cython is declared to be build-time prerequisite:

https://lab.nexedi.com/kirr/pygolang/blob/4ca65816/pyproject.toml

It is just setuptools behaviour to not try to translate .pyx -> .cpp if Cython is not present (would be good to fix for pygolang packaging, so that the error is more clear).

This way pip installs pygolang fine from its source tarball.

You will likely need to add the missing packages that are required to be present at build time - cython and gevent to BuildRequires in python-pygolang.spec.

Kirill

P.S. thanks for check-manifest hint: https://lab.nexedi.com/kirr/pygolang/commit/59892def.

[mdavidsaver]

I think I see the disconnect now. The model I've been working under would have "bundled" libraries exist as separate python packages, and be pulled in via the usual distutils/setuptools requires mechanism. In the case of my epicscorelibs package, I choose this approach with knowledge of python bindings other than my own which could make use of the "bundled" libraries.

[mdavidsaver]

So I guess what is being asked for is for:

https://github.com/mdavidsaver/setuptools_dso/blob/5c42c1baa321838bcf57fa16d260be8a0881f298/example/setup.py#L15-L17

to have more control of the library search path?

At the moment, the dsos= is used to define a search path which is effectively -Lbuild/<pkg> -L<site-packages>/<pkg>. "effectively" because I'm doing this search, not gcc. So only one of these two would be passed on.

https://github.com/mdavidsaver/setuptools_dso/blob/5c42c1baa321838bcf57fa16d260be8a0881f298/src/setuptools_dso/dsocmd.py#L89-L97

Since I'm already importing code from the packaging containing the DSO. Maybe the thing to do is to allow code in that package to influence compiler/linker flags for the Extension/DSO being linked against it? This would then give an opportunity for this code to defer to a system copy.

[jayvdb]

Yes. Your model of building libraries is very important, and it is great that you have built a setuptools plugin to facilitate that as there are so many custom hacks to support that - almost every C module has setup.py voodoo to allow using a vendored copy of the lib or the system supplied one.

But allowing builders to choose to use system libraries will mean that packages which use setuptools-dso can also be re-used by Linux distos, which have a firm policy of not allowing other packages to use their own version of libraries which are already managed by the distro. i.e. the Python maintainers in the distro should not be adding copies of the core libraries that are maintained in the distro by the C-libraries team. Those core libraries go through security processes that the Python libraries do not, and those core libraries run the test suites for those libraries, whereas the Python-vendored copies of them would only undergo testing via the Python test suite which is usually much much smaller.

If we can get one setuptools tool which supports both modes, I believe a lot of existing packages will be interested in adopting it, or where there is resistance from the authors, other people can create new packages using setuptools-dso which wrap the existing packages.

[navytux]

Adding to pkg-config story: it would be nice if setuptools_dso would not only allow to use pkg-config to discover third-party modules, but also provide pkg-config files for DSOs installed by a project.

My context for this is: pygolang provides both python/cython-level modules, and plain C/C++ library/dso libgolang which is independent from Python. As there are users of libgolang outside Python context, it would be handy for those users to use pkg-config --cflags libgolang and pkg-config --libs libgolang in their builds.

[mdavidsaver]

Emitting .pc files is simple enough. Though I wonder where to put them, and how pkg-config would find them? Presumably adding something to $PKG_CONFIG_PATH.

[jayvdb]

Just a warning, there is only one place that they can go safely during the build, and that is in the build/.. tree. Anything else will cause setuptools to raise a SandboxViolation under certain conditions (easy_install) - I've learnt that the hard way, and not found a way to bypass it consistently pypa/setuptools#1151 .

And due to pips new build isolation, that means that the .pc files can only be seen by the one package which created them unless they are installed by a package which is listed as a build dependency.

[navytux]

@mdavidsaver, @jayvdb, thanks for feedback. Some thoughs on how to do: Python installs pc file for itself, for example:

kirr@deco:~$ dpkg -L libpython2.7-dev |grep pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/python-2.7.pc

(see python/cpython@f2caeed9c694)

And the path where it installs it can be quered via sysconfig:

In [1]: import sysconfig

In [2]: sysconfig.get_config_var('LIBPC')
Out[2]: '/usr/lib/x86_64-linux-gnu/pkgconfig'

We could use just that as the target path.

Yes, we should likely install .pc under build/... destdir during the build and only to system-wide prefix on install commands.

To properly support virtualenv, we should likely first add pkg-config support to virtualenv codebase - i.e. that it should add corresponding pkgconfig/ dir into created venv and adjust $PKG_CONFIG_PATH when entering the environment. Then setuptools_dso should install .pc into that location.

[jayvdb]

If it is necessary to 'install' stuff during build, using subprocess should escape the sandbox.

[navytux]

@jayvdb thanks for the hint. Good to keep this in mind.