A uv Environment Is Not Compatible With pyo3 Out Of The Box

[mcmah309]

Summary

It seems a uv environment is not compatible with pyo3 as libpython is not found for the active environment, nor is the python codec.

PyO3/pyo3#4813 (comment)

Additionally the .cargo/config.toml needs to include

[env]
PYO3_PYTHON = { value = ".venv/bin/python", relative = true, force = true }

Example

Setup:

uv venv .venv --python 3.12
. .venv/bin/activate

src/lib.rs

use std::path::PathBuf;

use pyo3::prelude::*;

use serde::{Deserialize, Serialize};

#[pyclass]
#[derive(Debug, Serialize, Deserialize)]
struct Group {
    name: String,
}

#[pyclass]
#[derive(Debug, Serialize, Deserialize)]
struct User {
    username: String,
    group: Option<Py<Group>>,
    friends: Vec<Py<User>>,
}

#[test]
fn test_serialize() {
    let friend1 = User {
        username: "friend 1".into(),
        group: None,
        friends: vec![],
    };
    let friend2 = User {
        username: "friend 2".into(),
        group: None,
        friends: vec![],
    };

    let user = Python::with_gil(|py| {
        let py_friend1 = Py::new(py, friend1).expect("failed to create friend 1");
        let py_friend2 = Py::new(py, friend2).expect("failed to create friend 2");

        let friends = vec![py_friend1, py_friend2];
        let py_group = Py::new(py, Group {
            name: "group name".into(),
        })
        .unwrap();

        User {
            username: "danya".into(),
            group: Some(py_group),
            friends,
        }
    });

    let serialized = serde_json::to_string(&user).expect("failed to serialize");
    assert_eq!(
        serialized,
        r#"{"username":"danya","group":{"name":"group name"},"friends":[{"username":"friend 1","group":null,"friends":[]},{"username":"friend 2","group":null,"friends":[]}]}"#
    );
}

#[test]
fn test_deserialize() {
    let serialized = r#"{"username": "danya", "friends":
        [{"username": "friend", "group": {"name": "danya's friends"}, "friends": []}]}"#;
    let user: User = serde_json::from_str(serialized).expect("failed to deserialize");

    assert_eq!(user.username, "danya");
    assert!(user.group.is_none());
    assert_eq!(user.friends.len(), 1usize);
    let friend = user.friends.first().unwrap();

    Python::with_gil(|py| {
        assert_eq!(friend.borrow(py).username, "friend");
        assert_eq!(
            friend.borrow(py).group.as_ref().unwrap().borrow(py).name,
            "danya's friends"
        )
    });
}

Cargo.toml

[package]
name = "example"
version = "0.1.0"
edition = "2024"

[dependencies]
pyo3 = { version = "0.23", features = ["auto-initialize", "serde"] }
serde = { version = "1", features = ["serde_derive"] }
serde_json = "1"

[notatallshaw]

I assume this is a duplicate of #6812

[ngoldbaum]

It's likely that PyO3 can fix this by updating pyo3_build_config to emit some extra information. Right now it's making an assumption that the python-build-standalone python is breaking. There is still a libpython that can be used.

[zanieb]

The context @geofft added to #6812 should be relevant (and I presume he will be interested in how pyo3's build configuration is handling this).

In general, I think our behavior is correct here and that we can't easily address this in uv.

[geofft]

Hi @mcmah309 - can you give me some more details on how to reproduce this? I'm not following this issue or the linked pyo3 one well enough to figure out what the actual steps are.

I tried both maturin and standalone pyo3 and they seem to work fine. First, creating a venv in ~/u/.venv, activating it, and using maturin init + maturin develop --uv gets me an importable module:

$ mkdir u
$ cd u
$ uv venv -p 3.12
Using CPython 3.12.8
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
$ . .venv/bin/activate
(u) $ maturin init
✔ 🤷 Which kind of bindings to use?
  📖 Documentation: https://maturin.rs/bindings.html · pyo3
  ✨ Done! Initialized project /home/nixos/u
(u) $ maturin develop --uv
(u) $ python -c 'import u'

Then if I deactivate that venv and make an empty crate in ~/v and add the pyo3 dependency, the build fails because it can't find a Python interpreter, but setting PYO3_PYTHON or activating the venv works fine:

(u) $ deactivate
$ cd
$ cargo init --lib v
     Created library package
$ cd v
$ echo 'pyo3 = { version = "0.23", features = ["auto-initialize", "serde"] }' >> Cargo.toml
$ cargo build
[...]
   Compiling pyo3-build-config v0.23.4
error: failed to run custom build command for `pyo3-build-config v0.23.4`

Caused by:
  process didn't exit successfully: `/home/nixos/v/target/debug/build/pyo3-build-config-298ef98562d07ed0/build-script-build` (exit status: 1)
[...]
  --- stderr
  error: no Python 3.x interpreter found
$ PYO3_PYTHON=~/u/.venv/bin/python cargo build
   Compiling v v0.1.0 (/home/nixos/v)
    Finished dev [unoptimized + debuginfo] target(s) in 12.53s
$ . ~/u/.venv/bin/activate
(u) $ cargo build
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling v v0.1.0 (/home/nixos/v)
    Finished dev [unoptimized + debuginfo] target(s) in 6.03s

(For completeness, my local system is the NixOS 24.05 arm64 live CD inside nix-shell -p cargo -p rustc, which is mostly notable in that there's no default python on my PATH and no /usr/lib so the only place it can possibly get a Python is from the venv.)

So I'd like to know exactly what you're running to build and what the error message you're getting is.

[mcmah309]

PYO3_PYTHON Should be all you need to build, but the error's happen when you cargo run. I don't see you running that command. If you still are not seeing errors, my guess would be it's an environment difference, I don't see you installing uv or maturin so maybe your shell is not pure and there is a python instance on the PATH, or LD_LIBRARY_PATH/PYTHONHOME is set? Here's the core of the container I use

####  base: bases/ubuntu/cuda.md  ####

FROM nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04

RUN apt-get update -y \
    && apt-get upgrade -y \
    && apt-get install -y --no-install-recommends --no-install-suggests ca-certificates \
    && update-ca-certificates

####  rust: dependent/apt/rust/nightly.md  ####

# Based off: https://github.com/rust-lang/docker-rust/blob/9f287282d513a84cb7c7f38f197838f15d37b6a9/nightly/bookworm/slim/Dockerfile

ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
    RUST_VERSION=nightly

RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        gcc \
        libc6-dev \
        wget \
        ; \
    dpkgArch="$(dpkg --print-architecture)"; \
    case "${dpkgArch##*-}" in \
        amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='6aeece6993e902708983b209d04c0d1dbb14ebb405ddb87def578d41f920f56d' ;; \
        armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='3c4114923305f1cd3b96ce3454e9e549ad4aa7c07c03aec73d1a785e98388bed' ;; \
        arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='1cffbf51e63e634c746f741de50649bbbcbd9dbe1de363c9ecef64e278dba2b2' ;; \
        i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='0a6bed6e9f21192a51f83977716466895706059afb880500ff1d0e751ada5237' ;; \
        ppc64el) rustArch='powerpc64le-unknown-linux-gnu'; rustupSha256='079430f58ad4da1d1f4f5f2f0bd321422373213246a93b3ddb53dad627f5aa38' ;; \
        s390x) rustArch='s390x-unknown-linux-gnu'; rustupSha256='e7f89da453c8ce5771c28279d1a01d5e83541d420695c74ec81a7ec5d287c51c' ;; \
        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
    esac; \
    url="https://static.rust-lang.org/rustup/archive/1.27.1/${rustArch}/rustup-init"; \
    wget "$url"; \
    echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
    chmod +x rustup-init; \
    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
    rm rustup-init; \
    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
    rustup --version; \
    cargo --version; \
    rustc --version; \
    apt-get remove -y --auto-remove \
        wget \
        ;
    # Make sure to remove trailing `\` when you copy over a new version.
    # rm -rf /var/lib/apt/lists/*;

####  rust_essentials: dependent/apt/rust/essentials.md  ####

# libssl-dev: Needed for openssl certificates (may need to add the pkg files to PKG_CONFIG_PATH)
# libasound2-dev: Needed for alsa (may need to add the pkg files to PKG_CONFIG_PATH)
# pkg-config: Quering local libraries for compilation. Needed by rustc.
RUN apt-get update && apt install -y libssl-dev libasound2-dev pkg-config

####  rust_components: dependent/rust/components.md  ####

RUN rustup component add rustfmt

####  uv: dependent/apt/python/uv.md  ####

RUN apt install -y build-essential libssl-dev pkg-config curl \
    
    && curl -LsSf https://astral.sh/uv/install.sh | sh \
    
    && . $HOME/.local/bin/env \
    && uv tool install mypy

####  ~INLINE~  ####

ENTRYPOINT ["/bin/bash"]

(may want to change the base image to ubuntu:24.04)

Basically just follow exactly what included below. The comment I linked to is a way to resolve the issues that happen with cargo run

Additionally the .cargo/config.toml needs to include

[env]
PYO3_PYTHON = { value = ".venv/bin/python", relative = true, force = true }

Example

Setup:

uv venv .venv --python 3.12
. .venv/bin/activate
src/lib.rs

use std::path::PathBuf;

use pyo3::prelude::*;

use serde::{Deserialize, Serialize};

#[pyclass]
#[derive(Debug, Serialize, Deserialize)]
struct Group {
name: String,
}

#[pyclass]
#[derive(Debug, Serialize, Deserialize)]
struct User {
username: String,
group: Option<Py>,
friends: Vec<Py>,
}

#[test]
fn test_serialize() {
let friend1 = User {
username: "friend 1".into(),
group: None,
friends: vec![],
};
let friend2 = User {
username: "friend 2".into(),
group: None,
friends: vec![],
};

let user = Python::with_gil(|py| {
    let py_friend1 = Py::new(py, friend1).expect("failed to create friend 1");
    let py_friend2 = Py::new(py, friend2).expect("failed to create friend 2");

    let friends = vec![py_friend1, py_friend2];
    let py_group = Py::new(py, Group {
        name: "group name".into(),
    })
    .unwrap();

    User {
        username: "danya".into(),
        group: Some(py_group),
        friends,
    }
});

let serialized = serde_json::to_string(&user).expect("failed to serialize");
assert_eq!(
    serialized,
    r#"{"username":"danya","group":{"name":"group name"},"friends":[{"username":"friend 1","group":null,"friends":[]},{"username":"friend 2","group":null,"friends":[]}]}"#
);

}

#[test]
fn test_deserialize() {
let serialized = r#"{"username": "danya", "friends":
[{"username": "friend", "group": {"name": "danya's friends"}, "friends": []}]}"#;
let user: User = serde_json::from_str(serialized).expect("failed to deserialize");

assert_eq!(user.username, "danya");
assert!(user.group.is_none());
assert_eq!(user.friends.len(), 1usize);
let friend = user.friends.first().unwrap();

Python::with_gil(|py| {
    assert_eq!(friend.borrow(py).username, "friend");
    assert_eq!(
        friend.borrow(py).group.as_ref().unwrap().borrow(py).name,
        "danya's friends"
    )
});

}
Cargo.toml

[package]
name = "example"
version = "0.1.0"
edition = "2024"

[dependencies]
pyo3 = { version = "0.23", features = ["auto-initialize", "serde"] }
serde = { version = "1", features = ["serde_derive"] }
serde_json = "1"

[geofft]

the error's happen when you cargo run. I don't see you running that command.

Oh, I see now - I can reproduce the problem by building a binary that uses Python as a library:

cargo init b
cd b
echo 'pyo3 = { version = "0.23" }' >> Cargo.toml
uv venv -p 3.12
. .venv/bin/activate
cat > src/main.rs << EOF
use pyo3::prelude::*;

fn main() {
    Python::with_gil(|_py| ());   
}
EOF
cargo run

which gets me

target/debug/b: error while loading shared libraries: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

And I see now why you had test cases, cargo test fails too in the same manner if you invoke Python::with_gil from #[test].

I think the right way to solve this is by adding an rpath to the link arguments. I'm not yet sure whether that belongs on our side or on pyo3's side, let me read a bit. Thanks for the details!