Windows CI for the Julia Pkg errors with 'Could not load symbol "duckdb_vector_size"' #13911

abelsiqueira · 2024-09-12T17:29:42Z

What happens?

We have a failing Julia package build with DuckDB_jll 1.1.0 for Windows GitHub runners: https://github.com/TulipaEnergy/TulipaEnergyModel.jl/actions/runs/10833112611/job/30062514253

Revelant part reproduced:

ERROR: LoadError: could not load symbol "duckdb_vector_size":
The specified procedure could not be found.

I saw other failures on 1.1.0, but I couldn't tell if they were related to this issue or not.

To Reproduce

I created a simple repo building a Julia package that runs DuckDB here: https://github.com/abelsiqueira/DuckDBDebug.jl/actions/runs/10835456425/job/30067028887

You can see:

It passes for ubuntu on DuckDB_jll 1.0.0 and 1.1.0
It passes for Windows on DuckDB_jll 1.0.0
It fails for Windows on DuckDB_jll 1.1.0

The code is very simple, and it doesn't actually run, since it fails on precompilation, but everything is included in the repo (https://github.com/abelsiqueira/DuckDBDebug.jl).

OS:

windows-latest on CII

DuckDB Version:

1.1.0

DuckDB Client:

Julia

Hardware:

No response

Full Name:

Abel Soares Siqueira

Affiliation:

Netherlands eScience Center

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Yes, I have

The text was updated successfully, but these errors were encountered:

ansaardollie · 2024-09-12T18:10:08Z

I am having the same issue, just got a new laptop and did a fresh install of Julia, tried adding the package and this error prevented the package from precompiling.

szarnyasg · 2024-09-13T07:18:58Z

Hi @abelsiqueira and @ansaardollie, thanks for opening this issue and joining the discussion. The DuckDB Julia API is experimental and not covered by community support. We welcome contributions fixing this issue but the core team is unlikely to fix this bug in the near future.

abelsiqueira · 2024-09-13T15:42:42Z

Thanks for the reply @szarnyasg. Just an update on our side, [email protected] is broken on local Windows machines as well, not just the CI.

Maybe limiting the DuckDB dependency on DuckDB_jll to =1.0.0 is a workaround until it's fixed.

For other users, you can add and pin DuckDB_jll to v1.0.0 with

pkg> add [email protected]
pkg> pin [email protected]

Since DuckDB compat bounds include DuckDB_jll 1.1.0, then you either do that or pin DuckDB itself to v0.10.3.

szarnyasg · 2024-09-13T18:12:51Z

Thanks for the additional info, this definitely warrants a look on our side.

Mytherin · 2024-09-19T06:49:30Z

Does this issue still persist with DuckDB 1.1, or is this issue specifically about using DuckDB 1.0 in combination with DuckDB_jll 1.1?

abelsiqueira · 2024-09-19T06:59:09Z

It was about DuckDB.jl 1.0 with jll 1.1, because DuckDB.jl 1.1.0 was not around yet. But I reran the tests in https://github.com/abelsiqueira/DuckDBDebug.jl/actions/runs/10835456425/job/30067028887 and they also fail with DuckDB.jl 1.1.0

abelsiqueira · 2024-09-19T07:00:17Z

I'm on the phone hope this is the link: https://github.com/abelsiqueira/DuckDBDebug.jl/actions/runs/10835456425/job/30358662742#step:8:56

jhmenke · 2024-09-20T14:11:52Z

I can confirm this issue on a number of different Windows 10 computers.

I guess the Windows libduckdb is not compiled correctly. A temporary fix was:

[compat]
DuckDB_jll = "=1.0.0"

in the Project.toml

abelsiqueira · 2024-10-01T13:12:07Z

Maybe version 1.1.1 fixes the issue? Can we have new DuckDB_jll and DuckDB.jl releases?

mzy2240 · 2024-10-01T19:21:19Z

Still an issue in Windows

DuckDB.jl 1.1 on Windows is broken with an error 'Could not load symbol duckdb_vector_size' This limits the DuckDB version until that is fixed. refs: - duckdb/duckdb#13911

DuckDB 1.0 allows DuckDB_jll 1.1.0, which is not correct. This limits DuckDB_jll as well. refs: - duckdb/duckdb#13911

GhostOfElectric · 2024-10-14T17:27:04Z

If the 1.1.1 C/C++ windows library is downloaded from the DuckDB website and an environment variable JULIA_DUCKDB_LIBRARY is set to point at duckdb.dll within it, then the latest version will successfully compile in Julia on windows.

jdpavek · 2024-10-17T07:27:26Z

Thanks, GhostOfElectric. that worked for me.

KronosTheLate · 2024-11-04T07:13:12Z

If the 1.1.1 C/C++ windows library is downloaded from the DuckDB website and an environment variable JULIA_DUCKDB_LIBRARY is set to point at duckdb.dll within it, then the latest version will successfully compile in Julia on windows.

If version 1.1.1 of the binary works with DuckDB.jl, then I figures that the fix would be updating the binary. I triggered that in this PR, which might just fix this issue on your next update.

For future reference, I am pretty sure that one could automate a github workflow to trigger on a new tag, and create a PR with the tiny changes required to trigger a new build. This would ensure that the binary stays updated.

KronosTheLate · 2024-11-04T07:33:27Z

The outline for a workflow `.github/workflows/JuliaRelease.yml` could be something like this:

name: On Tag

on:
  push:
    tags:        
      - 'v[0-9]+.[0-9]+.[0-9]+'

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

permissions:
  id-token: write
  contents: read

jobs:
  update_duckdb:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Clone Yggdrasil
        run: git clone https://github.com/JuliaPackaging/Yggdrasil
    
      - name: Create branch
        run: |
          TAG=${{ github.ref }}
          BRANCH_NAME=update_duckdb_jll_${TAG#refs/tags/}
          git switch -c $BRANCH_NAME

      - name: Update version
        run: |
          COMMIT_HASH=$(git rev-parse HEAD)
          sed -i "s|version = v\".*\"|version = v\"${TAG#refs/tags/}\"|g" D/DuckDB/build_tarballs.jl
          sed -i "s|GitSource(\"https://github.com/duckdb/duckdb.git\", \".*\")|GitSource(\"https://github.com/duckdb/duckdb.git\", \"$COMMIT_HASH\")|g" D/DuckDB/build_tarballs.jl

      - name: Commit changes
        run: |
          git add .
          git commit -m "Update DuckDB to commit $COMMIT_HASH"

      - name: Push changes
        run: git push --set-upstream origin $BRANCH_NAME

      - name: Create pull request
        run: gh pr create -B base_branch -H $BRANCH_NAME --title 'Merge $BRANCH_NAME into base_branch' --body 'Created by GitHub action'

KronosTheLate · 2024-11-04T12:26:14Z

The jll is now updated to 1.1.2, but I still got the same precompilation error. The fix provided by @GhostOfElectric worked. So it seems like the 1.1.2 C/C++ library downloaded from here differs from the 1.1.2 library built by Yggdrasil.jl which ends up here.

This might be related to how the windows build has specific requirements:

KronosTheLate · 2024-11-04T14:45:40Z

I created this discouse post to get exta sets of eyes on this. I suspect the right person could fix this in a couple of minutes.

Attempting to fix [this issue](duckdb/duckdb#13911) by adding a special case for windows. Tried to make that special case a mixture of the generic build script, and the contents [here](https://duckdb.org/docs/dev/building/build_instructions.html#windows)

abelsiqueira added the needs triage label Sep 12, 2024

szarnyasg added the Julia label Sep 13, 2024

szarnyasg added under review and removed needs triage labels Sep 13, 2024

abelsiqueira mentioned this issue Oct 7, 2024

Limit DuckDB version to 1.0 until duckdb/duckdb#13911 is fixed TulipaEnergy/TulipaEnergyModel.jl#842

Merged

abelsiqueira added a commit to abelsiqueira/TulipaEnergyModel.jl that referenced this issue Oct 7, 2024

DuckDB_jll needs explicit compat bounds

e59a3ea

DuckDB 1.0 allows DuckDB_jll 1.1.0, which is not correct. This limits DuckDB_jll as well. refs: - duckdb/duckdb#13911

abelsiqueira mentioned this issue Oct 7, 2024

DuckDB_jll needs explicit compat bounds TulipaEnergy/TulipaEnergyModel.jl#843

Merged

datejada pushed a commit to TulipaEnergy/TulipaEnergyModel.jl that referenced this issue Oct 7, 2024

DuckDB_jll needs explicit compat bounds (#843)

f50fd5d

DuckDB 1.0 allows DuckDB_jll 1.1.0, which is not correct. This limits DuckDB_jll as well. refs: - duckdb/duckdb#13911

datejada mentioned this issue Oct 7, 2024

[Bug] Add compat bound for DuckDB_jll and DuckDB versions TulipaEnergy/TulipaClustering.jl#56

Closed

KronosTheLate linked a pull request Nov 5, 2024 that will close this issue

[DuckDB] Add special case for windows JuliaPackaging/Yggdrasil#9737

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows CI for the Julia Pkg errors with 'Could not load symbol "duckdb_vector_size"' #13911

Windows CI for the Julia Pkg errors with 'Could not load symbol "duckdb_vector_size"' #13911

abelsiqueira commented Sep 12, 2024

ansaardollie commented Sep 12, 2024

szarnyasg commented Sep 13, 2024

abelsiqueira commented Sep 13, 2024

szarnyasg commented Sep 13, 2024

Mytherin commented Sep 19, 2024

abelsiqueira commented Sep 19, 2024

abelsiqueira commented Sep 19, 2024 •

edited

Loading

jhmenke commented Sep 20, 2024

abelsiqueira commented Oct 1, 2024

mzy2240 commented Oct 1, 2024

GhostOfElectric commented Oct 14, 2024 •

edited

Loading

jdpavek commented Oct 17, 2024

KronosTheLate commented Nov 4, 2024

KronosTheLate commented Nov 4, 2024 •

edited

Loading

KronosTheLate commented Nov 4, 2024 •

edited

Loading

KronosTheLate commented Nov 4, 2024

Windows CI for the Julia Pkg errors with 'Could not load symbol "duckdb_vector_size"' #13911

Windows CI for the Julia Pkg errors with 'Could not load symbol "duckdb_vector_size"' #13911

Comments

abelsiqueira commented Sep 12, 2024

What happens?

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

Did you include all relevant data sets for reproducing the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

ansaardollie commented Sep 12, 2024

szarnyasg commented Sep 13, 2024

abelsiqueira commented Sep 13, 2024

szarnyasg commented Sep 13, 2024

Mytherin commented Sep 19, 2024

abelsiqueira commented Sep 19, 2024

abelsiqueira commented Sep 19, 2024 • edited Loading

jhmenke commented Sep 20, 2024

abelsiqueira commented Oct 1, 2024

mzy2240 commented Oct 1, 2024

GhostOfElectric commented Oct 14, 2024 • edited Loading

jdpavek commented Oct 17, 2024

KronosTheLate commented Nov 4, 2024

KronosTheLate commented Nov 4, 2024 • edited Loading

KronosTheLate commented Nov 4, 2024 • edited Loading

KronosTheLate commented Nov 4, 2024

abelsiqueira commented Sep 19, 2024 •

edited

Loading

GhostOfElectric commented Oct 14, 2024 •

edited

Loading

KronosTheLate commented Nov 4, 2024 •

edited

Loading

KronosTheLate commented Nov 4, 2024 •

edited

Loading