Use Gurobi_jll for binaries #545

odow · 2024-03-04T02:14:17Z

See https://github.com/jump-dev/Gurobi_jll.jl

There are still a few unresolved issues:

Tag a release of Gurobi_jll
Check if binaries are installed on Pkg.add, or only if needed
Document how to set a license if using provided binaries
Test locally

A potential first-pass is to use the binaries only on CI (or linux, to maintain compat).

Would fix issues like
https://discourse.julialang.org/t/how-to-use-gurobi-11-in-julia-launched-by-anaconda/111271/2

Closes #509

codecov · 2024-03-04T02:37:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.24%. Comparing base (3ffff36) to head (8a06d34).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #545      +/-   ##
==========================================
+ Coverage   80.46%   83.24%   +2.78%     
==========================================
  Files           6        6              
  Lines        2928     3635     +707     
==========================================
+ Hits         2356     3026     +670     
- Misses        572      609      +37

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

.github/workflows/ci.yml

odow · 2024-03-04T22:38:04Z

Hmm. Something isn't right:

(base) oscar@Oscars-MBP /tmp % julia --project=xpr
The latest version of Julia in the `1.10` channel is 1.10.2+0.x64.apple.darwin14. You currently have `1.10.1+0.x64.apple.darwin14` installed. Run:

  juliaup update

to install Julia 1.10.2+0.x64.apple.darwin14 and update the `1.10` channel to that version.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.1 (2024-02-13)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(xpr) pkg> st
Status `/private/tmp/xpr/Project.toml` (empty project)

(xpr) pkg> add https://github.com/jump-dev/Xpress_jll.jl
     Cloning git-repo `https://github.com/jump-dev/Xpress_jll.jl`
    Updating git-repo `https://github.com/jump-dev/Xpress_jll.jl`
   Resolving package versions...
    Updating `/private/tmp/xpr/Project.toml`
  [308bddfa] + Xpress_jll v9.3.0 `https://github.com/jump-dev/Xpress_jll.jl#master`
    Updating `/private/tmp/xpr/Manifest.toml`
  [692b3bcd] + JLLWrappers v1.5.0
  [21216c6a] + Preferences v1.4.3
  [308bddfa] + Xpress_jll v9.3.0 `https://github.com/jump-dev/Xpress_jll.jl#master`
  [56f22d72] + Artifacts
  [ade2ca70] + Dates
  [8f399da3] + Libdl
  [de0858da] + Printf
  [fa267f1f] + TOML v1.0.3
  [4ec0a83e] + Unicode
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 2 already precompiled.

julia> import Xpress_jll

(xpr) pkg> st
Status `/private/tmp/xpr/Project.toml`
  [308bddfa] Xpress_jll v9.3.0 `https://github.com/jump-dev/Xpress_jll.jl#master`

(xpr) pkg> add https://github.com/jump-dev/Gurobi_jll
     Cloning git-repo `https://github.com/jump-dev/Gurobi_jll`
ERROR: failed to clone from https://github.com/jump-dev/Gurobi_jll, error: GitError(Code:ERROR, Class:HTTP, unexpected http status code: 404)

(xpr) pkg> add https://github.com/jump-dev/Gurobi_jll.jl
    Updating git-repo `https://github.com/jump-dev/Gurobi_jll.jl`
   Resolving package versions...
    Updating `/private/tmp/xpr/Project.toml`
  [c018c7e6] + Gurobi_jll v11.0.0 `https://github.com/jump-dev/Gurobi_jll.jl#master`
    Updating `/private/tmp/xpr/Manifest.toml`
  [c018c7e6] + Gurobi_jll v11.0.0 `https://github.com/jump-dev/Gurobi_jll.jl#master`

julia> import Gurobi_jll
zsh: killed     julia --project=xpr
(base) oscar@Oscars-MBP /tmp %

odow · 2024-03-04T23:23:20Z

The issue appears to be

julia> path_name = "/Users/oscar/.julia/artifacts/5a3b623cd7ac3c59c02bb5063b4db60287f33a1f/lib/libgurobi110.dylib"
"/Users/oscar/.julia/artifacts/5a3b623cd7ac3c59c02bb5063b4db60287f33a1f/lib/libgurobi110.dylib"

julia> using Libdl; dlopen(path_name, RTLD_LAZY | RTLD_DEEPBIND)
zsh: killed     julia --project=xpr

I wonder if it is because libgurobi110 is already available

julia> using Libdl

julia> dlpath("libgurobi110.dylib")
"/Library/gurobi1100/macos_universal2/lib/libgurobi110.dylib"

(base) oscar@Oscars-MBP 5a3b623cd7ac3c59c02bb5063b4db60287f33a1f % otool -L lib/libgurobi110.dylib
lib/libgurobi110.dylib:
	@rpath/libgurobi110.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1241.100.11)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)

(base) oscar@Oscars-MBP 5a3b623cd7ac3c59c02bb5063b4db60287f33a1f % dyld_info -dependents lib/libgurobi110.dylib 
zsh: killed     dyld_info -dependents lib/libgurobi110.dylib

(base) oscar@Oscars-MBP 5a3b623cd7ac3c59c02bb5063b4db60287f33a1f % dyld_info -dependents /Library/gurobi1100/macos_universal2/lib/libgurobi110.dylib
/Library/gurobi1100/macos_universal2/lib/libgurobi110.dylib [x86_64]:
    -dependents:
        attributes     load path
                       /usr/lib/libSystem.B.dylib
                       /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
                       /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
                       /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
                       /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
                       /System/Library/Frameworks/Security.framework/Versions/A/Security
/Library/gurobi1100/macos_universal2/lib/libgurobi110.dylib [arm64]:
    -dependents:
        attributes     load path
                       /usr/lib/libSystem.B.dylib
                       /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
                       /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
                       /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
                       /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
                       /System/Library/Frameworks/Security.framework/Versions/A/Security

odow · 2024-03-07T07:10:57Z

Here's a user that this would help: https://discourse.julialang.org/t/how-to-use-gurobi-11-in-julia-launched-by-anaconda/111271/5?u=odow

jd-foster · 2024-04-05T04:27:35Z

I can't reproduce your issue following your steps (julia 1.10.2 (arm64 version) on M2 Pro). It seems to run fine for me but I don't have Gurobi 11 installed. I know you've thought of this already, but have you definitely ruled out Rosetta incompatibilities?

odow · 2024-04-05T04:32:17Z

I stopped digging into this. I was waiting for Gurobi to get back to me on whether they're okay with this general approach.

Project.toml

odow · 2024-04-30T01:41:31Z

Can people try this locally? I'm still getting jump-dev/Gurobi_jll.jl#5

jd-foster · 2024-04-30T10:14:32Z

Still seems fine on an M2 at least. I can try on an Intel Mac.

(Remote possibility: clear out ~/.julia/compiled/v1.10/Gurobi_jll/ and ~/.julia/compiled/v1.11/Gurobi_jll/ ???)

simonbowly · 2024-04-30T11:54:31Z

Tested on M3 mac and linux (intel, glibc). No issues so far!

torressa · 2024-04-30T14:02:31Z

Tested on an M1 and Linux, both worked!

odow · 2024-04-30T19:03:05Z

Yeah I've obviously borked something.

I guess one question for @simonbowly and @torressa: should have we have this enabled by default?

Xpress asked us not to: https://github.com/jump-dev/Xpress.jl?tab=readme-ov-file#use-with-xpress_jll

But I would be in favor of enabling by default. It should mean that we just start having zero installation questions because everything will magically work. We should just add documentation on how to use grbgetkey to initialize a license.

simonbowly · 2024-05-11T02:15:30Z

Sorry I missed this, travelling at the moment.

Yes I'd say enable it by default if that's your preference. Whatever is most familiar/easy for JuMP users. I assume it will still be possible to manually build Gurobi.jl against a local install (say if someone needs a specific technical release not covered by Gurobi_jll)?

odow · 2024-05-12T20:26:51Z

Yes I'd say enable it by default if that's your preference

Okay, great.

I assume it will still be possible to manually build Gurobi.jl against a local install

Yes, I'll sort some documentation for this.

odow · 2024-05-14T02:17:48Z

I still haven't gotten to the bottom of my local crash, although I verified that things work okay on a different machine...

I propose that we take a look over the documentation, merge this, and then invite the community to test ] add Gurobi#master before we tag a new release.

odow · 2024-05-14T07:13:41Z

Elliot saves the day again:

jd-foster · 2024-05-14T10:08:33Z

Ok, I finally got around to trying this on my Intel Mac and I can reproduce (so it's not just you!).

odow · 2024-05-14T19:59:10Z

@jd-foster does it work if you run codesign --remove-signature ~/.julia/artifacts/.julia/artifacts/48494ffef82cbb1907e4dc8974431fe60a12ef06/lib/libgurobi110.dylib?

If so, we can add this to the JLL

odow · 2024-05-14T21:44:21Z

See jump-dev/Gurobi_jll.jl#13

Install with: ] add Gurobi_jll#od/fix-macos

jd-foster · 2024-05-14T22:26:01Z

does it work if you run codesign --remove-signature

Yes, that part also works, and installing with the od/fix-macos branch works too.

odow · 2024-05-14T22:33:35Z

Great. I'll tag a new release JuliaRegistries/General#106832

odow · 2024-05-14T23:58:08Z

An update should do the trick:

julia> using Gurobi

julia> Gurobi.libgurobi
"/Users/oscar/.julia/artifacts/c6002a5d5f15ef98c2821ea90c04c1ce087df37f/lib/libgurobi110.dylib"

(Gurobi) pkg> test
     Testing Gurobi
      Status `/private/var/folders/bg/dzq_hhvx1dxgy6gb5510pxj80000gn/T/jl_pAJTLT/Project.toml`
  [2e9cd046] Gurobi v1.2.3 `~/.julia/dev/Gurobi`
  [b8f27783] MathOptInterface v1.29.0
  [c018c7e6] Gurobi_jll v11.0.2
...

Set parameter OutputFlag to value 1
Set parameter OutputFlag to value 1
Test Summary:          | Pass  Total     Time
MathOptInterface Tests | 4438   4438  4m48.2s
     Testing Gurobi tests passed

odow · 2024-05-21T03:07:51Z

@simonbowly any comments before I merge?

simonbowly · 2024-05-22T13:09:15Z

No comments, looks good to me.

odow commented Mar 4, 2024

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

odow changed the title ~~WIP: use Gurobi_jll for binaries~~ DNMY: use Gurobi_jll for binaries Mar 4, 2024

odow mentioned this pull request Mar 4, 2024

Use Xpress_jll for binaries jump-dev/Xpress.jl#220

Merged

1 task

odow commented Mar 4, 2024

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

odow mentioned this pull request Apr 26, 2024

Debug kill jump-dev/Gurobi_jll.jl#5

Closed

odow force-pushed the od/gurobi_jll branch from e1fecab to 6b714fa Compare April 29, 2024 22:31

odow commented Apr 29, 2024

View reviewed changes

Project.toml Outdated Show resolved Hide resolved

torressa assigned torressa and unassigned torressa Apr 30, 2024

odow and others added 8 commits May 14, 2024 13:13

WIP: use Gurobi_jll for binaries

aca4687

Update

dabf60c

upate

7062a8c

Update Project.toml

33088bc

Update

868adc2

Update

4d64f6a

Update ci.yml

2f4d5b8

Add docs

8a06d34

odow force-pushed the od/gurobi_jll branch from 6ec942a to 8a06d34 Compare May 14, 2024 02:15

odow changed the title ~~DNMY: use Gurobi_jll for binaries~~ Use Gurobi_jll for binaries May 14, 2024

odow requested review from simonbowly and torressa May 14, 2024 02:56

odow closed this May 14, 2024

odow reopened this May 14, 2024

odow merged commit 37ccc87 into master May 22, 2024
30 checks passed

odow deleted the od/gurobi_jll branch May 22, 2024 19:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Gurobi_jll for binaries #545

Use Gurobi_jll for binaries #545

odow commented Mar 4, 2024 •

edited

Loading

codecov bot commented Mar 4, 2024 •

edited

Loading

odow commented Mar 4, 2024 •

edited

Loading

odow commented Mar 4, 2024 •

edited

Loading

odow commented Mar 7, 2024

jd-foster commented Apr 5, 2024

odow commented Apr 5, 2024

odow commented Apr 30, 2024

jd-foster commented Apr 30, 2024 •

edited

Loading

simonbowly commented Apr 30, 2024

torressa commented Apr 30, 2024

odow commented Apr 30, 2024

simonbowly commented May 11, 2024 •

edited

Loading

odow commented May 12, 2024

odow commented May 14, 2024

odow commented May 14, 2024

jd-foster commented May 14, 2024

odow commented May 14, 2024

odow commented May 14, 2024

jd-foster commented May 14, 2024 •

edited

Loading

odow commented May 14, 2024 •

edited

Loading

odow commented May 14, 2024

odow commented May 21, 2024

simonbowly commented May 22, 2024

Use Gurobi_jll for binaries #545

Use Gurobi_jll for binaries #545

Conversation

odow commented Mar 4, 2024 • edited Loading

codecov bot commented Mar 4, 2024 • edited Loading

Codecov Report

odow commented Mar 4, 2024 • edited Loading

odow commented Mar 4, 2024 • edited Loading

odow commented Mar 7, 2024

jd-foster commented Apr 5, 2024

odow commented Apr 5, 2024

odow commented Apr 30, 2024

jd-foster commented Apr 30, 2024 • edited Loading

simonbowly commented Apr 30, 2024

torressa commented Apr 30, 2024

odow commented Apr 30, 2024

simonbowly commented May 11, 2024 • edited Loading

odow commented May 12, 2024

odow commented May 14, 2024

odow commented May 14, 2024

jd-foster commented May 14, 2024

odow commented May 14, 2024

odow commented May 14, 2024

jd-foster commented May 14, 2024 • edited Loading

odow commented May 14, 2024 • edited Loading

odow commented May 14, 2024

odow commented May 21, 2024

simonbowly commented May 22, 2024

odow commented Mar 4, 2024 •

edited

Loading

codecov bot commented Mar 4, 2024 •

edited

Loading

odow commented Mar 4, 2024 •

edited

Loading

odow commented Mar 4, 2024 •

edited

Loading

jd-foster commented Apr 30, 2024 •

edited

Loading

simonbowly commented May 11, 2024 •

edited

Loading

jd-foster commented May 14, 2024 •

edited

Loading

odow commented May 14, 2024 •

edited

Loading