Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Loading/precompiling packages requires them to be available on Distributed workers #80

Open
Uroc327 opened this issue Apr 14, 2022 · 0 comments
Open

Loading/precompiling packages requires them to be available on Distributed workers #80

Uroc327 opened this issue Apr 14, 2022 · 0 comments

Comments

@Uroc327
Copy link

Uroc327 commented Apr 14, 2022

I have the julia REPL running on my local machine and I use Distributed to connect to a headless remote server via ssh. I want to use these workers to do some @distributed computations and then plot the results on my local REPL, e.g. using GLMakie.

As GLMakie requires X11 and graphics drivers to be installed, it is only installed on my local machine but not on the server. When I try to include GLMakie or call it after having connected to the headless server, the call fails.

Example:

using GLMakie
using Distributed

addprocs([...]) # This should connect to the headless server

scatter(randn(10), randn(10))

This fails with the error message

Errors encountered while load FileIO.File{FileIO.DataFormat{:PNG}, String}("/home/Uroc327/.julia/packages/Makie/lgPZh/assets/icons/icon-128.png").
All errors:
===========================================
On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
===========================================
ArgumentError: Package ImageMagick [6218d12a-5da1-5696-b52f-db25d2ecc6d1] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

===========================================

Fatal error:
Error showing value of type Makie.FigureAxisPlot:
ERROR: On worker 2:
ArgumentError: Package PNGFiles [f57f5aa1-a3ce-4bc8-8ab9-96f992907883] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require
   @ ./loading.jl:1012
 [2] require
   @ ./loading.jl:936
 [3] JuliaLang/julia#1
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:79
 [4] JuliaLang/julia#103
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:274
 [5] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
 [6] run_work_thunk
   @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:72
 [7] JuliaLang/julia#96
   @ ./task.jl:411
Stacktrace:
  [1] macro expansion
    @ ./task.jl:388 [inlined]
  [2] _require_callback(mod::Base.PkgId)
    @ Distributed /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/Distributed/src/Distributed.jl:76
  [3] JuliaLang/julia#1
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:54 [inlined]
  [4] lock(f::ImageIO.var"#1#2"{Base.PkgId}, l::ReentrantLock)
    @ Base ./lock.jl:187
  [5] checked_import
    @ ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:53 [inlined]
  [6] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [7] load(f::FileIO.File{FileIO.DataFormat{:PNG}, String})
    @ ImageIO ~/.julia/packages/ImageIO/CzrzY/src/ImageIO.jl:59
  [8] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
  [9] invokelatest
    @ ./essentials.jl:706 [inlined]
 [10] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:219
 [11] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
 [12] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
 [13] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
 [14] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
 [15] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
 [16] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [17] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [18] getindex
    @ ./broadcast.jl:575 [inlined]
 [19] copy
    @ ./broadcast.jl:922 [inlined]
 [20] materialize
    @ ./broadcast.jl:883 [inlined]
 [21] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [22] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [23] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [24] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [25] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [26] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [27] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [28] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [29] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [30] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [31] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [32] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [33] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798
Stacktrace:
  [1] handle_error(e::CompositeException, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:61
  [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/error_handling.jl:56
  [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:228
  [4] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:197 [inlined]
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185
  [6] action
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:185 [inlined]
  [7] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:113
  [8] load
    @ ~/.julia/packages/FileIO/u9YLx/src/loadsave.jl:110 [inlined]
  [9] _broadcast_getindex_evalf
    @ ./broadcast.jl:648 [inlined]
 [10] _broadcast_getindex
    @ ./broadcast.jl:621 [inlined]
 [11] getindex
    @ ./broadcast.jl:575 [inlined]
 [12] copy
    @ ./broadcast.jl:922 [inlined]
 [13] materialize
    @ ./broadcast.jl:883 [inlined]
 [14] icon()
    @ Makie ~/.julia/packages/Makie/lgPZh/src/Makie.jl:262
 [15] GLMakie.Screen(; resolution::Tuple{Int64, Int64}, visible::Bool, title::String, kw_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:349
 [16] Screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:303 [inlined]
 [17] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:249 [inlined]
 [18] global_gl_screen(resolution::Tuple{Int64, Int64}, visibility::Bool, tries::Int64)
    @ GLMakie ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:394
 [19] global_gl_screen
    @ ~/.julia/packages/GLMakie/TfSB1/src/screen.jl:393 [inlined]
 [20] backend_display
    @ ~/.julia/packages/GLMakie/TfSB1/src/display.jl:2 [inlined]
 [21] display(scene::Scene)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:59
 [22] #display#915
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [23] display
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:53 [inlined]
 [24] #display#914
    @ ~/.julia/packages/Makie/lgPZh/src/display.jl:52 [inlined]
 [25] display(fap::Makie.FigureAxisPlot)
    @ Makie ~/.julia/packages/Makie/lgPZh/src/display.jl:52
 [26] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /builddir/julia-1.6.1/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798

When including GLMakie after having connected to the remote machine,

[...]
addprocs([...])
using GLMakie

the error message is also of the type is required but does not seem to be installed.

julia> versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44* (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

julia> ] st
      Status `~/.julia/environments/v1.6/Project.toml`
  [6e4b80f9] BenchmarkTools v1.3.1
  [1e616198] COSMO v0.8.5
  [13f3f980] CairoMakie v0.7.5
  [79afa230] Continuables v1.0.1
  [f65535da] Convex v0.15.1
  [f68482b8] Cthulhu v1.6.1
  [a93c6f00] DataFrames v1.3.2
  [31c24e10] Distributions v0.25.53
  [634d3b9d] DrWatson v2.9.1
  [2fe49d83] Expectations v1.8.0
  [7a1cc6ca] FFTW v1.4.6
  [41a02a25] Folds v0.2.7
  [e9467ef8] GLMakie v0.5.5
  [c3a54625] JET v0.4.6
  [033835bb] JLD2 v0.4.22
  [4076af6c] JuMP v0.23.2
  [b964fa9f] LaTeXStrings v1.3.0
  [b4c46c6c] LinuxPerf v0.3.2
  [2ab3a3ac] LogExpFunctions v0.3.10
  [bdcacae8] LoopVectorization v0.12.103
  [23992714] MAT v0.10.3
  [b8f27783] MathOptInterface v1.1.1
  [eff96d63] Measurements v2.7.1
  [01bcebdf] Nonconvex v1.0.2
  [5fb14364] OhMyREPL v0.5.12
  [429524aa] Optim v1.6.2
  [e4faabce] PProf v2.1.3
  [9b87118b] PackageCompiler v2.0.5
  [91a5bcdd] Plots v1.27.3
  [c3e4b0f8] Pluto v0.18.4
  [7f904dfe] PlutoUI v0.7.38
  [92933f4c] ProgressMeter v1.7.2
  [295af30f] Revise v3.2.1
  [f2b01f46] Roots v1.4.0
  [276daf66] SpecialFunctions v2.1.4
  [90137ffa] StaticArrays v1.4.3
  [2913bbd2] StatsBase v0.33.16
  [f3b207a7] StatsPlots v0.14.33
  [0c5d862f] Symbolics v4.3.0
  [a759f4b9] TimerOutputs v0.5.16
  [37b6cedf] Traceur v0.3.1
  [28d57a85] Transducers v0.4.73
  [1986cc42] Unitful v1.11.0
  [e88e6eb3] Zygote v0.6.37

] instantiate and ] precompile executed successfully before all of this.

This was MakieOrg/Makie.jl/issues/1817 .
@vtjnash vtjnash transferred this issue from JuliaLang/julia Feb 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Uroc327