Error when using anonymous functions with kwargs

[pablosanjose]

This fails since at least 1.6 up until nightly

using Distributed
addprocs(1)
@everywhere begin
    f1 = o -> o
    f2 = (o, m=1) -> o
    fbad = (o; m=1) -> o
end
remotecall_fetch(f1, 1, 1)   # 1
remotecall_fetch(f2, 1, 1)   # 1
remotecall_fetch(fbad, 1, 1) # 1
remotecall_fetch(f1, 2, 1)   # 1
remotecall_fetch(f2, 2, 1)   # 1
remotecall_fetch(fbad, 2, 1) # errors!

The error of the last call is

ERROR: On worker 2:
MethodError: no method matching var"#3#4"(::Int64, ::Serialization.__deserialized_types__.var"#3#7", ::Int64)
The function `#3#4` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  var"#3#4"(::Any, ::var"#3#7", ::Any) (method too new to be called from this world context.)
   @ Main REPL[3]:4

Stacktrace:
 [1] #3
   @ ./none:-1 [inlined]
 [2] #3
   @ ./REPL[3]:0
 [3] #invokelatest#2
   @ ./essentials.jl:1045
 [4] invokelatest
   @ ./essentials.jl:1042
 [5] #110
   @ ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:287
 [6] run_work_thunk
   @ ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:70
 [7] #109
   @ ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:287
Stacktrace:
 [1] remotecall_fetch(f::Function, w::Distributed.Worker, args::Int64; kwargs::@Kwargs{})
   @ Distributed ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:465
 [2] remotecall_fetch(f::Function, w::Distributed.Worker, args::Int64)
   @ Distributed ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:454
 [3] remotecall_fetch(f::Function, id::Int64, args::Int64)
   @ Distributed ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:492
 [4] top-level scope
   @ REPL[9]:1

[pablosanjose]

Another intriguing observation: this does not fail

using Distributed
addprocs(1)
@everywhere begin
  f3(o; m = 1) = o
end

remotecall_fetch(f3, 1, 1)  # 1
remotecall_fetch(f3, 2, 1)  # 1

[exaexa]

Hi all,
just copying out the relevant pieces from slack here:

Pablo created this useful test:

Distributed.remotecall_eval(Main, 1, :(f = (; p = 2) -> 0))
Distributed.remotecall_eval(Main, 2, :(f = (; p = 2) -> 0))

remotecall_fetch(f, 2)  #error

Distributed.remotecall_eval(Main, 1, :(g = () -> 0))
Distributed.remotecall_eval(Main, 2, :(g = () -> 0))

remotecall_fetch(g, 2)  # no error

We can observe what's actually happening on the workers:

julia> Distributed.remotecall_eval(Main, 2, :(code_lowered(f)))
1-element Vector{Core.CodeInfo}:
 CodeInfo(
1 ─ %1 = Main.:(var"#3#4")
│   %2 = (%1)(2, #self#)
└──      return %2
)

julia> Distributed.remotecall_eval(Main, 2, :(code_lowered(g)))    # g is observably simpler
1-element Vector{Core.CodeInfo}:
 CodeInfo(
1 ─     return 0
)

julia> Distributed.remotecall_eval(Main, 2, :(code_lowered(var"#3#4")))
1-element Vector{Core.CodeInfo}:
 CodeInfo(
1 ─     nothing
└──     return 0
)

Now the actual error is:

julia> remotecall_fetch(f, 2)
ERROR: On worker 2:
MethodError: no method matching var"#3#4"(::Int64, ::Serialization.__deserialized_types__.var"##230")
The function `#3#4` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  var"#3#4"(::Any, ::var"#3#5")
   @ Main REPL[6]:1

I assume that the sequence of events is as follows:

• remotecall_fetch tries to move f to worker 2 as an object, it gets appropriately packed and later unpacked as var"#3#5"
• the tmp-symbol var"#3#4" exists on the remote so resolves just okay, but it refers to something unidentifiable with the original #3#4 which should now be #3#5
• now we're trying to use the new #3#5 as the parameter of #3#4 and we get the typecheck error.

Notably, if passing the evaluation code through as normal manually-escaped expressions, all works OK, because f does not get mangled with:

julia> Distributed.remotecall_eval(Main, 2, :(f()))
0

[pablosanjose]

Another clue from our conversation in slack. This variant works

using Distributed
addprocs(1);           # no error without addprocs

genf() = (; p = 2) -> 0

@everywhere begin
  test(f) = f()
end

f = genf()                ## this works
# f = (; p = 2) -> 0      ## this fails

pmap((0,1)) do x   # no error if f = genf()!
    test(f)
end

There is a key difference between f = genf() and f = (; p = 2) -> 0:

julia> f = (; p = 2) -> 0
#34 (generic function with 1 method)

julia> code_lowered(f)
1-element Vector{Core.CodeInfo}:
 CodeInfo(
1 ─ %1 = Main.:(var"#34#35")
│   %2 = (%1)(2, #self#)
└──      return %2
)

julia> f = genf()
#11 (generic function with 1 method)

julia> code_lowered(f)
1-element Vector{Core.CodeInfo}:
 CodeInfo(
1 ─ %1 = Core.getfield(#self#, Symbol("#11#12"))
│   %2 = (%1)(2, #self#)
└──      return %2

[pablosanjose]

I think we can reduce this further, and isolate it to an issue in Serialization itself (so we may need to move this issue)

julia> f = (; p = 2) -> 0;

julia> using Serialization

julia> serialize("test.dat", f)

julia> deserialize("test.dat")
#1 (generic function with 1 method)

julia> ff = deserialize("test.dat")
#1 (generic function with 1 method)

julia> ff()
ERROR: MethodError: no method matching var"#1#2"(::Int64, ::Serialization.__deserialized_types__.var"##230")
The function `#1#2` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  var"#1#2"(::Any, ::var"#1#3")
   @ Main REPL[2]:1

Stacktrace:
 [1] (::Serialization.__deserialized_types__.var"##230")()
   @ Main ./REPL[2]:1
 [2] top-level scope
   @ REPL[6]:1

This error does not arise if we do

julia> genf() = (; p = 2) -> 0; f = genf();

[pablosanjose]

Closing in favor of JuliaLang/julia#56815