Parallel::Deadworker on repetitive method use

[jzakiya]

I've created a primes testing method using the Miller-Rabin algorithm using the Parallel.each method. It works great when used with time between usages. When used in a loop to find the primes within a range above a certain size it throws Parallel::Deadworker errors.

Using Parallel::Kill or Parallel::Break seems to have no (little) affect.

  def prime4?    # perform miller-rabin tests in parallel
    witnesses = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41]
    return true if witnesses.include? self
    m = witnesses.reduce(:*)
    return false unless self > 1 and m.gcd(self % m) == 1
    wits = Prime::A014233NEW.sort.detect {|range, wits| range > self} # [k, [wit_prms]] or nil
    Parallel.each(wits && wits[1] || witnesses) { |p| return false unless miller_rabin_test(p) }
    true
  end

 :088 > s = []; (2..2000).each{|i| s << i if i.prime4? }
/home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork': unexpected return
Parallel::DeadWorker: Parallel::DeadWorker
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:60:in `rescue in work'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:57:in `work'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:351:in `block (4 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:462:in `with_instrumentation'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:350:in `block (3 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:338:in `loop'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:338:in `block (2 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:192:in `block (2 levels) in in_threads'

[grosser]

don't return inside the block ... use something like raise Parallel::Break unless miller_rabin_test(p)

you might be running into some process limits due to forking so fast :(

can you paste the full source or ideally a super smaller example that fails ?

[jzakiya]

Here is the link to the gist of the source code for the whole file.

https://gist.github.com/jzakiya/cfb1da397b03dff1ff7c1dbe0d57720e

[grosser]

running it just fine with:

1000.times do
puts 103344534334553454310101001010.prime4?
end

does that fail for you ?

On Tue, Aug 2, 2016 at 7:40 PM, Jabari Zakiya [email protected]
wrote:

Here is the link to the gist of the source code for the whole file.

https://gist.github.com/jzakiya/cfb1da397b03dff1ff7c1dbe0d57720e

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#185 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAsZ4_9SAYDyD5zgufJdZRl3eMqd4Mdks5qb_-pgaJpZM4JbC0a
.

[jzakiya]

The original code snippet that produced the error was this:
s = []; (2..2000).each{|i| s << i if i.prime4? }

If you use the serial versions prime? or prime1? they work without errors.
In the above snippet, it always fails at the same point.
After it fails if you do: s.size the value is 283, i.e. it identifies the first 283 primes before failing.

Also, the below example fails too for all the versions using Parallel.each, i.e. prime2|3|4? .
Is this because of the return inside the block, and/or thread hashing stuff?
BTW, this is on a 32-bit Linux distro laptop using a I5 cpu, but the same behavior exists on my 64-bit I7 Linux distro laptop.

2.3.1 :010 > 103344534334553454310101001010137.prime4?
/home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork': unexpected return
/home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork'/home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork': unexpected return
/home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:395:in `fork': unexpected return
: unexpected return
Parallel::DeadWorker: Parallel::DeadWorker
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:60:in `rescue in work'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:57:in `work'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:351:in `block (4 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:462:in `with_instrumentation'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:350:in `block (3 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:338:in `loop'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:338:in `block (2 levels) in work_in_processes'
        from /home/jzakiya/.rvm/gems/ruby-2.3.1/gems/parallel-1.9.0/lib/parallel.rb:192:in `block (2 levels) in in_threads'

Here is the method in parallel.rb:395

    def worker(job_factory, options, &block)
      child_read, parent_write = IO.pipe
      parent_read, child_write = IO.pipe

      pid = Process.fork do       # line 395 in parallel.rb file
        self.worker_number = options[:worker_number]

        begin
          options.delete(:started_workers).each(&:close_pipes)

          parent_write.close
          parent_read.close

          process_incoming_jobs(child_read, child_write, job_factory, options, &block)
        ensure
          child_read.close
          child_write.close
        end
      end