Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run tests on multiple cores #180

Open
amigalemming opened this issue Jul 25, 2017 · 9 comments · May be fixed by #410
Open

Run tests on multiple cores #180

amigalemming opened this issue Jul 25, 2017 · 9 comments · May be fixed by #410
Assignees
Milestone

Comments

@amigalemming
Copy link
Contributor

There are packages like pqc and tasty-quickcheck that allow to run tests in parallel on multiple cores. Since they run tests on different properties in parallel, they do no longer show the increasing test counter. Would it be possible to run different tests on the same property in parallel? This would still enable the live counter. Maybe it would even save some memory if tests on the same property share some data.
Maybe the parallel testing could be enabled by a new field of Args for the number of parallel testing threads?

@nick8325
Copy link
Owner

That would certainly be a nice feature!

It would require re-working the main QuickCheck loop in Test.QuickCheck.Test to spawn several tests and collect the reults. Possibly a bit of work, but something we probably ought to do.

@sol
Copy link
Contributor

sol commented Aug 1, 2017

@amigalemming Just as a side note, it is possible to show progress information, even when running QuickCheck properties in parallel. hspec does exactly that.

The hairy details:

  • We assume that the user runs a test suite with n tests.
  • The user specifies N, the number of parallel tests to run at a time.
  • Hspec's runner spawns one thread for each test, t₁ to tₙ.
  • Each thread tₓ has an associated Mₓ :: MVar (Either Progress Result).
  • Each thread waits on a semaphore with the capacity N.
  • Once a thread tₓ is signaled to run, it runs the associated test; while doing so it writes progress information into Mₓ and once it's done the result (that is write if the MVar is empty, otherwise replace).
  • The applications main thread folds over the list M₁..ₙ
  • For each Mₓ it reads the current value and writes progress information as long as it gets a Left-value.
  • Once it reads a Right-value it reports the result to the user and moves on to Mₓ₊₁.

With this approach we see progress information for the currently running test with the lowest x only. In practice this works out nicely.

/cc @feuerbach

@amigalemming
Copy link
Contributor Author

It seems that QuickCheck-2.11 uses more cores, at least top reports CPU usage higher than 300%. If so, I think it is not a good idea to parallelize without permission by the user. How can I limit the number of cores/threads used?

@nick8325
Copy link
Owner

No, it doesn't parallelise automatically! Not sure what's going on there.

@Rewbert
Copy link
Collaborator

Rewbert commented Apr 25, 2024

I have implemented this :) With some blood, sweat, and tears, it should be in 2.16. Stay tuned

@phadej
Copy link
Contributor

phadej commented Apr 25, 2024

How soon 2.16 is expected? If it's "soon", than I won't care updating to QuickCheck-2.15 at all in between.

@Rewbert
Copy link
Collaborator

Rewbert commented Apr 25, 2024

@phadej There is no timeline. I think it will be a nightmare to merge, but worth it.

@Rewbert
Copy link
Collaborator

Rewbert commented Apr 25, 2024

To clarify; I would not wait if I were you.

@Rewbert
Copy link
Collaborator

Rewbert commented May 16, 2024

Here is a link to the paper describing the work done on this: https://arxiv.org/pdf/2404.16062

@Rewbert Rewbert linked a pull request Jun 18, 2024 that will close this issue
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants