mirai 2.0.0

New Architecture

• Distributed computing now uses a single socket and URL at which all daemons connect (with or without dispatcher).
  • Allows a more efficient tcp:// or tls+tcp:// connection in all cases.
  • The number of connected daemons may be upscaled or downscaled at any time without limit.

New Features

• daemons(dispatcher = TRUE) provides a new and more efficient architecture for dispatcher. This argument reverts to a logical value, although 'process' is still accepted and retains the previous behaviour of the v1 dispatcher.
• daemons() gains argument 'serial' to register serialization configurations when using dispatcher. These automatically apply to all daemons that connect.
• stop_mirai() is now able to cancel remote mirai tasks (when using dispatcher), returning a logical value indicating whether cancellation was successful.
• A 'miraiError' now preserves the original condition object. This means that rlang::abort() custom metadata may now be accessed using $ on the 'miraiError' (thanks @James-G-Hill #173).

Updates

• status() using the new dispatcher is updated to provide more concise and insightful information.
• everywhere() updates:
  • Enhanced to return a list of mirai, which may be waited for and inspected (thanks @dgkf #164).
  • Drops argument '.serial' as serialization configurations are now registered via an argument at daemons().
• daemon() updates:
  • Gains the new argument 'dispatcher', which should be set to TRUE when connecting to dispatcher and FALSE when connecting directly to host.
  • Gains argument 'id' which accepts an integer value that allows status() to track connection and disconnection events.
  • '...' has been moved up to prevent partial matching on any of the optional arguments.
  • 'cleanup' argument simplified to a TRUE/FALSE value.
  • 'timerstart' argument removed.
• launch_local() and launch_remote() updates:
  • Enhanced to now launch daemons with the originally-supplied arguments by default.
  • Simplified to take the argument 'n' instead of 'url' for how many daemons to launch.
  • launch_local() now returns the number of daemons launched rather than invisible NULL.
• collect_mirai() is now interruptible and takes a '...' argument accepting the collection options provided to the 'mirai_map' [] method, such as .flat etc.
• ssh_config() simplified to take the argument 'port' instead of 'host'. For SSH tunnelling, this is the port that will be used, and the hostname is now required to be '127.0.0.1' (no longer accepting 'localhost').
• host_url() argument 'ws' is removed as a TCP URL is now always recommended (although websocket URLs are still supported).
• saisei() is defunct as no longer required, but still available for use with the old v1 dispatcher.
• daemons(dispatcher = "thread") (experimental threaded dispatcher) has been retired - as this was based on the old dispatcher architecture and future development will focus on the current design. Specifying 'dispatcher = thread' is defunct, but will point to 'dispatcher = process' for the time being.
• Requires nanonext >= 1.4.0.

mirai 1.3.1

Updates

• Cleanup of packages only detaches them from the search path and does not attempt to unload them, as it is not always safe to do so. Fixes daemon crashes using packages such as data.table (thanks @D3SL, #166).
• serialization() deprecated in mirai 1.2.0 is now removed.

mirai 1.3.0

New Features

• daemons(dispatcher = "thread") implements threaded dispatcher (experimental), a faster and more efficient alternative to running dispatcher in a separate process.
• mirai_map() adds [.progress_cli] as an alternative progress indicator, using the cli package to show % complete and ETA.
• daemons() gains argument 'force' to control whether further calls reset previous settings for the same compute profile.
• daemon() gains argument 'asyncdial' to allow control of connection behaviour independently of what happens when the daemon exits.

Behavioural Changes

• For daemons():
  • Argument 'dispatcher' now takes the character options 'process', 'thread' and 'none'. Previous values of TRUE/FALSE continue to be accepted (thanks @hadley #157).
  • Return value is now always an integer - either the number of daemons set if using dispatcher, or the number of daemons launched locally (zero if using a remote launcher).
  • Invalid type of ... arguments are now dropped instead of throwing an error. This allows ... containing unused arguments to be more easily passed from other functions.
• For mirai_map():
  • Now only performs multiple map over the rows of matrices and dataframes (thanks @andrewGhazi, #147).
  • Combining collection options is now easier, in the fashion of: x[.stop, .progress].
  • Collection options now work even if mirai is not on the search path e.g. mirai::mirai_map(1:4, Sys.sleep)[.progress].
• dispatcher() drops argument 'asyncdial' as it is rarely useful to set this here.
• everywhere() now errors if the specified compute profile is not yet set up, rather than fail silently.
• launch_local() and launch_remote() now strictly require daemons to be set, and will error otherwise.
• serial_config() now validates the arguments provided and returns them as a list. This means any saved configurations from previous package versions must be re-generated.

Updates

• Fixes daemons() to correctly handle a vector of URLs passed to 'url' again.
• Fixes flatmap with mirai_map()[.flat] assigning a variable 'typ' to the calling environment.
• Performance enhancements for mirai(), mirai_map() and the promises method.
• Requires nanonext >= 1.3.0.
• The package has a shiny new hex logo.

mirai 1.2.0

• everywhere() adds argument '.serial' to accept serialization configurations created by serial_config(). These allow normally non-exportable reference objects such as Arrow Tables or torch tensors to be used seamlessly across parallel processes without additional marshalling steps. Configurations apply on a per compute profile basis.
• serialization() is now deprecated in favour of the above usage of everywhere(), and will be removed in a future version.
• mirai_map() enhanced to perform multiple map over 2D lists/vectors, allowing advanced patterns such as mapping over the rows of a dataframe or matrix.
• 'mirai_map' [] method gains the option [.flat] to collect and flatten results, avoiding coercion.
• Collecting a 'mirai_map' no longer spuriously introduces empty names where none were present originally.
• Faster local daemons(dispatcher = FALSE) and make_cluster() by using asynchronous launches (thanks @mtmorgan #123).
• Local dispatcher daemons now synchronize with host, the same as non-dispatcher daemons (prevents use before all have connected).
• Fixes rare cases of everywhere() not reaching all daemons when using dispatcher.
• More efficient dispatcher startup by only loading the base package, in addition to not reading startup configurations (thanks @krlmlr).
• Removes hard dependency on stats and utils base packages.
• Requires nanonext >= 1.2.0.

mirai 1.1.1

• serialization() function signature and return value slightly modified for clarity. Successful registration / cancellation messages are no longer printed to the console.
• dispatcher() argument 'retry' now defaults to FALSE for consistency with non-dispatcher behaviour.
• remote_config() gains argument 'quote' to control whether or not to quote the daemon launch command, and now works with Slurm (thanks @michaelmayer2 #119).
• Ephemeral daemons now exit as soon as permissible, eliminating the 2s linger period.
• Requires nanonext >= 1.1.1.

mirai 1.1.0

• Adds mirai_map() for asynchronous parallel/distributed map using mirai, with promises integration. Allows recovery from partial failure or else early stopping, together with optional progress reporting.
  • x[] collects the results of a mirai_map x, waiting for all asynchronous operations to complete.
  • x[.progress] collects the results whilst showing a text progress bar.
  • x[.stop] collects the results applying early-stopping, which stops at the first error, and aborts remaining in-progress operations.
• Adds the 'mirai' method x[] as a more efficient equivalent of the interruptible call_mirai_(x)$data.
• Adds collect_mirai() as a more efficient equivalent of the non-interruptible call_mirai(x)$data.
• unresolved(), call_mirai(), collect_mirai() and stop_mirai() now accept a list of 'mirai' such as that returned by mirai_map().
• Improved mirai print method indicates whether a mirai has resolved.
• Calling daemons() with new settings when the compute profile is already set now implicitly resets daemons before applying the new settings instead of silently doing nothing.
• Argument 'resilience' retired at daemons() as automatic re-tries are no longer performed for non-dispatcher daemons.
• New argument 'retry' at dispatcher() governs whether to auto-retry in the dispatcher case.
• Fixes promises method for potential crashes when launching improbably short-lived mirai.
• Fixes bug that could cause a hang or crash when launching additional non-dispatcher daemons.
• Requires nanonext >= 1.1.0.

mirai 1.0.0

• Implements completely event-driven (non-polling) promises (thanks @jcheng5 for prototyping).
  • This is an innovation which allows higher responsiveness and massive scalability for 'mirai' promises.
• Behavioural changes to mirai() and everywhere():
  • (breaking change) no longer permits an unnamed list to be supplied to '.args'.
  • allows an environment e.g. environment() to be supplied to '.args' or as the only element of '...'.
  • allows evaluation of a symbol in the 'mirai' environment, e.g. mirai(x, x = 1).
• ssh_config() improvements:
  • new argument 'host' allows specifying the localhost URL and port to create a standalone configuration object.
  • order of arguments 'tunnel' and 'timeout' reversed.
• stop_mirai() now resolves to an 'errorValue' 20 (operation canceled) in the case the asynchronous task was still ongoing (thanks @jcheng5 #110).
• Rejected promises now show the complete error code and message in the case of an 'errorValue'.
• A 'miraiError' reverts to not including a trailing line break (as prior to mirai 0.13.2).
• Non-dispatcher local daemons now synchronize with host in all cases (prevents use before all have connected).
• [ method for 'miraiCluster' no longer produces a 'miraiCluster' object (thanks @HenrikBengtsson #83).
• Faster startup time as the parallel package is now only loaded when first used.
• Requires nanonext >= 1.0.0.

mirai 0.13.2

• mirai() and everywhere() behaviour changed such that '...' args are now assigned to the global environment of the daemon process.
• Adds with() method for mirai daemons, allowing for example: with(daemons(4), {expr}), where the daemons last for the duration of 'expr'.
• Adds register_cluster() for registering 'miraiCluster' as a parallel Cluster type (requires R >= 4.4).
• Adds is.promising() method for 'mirai' for the promises package.
• A 'miraiError' now includes the full call stack, which may be accessed at $stack.trace, and includes the trailing line break for consistency with 'as.character.error()'.
• mirai promises now preserve deep stacks when a 'miraiError' occurs within a Shiny app (thanks @jcheng5 #104).
• Simplified registration for 'parallel' and 'promises' methods (thanks @jcheng5 #103).
• Fixes to promises error handling and Shiny vignette (thanks @jcheng5 #98 #99).
• Requires R >= 3.6.

mirai 0.13.1

• Fixes regression in mirai 0.12.1, which introduced the potential for unintentional low level errors to emerge when querying dispatcher (thanks @dsweber2 for reporting in downstream {targets}).

mirai 0.13.0

• serialization adds arguments 'class' and 'vec' for custom serialisation of all reference object types.
• Requires nanonext >= 0.13.3.