1.1 Related work

The widespread adoption of multi-core processors in the 2000s created a growing need to utilize these resources effectively for computational tasks in R. The first packages to address this need were snow (Rossini et al., 2007) and multicore. With R version 2.14.0 (released in 2011), parallel computing capabilities were integrated into the base R system through the parallel package. The parallel package provides mclapply() and parLapply() as parallel versions of lapply() for multicore and cluster computing, respectively. Although widely used, both functions block the R session until all tasks have completed, preventing retrieval of partial results. Moreover, their static task allocation can lead to poor load balancing when tasks have heterogeneous runtimes.

The future package (Bengtsson, 2021) introduced a unified parallel computing interface in R, supporting various backends such as multisession, multicore, and callr, with the future.apply package providing parallel versions of the *apply() family of functions. However, the multisession and cluster backends are limited to 125 workers. Furthermore, per-task overhead — from process spawning (multicore, callr) or serialization across persistent workers (multisession, cluster) — can be significant for short-running tasks.

The mirai package (Gao, 2025) evaluates an R expression asynchronously in a parallel process, either locally or distributed across the network. The package has a strong focus on throughput, low overhead, and modern networking, eliminating many drawbacks of the older approaches. mirai provides a comprehensive daemon management interface for launching and configuring persistent daemons across local and remote systems (including SSH-based deployment and HPC schedulers). rush leverages this mechanism to start workers on remote machines.

The batchtools package (Lang et al., 2017) facilitates the execution of long-running tasks and large-scale computer experiments on HPC clusters. Because communication between the main process and workers relies entirely on the file system, the package incurs substantial per-task overhead from both job scheduling and file I/O, making it less suitable for high-throughput workloads with many short-running tasks. The more recent crew package (Landau, 2026) targets a similar use case, supporting long-running tasks in distributed systems ranging from traditional HPC clusters to cloud computing platforms, but likewise depends on a centralized controller for task coordination.

The parallel and distributed computing ecosystem in R is complemented by database connectivity layers that enable shared state and coordination. For relational database management systems, DBI (R Special Interest Group on Databases (R-SIG-DB) et al., 2024) defines a standardized interface that separates a common frontend from database-specific backends. For Redis, redux provides R bindings to the Redis API and exposes the command set through a low-level and a high-level interface.

The rrq (FitzJohn & Ashton, 2026) package is a task queue system for R using Redis as a coordination backend. It addresses the limitations of other packages by providing a non-blocking interface to parallel computing and keeping the overhead per task low. The package supports independent task queues, priority levels within queues, and inter-task dependencies. rrq is implemented on top of redux, which provides the underlying Redis connectivity and command interface. Its error-handling mechanism, particularly its heartbeat approach, directly informed the design of the corresponding mechanism in rush.

Python has a rich ecosystem for parallel and distributed computing. Dask (Rocklin, 2015) enables parallel and distributed execution via a dynamic task scheduler, and Ray (Moritz et al., 2018) is a general-purpose distributed computing framework. For HPO specifically, many frameworks adopt a centralized architecture in which a central scheduler dispatches configurations to workers: Ray Tune (Liaw et al., 2018), Syne Tune (Salinas et al., 2022), Google Vizier (Golovin et al., 2017), Ax (Olson et al., 2025), Dragonfly (Kandasamy et al., 2020), and Keras Tuner (O’Malley et al., 2019). In contrast, frameworks such as Optuna (Akiba et al., 2019), DeepHyper (Egelé et al., 2025), Hyperopt (Bergstra et al., 2013), and Orion (Bouthillier et al., 2023) support decentralized operation by coordinating workers through a shared database (SQL, MongoDB, or Redis), enabling each worker to independently propose and evaluate configurations.

In summary, the existing R ecosystem for parallel and distributed computing is rich but uniformly follows a controller-worker architecture. Packages such as parallel, future, and crew centralize task proposal and result collection in a single process, which can become a bottleneck at scale. While rrq and mirai reduce per-task overhead, neither supports decentralized optimization loops in which workers independently propose and evaluate tasks. At least in the domains of expensive black-box optimization and HPO, Python offers frameworks such as DeepHyper and Optuna that support decentralized workflows, but no comparable solution exists in R. rush addresses this gap by combining a decentralized architecture with low-overhead Redis-based communication.

2.1 Manager

The Rush manager class is responsible for creating, monitoring, and stopping a network. It is initialized using the rsh() function, which requires a network identifier and a config argument that specifies how to connect to the Redis database via the redux package. A running Redis server must be started separately beforehand.²

library(rush)

config = redux::redis_config()

rush = rsh(network = "demo-network", config = config)

Workers are started using the $start_workers() method, which accepts the worker loop (see Section 2.2) and the number of workers as arguments. Any additional named arguments are forwarded to the worker loop function. Workers run on mirai daemons (Gao, 2025), which are launched via mirai::daemons(). Calling mirai::daemons(n) opens a listener socket on the host, spawns a lightweight dispatcher implemented in C, and launches n persistent R processes that dial into the dispatcher and enter a receive–evaluate–return loop. Note that while rush still uses this mechanism to set up workers, most of the communication is actually facilitated via the Redis database, e.g., worker loop arguments, task inputs, outputs, and states. Daemons can be spawned on the local machine, on remote nodes via SSH, or on HPC clusters via job schedulers such as SLURM or SGE — in all cases, the daemon process dials back to the host and the same rush worker loop runs transparently. Alternatively, workers can be started locally via the processx package using $start_local_workers(), or a standalone script can be generated with $worker_script() for manual deployment; the only requirement is that the worker can connect to the Redis database.

mirai::daemons(4)

rush$start_workers(
  worker_loop = worker_loop,
  n_workers = 4)
rush$wait_for_workers(4)

Since $start_workers() returns immediately, $wait_for_workers(n) blocks until n workers have registered in Redis. Printing the Rush object reports the number of running workers and a summary of task states across the network (see Section 2.3).

rush

-- <Rush> --------------------------------------------------------------
* Running Workers: 4
* Queued Tasks: 0
* Running Tasks: 0
* Finished Tasks: 0
* Failed Tasks: 0

The $worker_info active binding returns a data.table (Barrett et al., 2026) with one row per worker, containing the worker identifier, the process ID, the host on which it runs, whether a heartbeat (see Section 2.5) is active, and its current state.

rush$worker_info

          worker_id   pid         hostname profile heartbeat   state
             <char> <int>           <char>  <char>    <lgcl>  <char>
1: pseudopolitic... 37025 ThinkPad-T14-...    <NA>     FALSE running
2: idiocratic_ca... 37028 ThinkPad-T14-...    <NA>     FALSE running
3: papery_sloth_... 37023 ThinkPad-T14-...    <NA>     FALSE running
4: precarnival_b... 37021 ThinkPad-T14-...    <NA>     FALSE running

The profile column identifies each worker’s compute profile, which can be used to group workers by available resources, e.g., CPUs or GPUs.

The workers can be stopped individually via $stop_workers(ids) or all together using the $reset() method, which also resets the network and database.

2.2 Worker loop

A worker loop is a plain R function that receives a RushWorker object for communicating with Redis.

worker_loop = function(rush, n_evals) {
  while (rush$n_finished_tasks < n_evals) {
    archive = rush$fetch_tasks_with_state(states = c("running", "finished"))
    xs = compute_task_inputs(archive)
    keys = rush$push_running_tasks(xss = list(xs))
    ys = compute_task_results(xs)
    rush$finish_tasks(keys, yss = list(ys))
  }
}

The above shows an abstract template for an exemplary situation where rush excels — and which is already fairly close to the asynchronous, decentralized HPO use case that we will cover in more detail in Section 3. In the (common) example above, we assume the following:

• The worker needs access to all available partial results to define and compute its next task.
• The inputs for the task also have to be computed and thus incur non-negligible costs.
• The computation of the task itself is time-consuming.

Inside the loop, the worker communicates with the shared Redis database through the following core methods of the RushWorker object:

• $push_running_tasks(xss) - Creates one or more new tasks with inputs xss in the database, marks them as "running", and returns their unique keys. Called before evaluation begins.
• $fetch_finished_tasks() - Retrieves all finished tasks from the database as a data.table (keys, inputs, outputs). Supports caching: previously fetched tasks are served from a local cache, and only new results are read from Redis.
• $fetch_tasks_with_state(states) - More general variant that retrieves tasks in specified states (e.g., both "running" and "finished"), useful when the worker needs to account for evaluations still in progress.
• $finish_tasks(keys, yss) - For tasks named by keys, writes corresponding results yss to the database and marks them as "finished".
• $n_finished_tasks - Active binding that returns the current global count of finished tasks across all workers, commonly used as a termination criterion.

Both $push_running_tasks() and $finish_tasks() accept an extra argument for associating auxiliary information with tasks and results that does not naturally fit into the inputs or outputs.

2.3 Tasks

Tasks are the basic units through which workers exchange information. Each task consists of four components: a unique key used to reference the task in the Redis database, a computational state, an input (xs), and an output (ys). The input and output are lists that may contain arbitrary data.

Tasks always have an associated computational state: "running", "finished", "failed", or "queued". The typical task lifecycle involves two transitions. The $push_running_tasks() method creates tasks marked as "running" and returns their keys. Upon completion, $finish_tasks() marks tasks as "finished" and stores the associated results. Tasks that encounter errors can be marked as "failed" using the $fail_tasks() method (see Section 2.5). The fourth state, "queued", supports a queue mechanism described in Section 2.4.

2.4 Queues

In addition to the autonomous loop pattern, rush supports a queue mechanism for cases in which tasks are created centrally and distributed to workers. The $push_tasks() method creates tasks in the "queued" state:

xss = lapply(1:20, function(i) {
  list(x1 = runif(1, -5, 5), x2 = runif(1, -5, 5))
})

rush$push_tasks(xss = xss)

Workers consume queued tasks using $pop_task(), which retrieves the next task, marks it as "running", and returns it. When the queue is empty, the method returns NULL, which serves as the termination signal:

worker_loop_queue = function(rush, fun) {
  repeat {
    task = rush$pop_task()
    if (is.null(task)) break
    ys = list(y = fun(task$xs$x1, task$xs$x2))
    rush$finish_tasks(task$key, yss = list(ys))
  }
}

The two patterns can be combined: workers first drain a queue of centrally created tasks, then transition to an autonomous loop for the remainder of the computation.

2.5 Error handling

rush provides error-handling mechanisms for two failure modes: R errors during task evaluation and unexpected worker crashes. For errors that occur during task evaluation, users must catch them within the worker loop and mark the corresponding task as "failed" using $fail_tasks(). If the error is not caught, the worker will crash.

tryCatch({
  ys = list(y = fun(xs$x1, xs$x2))
  rush$finish_tasks(key, yss = list(ys))
}, error = function(e) {
  condition = list(message = e$message)
  rush$fail_tasks(key, conditions = list(condition))
})

When a worker fails unexpectedly due to a crash, segmentation fault, or lost network connection, its tasks may remain in the "running" state indefinitely. The $detect_lost_workers() method identifies such workers and updates their state to "terminated". For workers started with $start_workers() or $start_local_workers(), lost worker detection works automatically by checking process status. Workers started via $worker_script() require an additional heartbeat mechanism: a background process periodically refreshes a Redis key with a set expiration timeout, and if the worker fails, the key expires, signaling that the worker is lost:

script = rush$worker_script(
  worker_loop = worker_loop,
  heartbeat_period = 1,
  heartbeat_expire = 3)
cat(strwrap(script, width = 76, exdent = 2), sep = "\n")

Rscript -e 'rush::start_worker(network_id = "demo-network", config =
  list(scheme = "redis", host = "127.0.0.1", port = "6379"),
  heartbeat_period = 1L, heartbeat_expire = 3L)'

2.6 Logging

Workers can write messages generated via the lgr (Fleck, 2026) package to the database. The lgr_thresholds argument of $start_workers() specifies the logging level for each logger by name, e.g., c("mlr3/rush" = "debug"). The mlr3/ prefix denotes the parent logger shared across the mlr3 ecosystem, into which rush integrates. Logging introduces a minor performance overhead and is disabled by default.

wl_log_message = function(rush) {
  lg = lgr::get_logger("mlr3/rush")
  lg$info("This is an info message from worker %s", rush$worker_id)
}

rush$start_workers(
  worker_loop = wl_log_message,
  n_workers = 2,
  lgr_thresholds = c("mlr3/rush" = "info"))

To retrieve all log entries, use the $read_log() method.

Sys.sleep(1)

rush$read_log()[, list(worker_id, msg)]

                         worker_id                             msg
                            <char>                          <char>
1: ungrammatical_beauceron_32fe... [rush] This is an info messa...
2:        accessible_bear_15e5052b [rush] This is an info messa...

Outputs and messages can be written to files. Set a path to a folder via the message_log and output_log arguments of $start_workers(). Messages and outputs are written to files named message_<worker_id>.log and output_<worker_id>.log in that folder, respectively.

rush$start_workers(
  worker_loop = wl_log_message,
  n_workers = 2,
  message_log = "logs",
  output_log = "logs")

2.7 Data storage

By default, rush stores all data in a Redis database. Redis is an open-source, in-memory key-value store commonly used as a database, cache, and message broker. It offers extremely low latency and supports multiple data structures such as strings, hashes, lists, and sets. rush maps its data model onto three Redis data structures: hashes, sets, and lists.

Redis hashes are a data structure for storing associative field–value pairs under a single key. They are optimized for memory efficiency and fast access to structured data. This makes them suitable for representing the input, output, and metadata of tasks. The key of the hash identifies the task in Redis and rush. A task is stored as a Redis hash with the following structure:

key : xs | ys | xs_extra

The field–value pairs are written by different methods, e.g., $push_running_tasks() writes xs and $finish_tasks() writes ys. The values of the fields are serialized lists or atomic values, e.g., deserializing the xs field gives list(x1 = 1, x2 = 2). This structure allows a hash to be efficiently converted into a table row and multiple hashes to be joined into a table. When retrieved, multiple hashes are joined into a tabular format:

| key | x1 | x2 | y | timestamp |
| 1.. |  3 |  4 | 7 |  12:04:11 |
| 2.. |  1 |  4 | 5 |  12:04:12 |
| 3.. |  1 |  1 | 2 |  12:04:13 |

Redis sets are unordered collections of unique elements that support efficient membership tests. rush uses sets to track task states; for example, the keys of all currently running tasks are stored in the "running_tasks" set.

Lists in Redis are ordered collections stored as linked lists that support efficient insertion and removal at both ends. rush uses lists for two purposes: managing the task queue and recording finished tasks. For the queue, the list structure is a natural choice, as it supports atomic push and pop operations. Storing the finished tasks in a list gives them an order by time, enabling workers to cheaply retrieve only the most recent results.

rush uses three mechanisms to coordinate concurrent access to the database. First, Redis executes commands on a single thread, making individual operations such as adding a key to a set or appending a key to a list atomic. Second, operations involving several keys are grouped into MULTI/EXEC transactions, which Redis executes without interruption by other commands. For example, $push_running_tasks() adds new keys to the "running_tasks" set and the list of all tasks in a single transaction. Third, potentially competing state transitions are implemented as atomic Lua scripts. These include transitions by $finish_tasks() and $fail_tasks(), insuring that a task is not recorded in two states simultaneously.

To minimize the number of Redis round-trips when workers repeatedly access many tasks (common in iterative algorithms), each Rush instance keeps a local data.table cache of the finished tasks it has already seen. On subsequent fetches, only the keys after the current cache length are requested; the corresponding hashes are read and appended to the cached table. This way, repeated queries scale with the number of newly finished tasks instead of the total result history.

The large_objects_path option allows large objects to be stored on a shared file system accessible to all workers. This can reduce network transfer overhead by allowing workers on the same node to benefit from cached file reads.

2.8 Retrieving results

Once all computational tasks are completed, the results can be retrieved from the database. As already explained, the $fetch_finished_tasks() method returns a data.table containing the task key, input, and result.

2.9 Testing

rush includes 105 unit tests written with testthat (Wickham, 2011). GitHub Actions runs R CMD check, including these tests, on Ubuntu with the release and development versions of R. The tests cover the core functionality, error handling, lost-worker detection, concurrent access, and caching behavior of the package.

3.1 The ask-and-tell pattern in optimization

The worker loop we discussed in Section 2.2 directly mirrors the ask-and-tell interface pattern that has become the dominant API design for black-box optimization frameworks. In this pattern, the optimizer exposes two methods — $ask(), which proposes the next candidate point(s), and $tell(), which receives the corresponding evaluation result(s) — while the user retains full control over the evaluation loop. In the worker loop template, the trio of $fetch_tasks_with_state(), compute_task_inputs(), and $push_running_tasks() corresponds to the ask step, while $finish_tasks() corresponds to the tell step. This design decouples the optimizer from the objective function evaluation, allowing users to integrate optimization into arbitrary workflows. Prominent implementations include Optuna (study.ask()/study.tell(), Akiba et al., 2019), Ax (get_next_trial()/complete_trial(), Olson et al., 2025), SMAC3 (Lindauer et al., 2022), and scikit-optimize (Head et al., 2018).

The ask-and-tell interface enables two fundamentally different parallelization strategies for optimization, which differ in where the optimization logic runs and how workers coordinate.

Synchronous batch parallelism. In this scenario, a single central process runs the optimizer. In each iteration, the optimizer asks for a batch of $q$ candidate points, which are then evaluated in parallel across $q$ workers using a standard parallel backend (e.g., future, mirai, or parallel). Once all $q$ evaluations complete, the results are told back to the optimizer, which updates its model and proposes the next batch. No shared state beyond the optimizer is needed, but depending on the complexity of the $ask() step, the central optimizer can become a bottleneck. Furthermore, synchronous batches waste resources when evaluation times are heterogeneous, as all workers must wait for the slowest evaluation in each batch to complete.

Asynchronous decentralized parallelism. In this scenario, each worker runs its own local optimization and ask-and-tell loop: it reads the shared history of completed (and possibly running) evaluations, asks its local optimizer for a proposal, evaluates it, and tells the result back to a shared store. Workers operate fully independently — they do not wait for each other and do not coordinate through a central controller. This eliminates both the controller bottleneck and the synchronization barrier. However, it requires infrastructure for shared-state coordination: workers must be able to read each other’s results and register their own tasks in a concurrent, conflict-free manner.

3.2 Asynchronous decentralized Bayesian optimization

Bayesian optimization (BO) is a sample-efficient strategy for optimizing expensive black-box functions. It maintains a probabilistic surrogate model (typically a Gaussian process or random forest) fitted to all completed evaluations, and selects the next evaluation point by optimizing an acquisition function that balances exploration and exploitation. In the standard sequential setting, a single process alternates between fitting the surrogate, proposing a point, and evaluating it. In ADBO (Egelé et al., 2023), this loop is replicated across multiple workers, each running independently. Every worker reads the shared history of evaluations from the database, fits its own local surrogate model, and proposes the next point autonomously — without waiting for other workers or coordinating through a central controller. For design points currently being evaluated on other workers (i.e., without a known objective value yet), a fake objective is imputed — usually a value between the minimum and maximum observed objective values in the archive. This penalizes the proposal of further design points near those currently under evaluation. Results are written back to the shared store as they complete, so all workers benefit from each other’s evaluations. To encourage diversity among proposals, each worker uses a stochastic acquisition function with different exploration–exploitation trade-offs (e.g., different $\lambda$ values in a confidence bound criterion). This architecture eliminates the central bottleneck that arises in synchronous or centralized parallel BO, where a single process must fit the surrogate and propose points for all workers; see Section 5.

We use the Branin function, defined over the domain $x_1 \in [-5, 10]$ and $x_2 \in [0, 15]$, as a toy objective:

branin = function(x1, x2) {
  (x2 - 5.1 / (4 * pi^2) * x1^2 + 5 / pi * x1 - 6)^2 +
    10 * (1 - 1 / (8 * pi)) * cos(x1) + 10
}

Next, we define an optimizer that proposes new points for evaluation. It takes as input an archive of running and completed tasks and returns a new candidate point xs for evaluation. The archive is represented as a data.table object. If the archive is empty, the optimizer proposes a random point. Otherwise, it imputes missing objective values for running tasks using the mean objective value of completed tasks. It then fits a random forest surrogate model with the ranger (Wright & Ziegler, 2017) R package and proposes a new point by minimizing a lower confidence bound acquisition function $\hat\mu(x) - \lambda\,\hat\sigma(x)$. The exploration parameter $\lambda$ is passed as an argument; each worker draws its own value from $\mathrm{Exp}(1)$ once at the start of the worker loop, following Egelé et al. (2023). Assigning each worker a different $\lambda$ realizes the diversification mechanism of ADBO: independent workers explore different exploration–exploitation trade-offs without any coordination. Fitting the surrogate model becomes increasingly expensive as the number of tasks in the archive grows. A detailed understanding of this optimizer is not necessary to follow the subsequent examples. Two properties are relevant for what follows: it uses previously evaluated points from other workers to inform new proposals, and proposal generation incurs a non-negligible computational cost.

optimizer = function(archive, lambda) {
  if (nrow(archive) == 0) {
    return(list(x1 = runif(1, -5, 10), x2 = runif(1, 0, 15)))
  }

  mean_y = mean(archive$y, na.rm = TRUE)
  archive["running", y := mean_y, on = "state"]

  surrogate = ranger::ranger(
    y ~ x1 + x2,
    data = archive,
    num.trees = 100L,
    keep.inbag = TRUE)

  xdt = data.table::data.table(
    x1 = runif(1000, -5, 10),
    x2 = runif(1000, 0, 15))
  p = predict(surrogate, xdt, type = "se", se.method = "jack")
  cb = p$predictions - lambda * p$se

  as.list(xdt[which.min(cb)])
}

The BO worker loop combines the optimizer with the rush API. Note that it uses $fetch_tasks_with_state() to retrieve both running and finished tasks, allowing the optimizer to impute values for evaluations still in progress. This is the ADBO pattern: each worker fits its own surrogate model to all available data and independently proposes the next configuration.

workerloop_adbo = function(rush, branin, optimizer) {
  # evaluate the initial design
  repeat {
    task = rush$pop_task()
    if (is.null(task)) break
    ys = list(y = branin(task$xs$x1, task$xs$x2))
    rush$finish_tasks(task$key, yss = list(ys))
  }

  # draw the worker-specific exploration parameter
  lambda = rexp(1)

  # ADBO loop
  while (rush$n_finished_tasks < 100) {
    archive = rush$fetch_tasks_with_state(states = c("running", "finished"))
    xs = optimizer(archive, lambda)

    key = rush$push_running_tasks(xss = list(xs))

    ys = list(y = branin(xs$x1, xs$x2))
    rush$finish_tasks(key, yss = list(ys))
  }
}

In practice, the optimization often starts with an initial design — a set of space-filling points that seed the surrogate model. The queue mechanism from Section 2.4 provides a natural way to distribute such an initial design: a structured design (e.g., Latin hypercube sampling) is created once with the lhs (Carnell, 2026) R package and pushed to the queue, and workers drain it before transitioning to the model-based loop.

lhs_points = lhs::maximinLHS(n = 25, k = 2)
x1_lower = -5
x1_range = 15
x2_lower = 0
x2_range = 15

xss = lapply(1:25, function(i) {
  # rescale to the domain
  list(
    x1 = lhs_points[i, 1] * x1_range + x1_lower,
    x2 = lhs_points[i, 2] * x2_range + x2_lower)
})

rush$push_tasks(xss = xss)

4.1 Experimental setup

The evaluation of short-running tasks is sensitive to the overhead introduced by rush. To quantify this overhead, we benchmark the core functions: $push_running_tasks(), $finish_tasks(), and $fetch_finished_tasks(). For $push_running_tasks() and $finish_tasks(), we vary the number of parameters (1, 10, 100) of the task and the payload size (1, 10, 100, 1,000, 10,000) of each parameter. In the case of $finish_tasks(), the payload size refers to the result. The payload size is the number of elements in a double vector where each double takes 8 bytes, i.e., a payload size of 10,000 corresponds to 80 kB. We benchmark $fetch_finished_tasks() both with and without caching to measure the effect of the caching mechanism. In addition to the number of parameters and payload size, we vary the number of tasks (1, 10, 100, 1,000, 10,000, 100,000). When caching is enabled, the cache holds all previously fetched tasks except the most recent one, so that each call retrieves only a single new result from the database. For both $fetch_finished_tasks() benchmarks, we cap the payload size at 100 to avoid creating overly large tables in memory. All benchmarks use rush version 1.3.0 with R 4.6.0 and Redis 8.10.1. All measurements are taken by a single worker against a local Redis instance communicating via a Unix socket.³ The runtime is measured with the microbenchmark (Mersmann et al., 2024) R package. We report the median and the 95th percentile over the replicates. All figures are created with the ggplot2 (Wickham, 2016), cowplot (Wilke, 2025), and scales (Wickham et al., 2025) R packages.

4.2 Results

The runtime of the functions $push_running_tasks() and $finish_tasks() is summarized in Table 1. For payload sizes up to 100, pushing a task to the database takes a median of 0.33–0.58 ms, and finishing a task takes slightly less, 0.25–0.36 ms. For the largest configuration (payload size 10,000 and 100 fields), the median push and finish times increase to 12.98 ms and 12.07 ms, respectively. The corresponding 95th percentiles are 25.41 ms and 20.80 ms, approximately 2.0 and 1.7 times the medians, indicating greater upper-tail latency for large payloads. For realistic HPO workloads, which typically involve tens of hyperparameters with scalar values (payload size 1), the median per-task overhead remains well below 1 ms.

For fetching, rush provides a built-in caching mechanism. The difference between fetching with and without caching is negligible for a small number of tasks. However, as the archive grows beyond 1,000 tasks, caching yields a substantial speedup, as shown in Figure 3 and Table 3. At 10,000 tasks and 10 parameters, fetching without the cache takes approximately 556 ms, compared to 3 ms with caching.

5.1 Experimental setup

We compare the effective CPU utilization of three parallelization strategies for BO on HPO tasks, representing increasing degrees of decentralization. We optimize nine hyperparameters of LightGBM (Ke et al., 2017), using the implementation in mlr3extralearners (Fischer et al., 2025), on four datasets, as summarized in Table 4 and Table 5. The datasets are downloaded from OpenML with mlr3oml (Lang & Fischer, 2026) and differ in their number of features and observations to induce a range of training runtimes. Each experiment runs for 10 minutes on 448 workers. The benchmarks are conducted on a Linux cluster at the Leibniz Supercomputing Centre (LRZ). Each node has 112 Intel Xeon Sapphire Rapids cores and 488 GiB of memory, running SUSE Linux Enterprise Server 15 SP6. We use four nodes for the benchmarks, resulting in 448 cores in total. All benchmarks use rush version 1.3.0.

Synchronous batch BO (CL). Evaluates $q$ hyperparameter configurations in parallel using the constant liar (CL) approach (Ginsbourger et al., 2010). The first point is obtained by maximizing the acquisition function. A fake value is then imputed, the surrogate model is updated, and the next point is proposed by maximizing the acquisition function again. Common imputation choices are the minimum, maximum, or predicted posterior mean of already evaluated points. In our setup, CL proposes 44 configurations evaluated with 10-fold cross-validation, resulting in 440 parallel evaluations. The remaining 8 cores cannot be used by design, since each configuration occupies 10 cores. mlr3 uses mirai to parallelize over the resampling iterations. Batch parallelization requires all workers to finish before the next batch can be proposed. We use early stopping to determine the optimal number of boosting iterations, which naturally induces heterogeneous runtimes: some models train for only a few iterations while others run for up to 5,000. This heterogeneity exposes the synchronization overhead inherent in batch approaches, as fast workers remain idle while waiting for slower ones to complete.

Asynchronous centralized BO (ACBO). Eliminates synchronization overhead by evaluating configurations asynchronously. When a worker becomes idle, the central controller proposes a new configuration and sends it to the worker without waiting for other evaluations to complete. Objective values for configurations currently under evaluation are imputed, as in CL. However, surrogate fitting and acquisition function optimization remain centralized and are executed sequentially by the controller. With many workers, this sequential bottleneck limits scalability: new configurations cannot be proposed fast enough to keep all workers occupied.

Asynchronous decentralized BO (ADBO). Distributes both surrogate updates and acquisition function optimization across the workers. Each worker maintains its own surrogate model and independently proposes the next configuration. When a worker completes an evaluation, it asynchronously shares the result via the database; other workers incorporate this information into their local models. As in CL and ACBO, objective values for configurations currently being evaluated are imputed. ADBO additionally uses stochastic acquisition functions to promote varying exploration–exploitation trade-offs between workers, e.g., by initializing the $\lambda$ parameter of the confidence bound acquisition function with different values for each worker.⁴ ADBO requires infrastructure that allows workers to exchange information asynchronously without a central controller — the architecture provided by rush.

All three strategies are implemented with mlr3tuning (Becker et al., 2025) and mlr3mbo (Becker, Schneider, et al., 2026) and use the mlr3mbo defaults: a random forest surrogate model (Wright & Ziegler, 2017), a confidence bound acquisition function, and local search for acquisition function optimization. The strategies differ in the number of cores used per configuration. CL distributes the 10 resampling iterations across 10 cores and evaluates 44 configurations in parallel. ACBO and ADBO execute the resampling iterations sequentially on a single core, allowing up to 448 configurations to be evaluated concurrently. For each task and replication, we randomly sample an initial design of 100 configurations from the search space in Table 4 and evaluate them using 10-fold cross-validation. The evaluation counts, wall-clock times, and utilization in Table 2 and the traces in Figure 5 exclude the initial design. Each task–algorithm combination is run five times. Benchmark jobs use fixed random seeds, with initial designs shared across strategies and fixed OpenML cross-validation splits. In asynchronous execution, result ordering depends on timing, so fixed seeds alone do not ensure exact reproducibility.

We measure effective CPU utilization following Egelé et al. (2023). Let $n_\text{workers}$ denote the number of workers, where each worker corresponds to a single CPU core, and $T_\text{walltime}$ be the elapsed wall-clock time. The total available CPU time is $T_\text{CPU} = T_\text{walltime} \cdot n_\text{workers}$. Let $T_\text{optimization}$ denote the cumulative time spent on surrogate fitting, acquisition optimization, and model training, summed across all workers. The effective CPU utilization is then $U = \frac{T_\text{optimization}}{T_\text{CPU}}$. Utilization thus counts the time spent proposing configurations as well as evaluating them. All three strategies fit one surrogate and optimize the acquisition function once per proposed configuration. To separate the two components, Table 6 additionally reports the learner utilization, which counts only the time spent evaluating configurations.

5.2 Results

The effective CPU utilization and the number of completed evaluations are summarized in Table 2 as means and standard deviations over the five replications. Across all tasks and all replications, ADBO achieves substantially higher utilization than both CL and ACBO, and this improvement is consistent across datasets with both short and long training runtimes. The utilization of ADBO varies by less than one percentage point between replications of the same task, so the standard deviations in Table 2 are small compared to the differences between the algorithms. ADBO also completes substantially more evaluations within the same wall-clock budget. Utilization includes the surrogate fitting needed to propose configurations: on credit-g, ADBO spends a mean cumulative 211,614 s fitting surrogates versus 11,164 s evaluating configurations (Table 6). ADBO spends more time per proposal on surrogate fitting than CL and ACBO because the cost of fitting a random forest grows with the size of the archive, which reaches over 15,000 evaluations for ADBO but stays below 650 for CL and ACBO. Counting only the time spent evaluating configurations, ADBO reaches a mean learner utilization of 4.2% on credit-g, compared to 0.1% for CL and ACBO, and between 56.2% and 99.2% on the other tasks, compared to at most 31.7% for CL and ACBO (Table 6).

All three methods reach similar final classification errors; ADBO achieves slightly lower mean errors, with differences of up to 0.8 percentage points on credit-g and less than 0.1 percentage points on the other tasks. High utilization alone therefore does not imply better optimization performance. The additional evaluations mainly help reach a given error earlier (Figure 5).

On tasks with very short training runtimes, centralized methods leave most compute resources idle. This effect is most pronounced on the credit-g task, where CL and ACBO complete only a few hundred evaluations while ADBO completes over 15,000 in every replication. The utilization of ADBO is between 270 and 300 times that of CL and ACBO across the five replications. Figure 4 illustrates the underlying mechanism. During the proposal phase, only a single core is busy, and for short-running objectives this phase dominates the wall-clock time.

On medium-cost tasks such as adult and KDDCup09_appetency, the same pattern persists. ACBO improves utilization relative to CL by eliminating synchronization overhead, but the centralized surrogate update remains a limiting factor: on adult, for instance, ACBO achieves a mean utilization of 2.4% compared to 95.7% for ADBO.

The airlines task has the longest training times, which reduces the relative overhead of surrogate updates and consequently increases utilization for all algorithms. Utilization also varies most between replications on this task, since a run evaluates only a few hundred configurations and their cost differs strongly: the utilization of ACBO ranges from 20% to 41% and its mean evaluation runtime from 132 s to 202 s across the five replications. Even in this favorable regime, ADBO maintains a substantial advantage, achieving between 2.4 and 4.9 times the utilization of ACBO in the individual replications. The decentralized architecture therefore scales robustly across workload regimes.

Beyond utilization, the parallelization strategy also affects budget adherence. The wall-clock runtime of CL can overshoot the nominal 10-minute budget, sometimes substantially (e.g. on KDDCup09_appetency; see Table 6), because CL checks the termination criterion only after an entire batch finishes. If a batch starts shortly before the time limit, the experiment continues until all evaluations complete, introducing significant budget overruns. In contrast, ACBO and ADBO check for termination after every completed evaluation, enforcing the wall-clock constraint more accurately. Asynchronous scheduling therefore improves both utilization and budget adherence.