Data Science Programmer · Parallel processing

Divide the work, retain the checks

I used independent groups of work to make use of multiple processor cores, then brought the results together for comparison.

Selected historical code · Generic placeholders · No original data or saved outputs

One workload, more than one core

The same tasks are shown first on one core, then distributed across four illustrative cores and combined into one result.

The same labelled tasks are processed sequentially on a single core, or divided among four illustrative cores before their results are combined. This shows work allocation, not measured timing or speed-up.The same labelled tasks are processed sequentially on a single core, or divided among four illustrative cores before their results are combined. This shows work allocation, not measured timing or speed-up.

01 / APPROACH

Run groups across workers

The parallel map applies the same calculation to each group. Timing surrounds the work, and the results are combined afterwards.

R · adapted excerpt

plan(multisession, workers = worker_count)

tictoc::tic()
results <- future_map(
  grouped_inputs,
  ~ process_group(.x[[2]], .x[[1]], .x[[3]])
)
tictoc::toc()

combined_results <- bind_rows(results)

worker_count, grouped_inputs and process_group represent omitted configuration, inputs and the calculation. No runtime or speed-up result is shown.

02 / APPROACH

Optional export for testing

A final export, left commented out, can make the combined result available to the testing workflow.

R · adapted excerpt

# write_csv(combined_results, file.path(
#   output_dir, "final-results-for-tests.csv"
# ))

The destination and object name are generic placeholders.