Data Science Programmer · Parallel processing
Divide the work, retain the checks
I used independent groups of work to make use of multiple processor cores, then brought the results together for comparison.
Selected historical code · Generic placeholders · No original data or saved outputs
One workload, more than one core
The same tasks are shown first on one core, then distributed across four illustrative cores and combined into one result.
01 / APPROACH
Run groups across workers
The parallel map applies the same calculation to each group. Timing surrounds the work, and the results are combined afterwards.
R · adapted excerpt
plan(multisession, workers = worker_count)
tictoc::tic()
results <- future_map(
grouped_inputs,
~ process_group(.x[[2]], .x[[1]], .x[[3]])
)
tictoc::toc()
combined_results <- bind_rows(results)worker_count, grouped_inputs and process_group represent omitted configuration, inputs and the calculation. No runtime or speed-up result is shown.
02 / APPROACH
Optional export for testing
A final export, left commented out, can make the combined result available to the testing workflow.
R · adapted excerpt
# write_csv(combined_results, file.path(
# output_dir, "final-results-for-tests.csv"
# ))The destination and object name are generic placeholders.