Skip to content

added multiprocessing to Bayesian sampling#251

Draft
rozyczko wants to merge 2 commits into
bayesianfrom
bayesian_mp
Draft

added multiprocessing to Bayesian sampling#251
rozyczko wants to merge 2 commits into
bayesianfrom
bayesian_mp

Conversation

@rozyczko
Copy link
Copy Markdown
Member

This pull request introduces multiprocessing support for Bayesian DREAM sampling with the BUMPS minimizer, allowing parallel evaluation of the population during MCMC sampling. The changes add a new n_workers parameter to the sampling API, enabling users to specify the number of worker processes for parallel execution. Serialization logic is implemented to allow fit problems to be sent to worker processes, with robust handling of objects that are not trivially pickleable. The PR also includes comprehensive tests and a benchmarking script to demonstrate and validate the new multiprocessing functionality.

Multiprocessing support for BUMPS DREAM sampling

  • Added a new n_workers parameter to the sample method in both minimizer_bumps.py and multi_fitter.py, allowing users to specify the number of worker processes for parallel population evaluation. If n_workers is greater than 1, a custom multiprocessing pool mapper is used; otherwise, the default sequential evaluation is retained.

  • Implemented the BumpsPoolMapper class and associated serialization/deserialization helpers in minimizer_bumps.py to enable robust transfer of fit problems and closures to worker processes, including special handling for weak references and scipp.Variable objects. Added dependency on cloudpickle for advanced serialization.

Error handling and API improvements

  • Improved error messages and input validation for multiprocessing, including checks that n_workers is at least 1 and that the fit problem is serializable. The documentation for the sample methods was updated to describe the new parameter and possible exceptions.

  • Added a benchmarking script sampling_mpi.py to the tools/benchmarks directory, allowing users to measure wall-clock speedup from multiprocessing in a realistic sampling scenario.

@rozyczko rozyczko added [scope] enhancement Adds/improves features (major.MINOR.patch) [area] fitting Umbrella for fitting related work [priority] high Should be prioritized soon labels May 21, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 21, 2026

Codecov Report

❌ Patch coverage is 55.55556% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.04%. Comparing base (f072fd6) to head (79caafb).

Files with missing lines Patch % Lines
.../easyscience/fitting/minimizers/minimizer_bumps.py 55.55% 60 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##           bayesian     #251      +/-   ##
============================================
- Coverage     82.75%   82.04%   -0.72%     
============================================
  Files            62       62              
  Lines          5021     5156     +135     
============================================
+ Hits           4155     4230      +75     
- Misses          866      926      +60     
Flag Coverage Δ
integration 43.44% <21.48%> (-0.60%) ⬇️
unittests 81.51% <55.55%> (-0.70%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/easyscience/fitting/multi_fitter.py 98.83% <ø> (ø)
.../easyscience/fitting/minimizers/minimizer_bumps.py 82.93% <55.55%> (-15.03%) ⬇️

@rozyczko
Copy link
Copy Markdown
Member Author

This has quite a few steps for pickling/serialization of legacy objects based on weakrefs and global object.
Currently required by reflectometry but we are moving away from these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[area] fitting Umbrella for fitting related work [priority] high Should be prioritized soon [scope] enhancement Adds/improves features (major.MINOR.patch)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant