Skip to content

Implement test data generator - #405

Merged
Aratz merged 32 commits into
devfrom
feature/pna-2602_test_data_generator
Jul 9, 2026
Merged

Implement test data generator#405
Aratz merged 32 commits into
devfrom
feature/pna-2602_test_data_generator

Conversation

@Aratz

@Aratz Aratz commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Description

This PR implements a test data generator. The generator is split into three parts:

  1. Generate the topology of the graph representing a cell. For now it uses a simple downsampling technique to match the desired average degree. If necessary, this can be improved later with techniques matching a more realistic degree distribution.
  2. Populate the graph with markers, splitting the markers from the panel file into three abundance levels. Hashing markers are also added at this stage.
  3. Generate reads by randomly sampling rows from an edgelist and save them to fastq files.

This PR then reworks the denoise tests (currently the slowest of the testbase) to use synthetic data. Overall this cuts the total run time by around 3 minutes.

Fixes: PNA-2602

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Each function and helper function of the test generator has its own tests. This has also been tested end to end by generating fastq files and putting them through the nf-core pipeline.

PR checklist:

  • This comment contains a description of changes (with reason).
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and documentation and corrected any misspellings
  • I have documented any significant changes to the code in CHANGELOG.md

@Aratz
Aratz requested review from Copilot, elhb and ptajvar June 25, 2026 15:37
@Aratz Aratz self-assigned this Jun 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

Comment thread tests/common/data_generator/topology.py
Comment thread tests/common/data_generator/molecules.py
@Aratz

Aratz commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Test failures seem to be due to the dependency updates. Will fix these tests to make them more robust to such numerical artifacts

@Aratz
Aratz marked this pull request as ready for review July 8, 2026 13:22

@ptajvar ptajvar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition! I am just wondering whether it should live under common or pna given that other than topology.py, the rest is mostly exclusive for pna. Looks good to go to me otherwise.

@Aratz

Aratz commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Ah, I see, I think maybe for now it can stay where it is, if we develop a new product or something we can see if it's worth to move it elsewhere 👍 Thank you!

@Aratz
Aratz merged commit 60cb166 into dev Jul 9, 2026
15 checks passed
@Aratz
Aratz deleted the feature/pna-2602_test_data_generator branch July 9, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants