initial syntax changes and bulk modules update#137
Merged
Conversation
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates a broad set of Nextflow / nf-core modules and tests, primarily modernizing how module software versions are emitted/collected (moving away from versions.yml files toward structured tuple outputs and the versions topic), alongside a few module interface and dependency updates.
Changes:
- Refactor version reporting across multiple modules (new
versions_*tuple emits,topic: versions, and corresponding test/snapshot updates). - Update module interfaces and tests (notably MultiQC input/output tuple structure; SAMtools/Bcftools/Mosdepth-related test harness/config updates).
- Bump/adjust several tool versions/containers and related test snapshots (e.g., ensembl-vep cache 115, ont-modkit 0.6.1, severus 1.6, etc.).
Reviewed changes
Copilot reviewed 94 out of 94 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| workflows/lrsomatic.nf | Adjust MultiQC file collection closures; integrates with module outputs and MultiQC run. |
| subworkflows/nf-core/bam_stats_samtools/main.nf | Remove versions channel emission/mixing from the subworkflow. |
| subworkflows/nf-core/bam_stats_samtools/tests/main.nf.test | Update snapshots asserted by tests (drop versions output). |
| subworkflows/nf-core/bam_stats_samtools/tests/main.nf.test.snap | Snapshot updates reflecting removed versions outputs and runtime metadata changes. |
| subworkflows/local/prepare_reference_files.nf | Update UNZIP flatMap closures; stop mixing UNTAR versions into local ch_versions. |
| modules/nf-core/unzip/main.nf | Stub cleanup (remove unused stub args assignment). |
| modules/nf-core/untar/main.nf | Replace versions.yml output with tuple versions_untar (topic versions). |
| modules/nf-core/untar/meta.yml | Document new versions_untar output and topics.versions structure. |
| modules/nf-core/untar/tests/main.nf.test | Update tests to snapshot untar output + versions tuples instead of whole process.out. |
| modules/nf-core/untar/tests/main.nf.test.snap | Snapshot updates for new output structure and versions tuples. |
| modules/nf-core/severus/main.nf | Update inputs/outputs and change --out-dir to .; remove unused stub args assignment. |
| modules/nf-core/severus/severus.diff | Patch tracking updates for severus module changes. |
| modules/nf-core/severus/tests/main.nf.test.snap | Snapshot updates for severus output changes and runtime metadata. |
| modules/nf-core/samtools/stats/main.nf | Add support for task.ext.args passthrough to samtools stats. |
| modules/nf-core/samtools/index/main.nf | Replace versions.yml with versions_samtools tuple output (topic versions). |
| modules/nf-core/samtools/index/meta.yml | Document versions_samtools tuple output and versions topic. |
| modules/nf-core/samtools/index/tests/main.nf.test | Update tests to snapshot index output + versions tuples. |
| modules/nf-core/samtools/index/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple structure. |
| modules/nf-core/samtools/idxstats/main.nf | Replace versions.yml with versions_samtools tuple output (topic versions). |
| modules/nf-core/samtools/idxstats/meta.yml | Document versions_samtools output and versions topic. |
| modules/nf-core/samtools/idxstats/tests/main.nf.test | Update tests to snapshot idxstats output + versions tuples. |
| modules/nf-core/samtools/idxstats/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple structure. |
| modules/nf-core/samtools/flagstat/main.nf | Replace versions.yml with versions_samtools tuple output (topic versions). |
| modules/nf-core/samtools/flagstat/meta.yml | Meta formatting + document versions_samtools output and versions topic. |
| modules/nf-core/samtools/flagstat/tests/main.nf.test.snap | Snapshot updates for versions tuple output and runtime metadata. |
| modules/nf-core/samtools/faidx/main.nf | Change input signature to a single tuple (meta, fasta, fai). |
| modules/nf-core/samtools/faidx/meta.yml | Align metadata with updated inputs and version command typing. |
| modules/nf-core/samtools/faidx/tests/main.nf.test | Update tests to match new input structure and sanitized output snapshotting. |
| modules/nf-core/samtools/faidx/tests/main.nf.test.snap | Snapshot updates reflecting output structure and runtime metadata. |
| modules/nf-core/samtools/cat/main.nf | Replace versions.yml with versions_samtools tuple output (topic versions). |
| modules/nf-core/samtools/cat/meta.yml | Document versions_samtools output and versions topic. |
| modules/nf-core/samtools/cat/tests/main.nf.test | Simplify assertions to snapshot full process.out. |
| modules/nf-core/samtools/cat/tests/main.nf.test.snap | Snapshot updates reflecting output structure and versions tuples. |
| modules/nf-core/pigz/uncompress/main.nf | Stub cleanup (remove unused stub args assignment). |
| modules/nf-core/pigz/uncompress/meta.yml | Add missing licence field for pigz tool metadata. |
| modules/nf-core/multiqc/main.nf | Change inputs/outputs to meta-aware tuples; support multiple --config entries; stub outputs include plot stub. |
| modules/nf-core/multiqc/meta.yml | Update module interface documentation (meta tuple inputs and meta-tagged outputs); container metadata updates. |
| modules/nf-core/multiqc/tests/main.nf.test | Update tests for new tuple I/O and add multiple-config test. |
| modules/nf-core/multiqc/tests/main.nf.test.snap | Snapshot updates reflecting new output tuple structure and added test case. |
| modules/nf-core/mosdepth/main.nf | Replace versions.yml with versions_mosdepth tuple output (topic versions); improve arg validation logic; add stub validations. |
| modules/nf-core/mosdepth/meta.yml | Document versions_mosdepth output and versions topic; update maintainers list. |
| modules/nf-core/mosdepth/environment.yml | Reorder dependencies (htslib first, mosdepth second). |
| modules/nf-core/mosdepth/tests/main.nf.test | Consolidate test arg configuration via params/module_args and shared nextflow.config; simplify assertions. |
| modules/nf-core/mosdepth/tests/main.nf.test.snap | Snapshot updates reflecting versions tuple output and runtime metadata. |
| modules/nf-core/mosdepth/tests/nextflow.config | New shared test config to route params.module_args into ext.args. |
| modules/nf-core/mosdepth/tests/window.config | Remove per-test config (replaced by params/module_args). |
| modules/nf-core/mosdepth/tests/threshold.config | Remove per-test config (replaced by params/module_args). |
| modules/nf-core/mosdepth/tests/quantized.config | Remove per-test config (replaced by params/module_args). |
| modules/nf-core/modkit/pileup/main.nf | Update container/tool version and replace versions.yml with versions_modkit tuple output (topic versions). |
| modules/nf-core/modkit/pileup/meta.yml | Document versions_modkit output and versions topic. |
| modules/nf-core/modkit/pileup/environment.yml | Bump ont-modkit dependency to 0.6.1. |
| modules/nf-core/modkit/pileup/tests/main.nf.test.snap | Snapshot updates for versions tuple output and runtime metadata. |
| modules/nf-core/minimap2/index/main.nf | Replace versions.yml with versions_minimap2 tuple output (topic versions). |
| modules/nf-core/minimap2/index/meta.yml | Document versions_minimap2 output and versions topic. |
| modules/nf-core/minimap2/index/tests/main.nf.test | Add stub-mode test and update assertions to new output structure. |
| modules/nf-core/minimap2/index/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple output. |
| modules/nf-core/minimap2/align/main.nf | Replace versions.yml with versions_minimap2 tuple output (topic versions); improve error messages and stub validation. |
| modules/nf-core/minimap2/align/meta.yml | Document versions_minimap2 output and versions topic. |
| modules/nf-core/minimap2/align/tests/main.nf.test | Update tests to assert versions_* outputs via findAll. |
| modules/nf-core/minimap2/align/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple output and runtime metadata. |
| modules/nf-core/ensemblvep/vep/main.nf | Update container/deps to ensembl-vep 115.2; restructure inputs; replace versions.yml with multiple versions_* tuple outputs; emit report to multiqc_files topic. |
| modules/nf-core/ensemblvep/vep/meta.yml | Update module interface docs for new inputs/outputs/topics and versions tuple structure. |
| modules/nf-core/ensemblvep/vep/environment.yml | Bump ensembl-vep to 115.2 and add perl-math-cdf dependency. |
| modules/nf-core/ensemblvep/vep/tests/nextflow.config | Update test cache version to 115. |
| modules/nf-core/ensemblvep/vep/tests/main.nf.test | Update tests for new cache tuple structure and new versions outputs. |
| modules/nf-core/ensemblvep/vep/tests/main.nf.test.snap | Snapshot updates reflecting new output structure and versions tuples. |
| modules/nf-core/ensemblvep/vep/ensemblvep-vep.diff | Patch tracking updates for the ensemblvep/vep module. |
| modules/nf-core/ensemblvep/download/main.nf | Replace versions.yml with versions_* tuple outputs (topic versions); update container/deps. |
| modules/nf-core/ensemblvep/download/meta.yml | Document new versions tuple outputs and versions topic. |
| modules/nf-core/ensemblvep/download/environment.yml | Simplify dependency pins and add perl-math-cdf. |
| modules/nf-core/ensemblvep/download/tests/main.nf.test | Rename tests and switch to sanitized output snapshotting. |
| modules/nf-core/ensemblvep/download/tests/main.nf.test.snap | Snapshot updates reflecting new output structure and runtime metadata. |
| modules/nf-core/bcftools/sort/main.nf | Replace versions.yml with versions_bcftools tuple output (topic versions); update container string formatting; add max-mem option; adjust output naming. |
| modules/nf-core/bcftools/sort/meta.yml | Document versions_bcftools output and versions topic. |
| modules/nf-core/bcftools/sort/tests/main.nf.test | Update tests to assert versions_* outputs via findAll. |
| modules/nf-core/bcftools/sort/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple output and runtime metadata. |
| modules/nf-core/bcftools/merge/main.nf | Restructure inputs; replace versions.yml with versions_bcftools tuple output (topic versions); update container formatting. |
| modules/nf-core/bcftools/merge/meta.yml | Update module input docs (bed field) and add versions tuple outputs/topic docs. |
| modules/nf-core/bcftools/merge/tests/nextflow.config | Switch to parameterized args (params.args_modules). |
| modules/nf-core/bcftools/merge/tests/*.config | Remove per-test config files in favor of centralized config/params. |
| modules/nf-core/bcftools/concat/main.nf | Replace versions.yml with versions_bcftools tuple output (topic versions); update container formatting; refactor output extension logic. |
| modules/nf-core/bcftools/concat/meta.yml | Document versions tuple output and versions topic. |
| modules/nf-core/bcftools/concat/tests/main.nf.test | Update tests to assert versions_* outputs via findAll. |
| modules/nf-core/bcftools/concat/tests/main.nf.test.snap | Snapshot updates reflecting new versions tuple output and runtime metadata. |
| modules.json | Update tracked upstream git_shas and installed_by lists for multiple modules. |
Comments suppressed due to low confidence (2)
workflows/lrsomatic.nf:658
MULTIQCmodule input signature has changed to a single tuple input (meta + files + config(s) + logo + rename/sample files). This workflow still callsMULTIQC(...)with the old 6-argument form (includingch_multiqc_custom_configas a separate arg), which will fail at runtime. Update the call to build the expected tuple (e.g., combine default + custom configs into one list formultiqc_config) and adjust downstream handling sinceMULTIQC.out.report/data/plotsare now tuples[meta, path]rather than plain paths.
modules/nf-core/severus/main.nf:68- The
stubsection still creates files under${prefix}/...(e.g.${prefix}/severus.log,${prefix}/all_SVs/...), but the declared outputs (and the real script using--out-dir .) now expect these files in the task working directory (e.g.severus.log,all_SVs/...). In stub mode this will cause missing-output failures. Update the stub to create directories/files in.to match the updated output patterns.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+37
| { assert snapshot( | ||
| file(process.out.report[0][1]).name, | ||
| file(process.out.data[0][1]).name, | ||
| process.out.findAll { key, val -> key.startsWith("versions") | ||
| }).match() } |
There was a problem hiding this comment.
The findAll closure passed into snapshot(...) is missing its closing } (and the surrounding snapshot(...) call therefore has mismatched parentheses). This will make the test file fail to parse/run; close the closure and keep the parentheses balanced.
Comment on lines
+63
to
+67
| { assert snapshot( | ||
| file(process.out.report[0][1]).name, | ||
| file(process.out.data[0][1]).name, | ||
| process.out.findAll { key, val -> key.startsWith("versions") | ||
| }).match() } |
There was a problem hiding this comment.
Same parsing issue as above: the findAll { ... } closure is missing its closing } inside the snapshot(...) call, which will break this test case.
Comment on lines
+92
to
+97
| { assert snapshot( | ||
| file(process.out.report[0][1]).name, | ||
| file(process.out.data[0][1]).name, | ||
| file(process.out.plots[0][1]).name, | ||
| process.out.findAll { key, val -> key.startsWith("versions") | ||
| }).match() } |
There was a problem hiding this comment.
Same issue here: the process.out.findAll { key, val -> key.startsWith("versions") closure is not closed before the snapshot(...) call ends. This will cause a syntax error when running the tests.
Comment on lines
+125
to
+130
| { assert snapshot( | ||
| file(process.out.report[0][1]).name, | ||
| file(process.out.data[0][1]).name, | ||
| file(process.out.plots[0][1]).name, | ||
| process.out.findAll { key, val -> key.startsWith("versions") | ||
| }).match() } |
There was a problem hiding this comment.
And here as well: the findAll { ... } closure inside snapshot(...) is missing its closing brace, so this test file will not compile.
Comment on lines
+61
to
+70
| versions_bcftools: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - bcftools: | ||
| type: string | ||
| description: The tool name | ||
| - "bcftools --version | sed '1!d; s/^.*bcftools //'": | ||
| type: string | ||
| description: The command used to generate the version of the tool |
There was a problem hiding this comment.
The versions command entry is declared as type: string, but it represents an expression that should be evaluated to obtain the version (consistent with other modules using type: eval, e.g. samtools/index). As-is, nf-core schema/linting may treat this as the literal command string rather than an evaluated expression. Update the versions command field to type: eval (both under versions_bcftools and topics.versions).
Comment on lines
+70
to
+79
| versions_bcftools: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - bcftools: | ||
| type: string | ||
| description: The tool name | ||
| - "bcftools --version | sed '1!d; s/^.*bcftools //'": | ||
| type: string | ||
| description: The command used to generate the version of the tool |
There was a problem hiding this comment.
The versions command entry is declared as type: string, but it’s an evaluatable expression (command) used to obtain the bcftools version. This should be type: eval (and likewise under topics.versions) to match the convention used elsewhere (e.g. samtools/index/meta.yml).
Comment on lines
22
to
33
| - vcfs: | ||
| type: file | ||
| description: | | ||
| List containing 2 or more vcf files | ||
| e.g. [ 'file1.vcf', 'file2.vcf' ] | ||
| ontologies: [] | ||
| - tbis: | ||
| type: file | ||
| description: | | ||
| List containing the tbi index files corresponding to the vcfs input files | ||
| e.g. [ 'file1.vcf.tbi', 'file2.vcf.tbi' ] | ||
| ontologies: [] |
There was a problem hiding this comment.
In this meta.yml, vcfs/tbis are described as lists of files, but their type is still file. Since the module input is a list channel (path(vcfs), path(tbis)), the meta.yml should use type: list for these inputs to stay accurate and avoid nf-core lint issues.
Comment on lines
+79
to
+88
| versions_bcftools: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - bcftools: | ||
| type: string | ||
| description: The tool name | ||
| - "bcftools --version | sed '1!d; s/^.*bcftools //'": | ||
| type: string | ||
| description: The command used to generate the version of the tool |
There was a problem hiding this comment.
The versions command entry is declared as type: string, but it represents an expression/command that should be evaluated to obtain the version (see samtools/index/meta.yml which uses type: eval). Consider changing this to type: eval (and likewise under topics.versions) to align with the schema used by other updated modules.
Comment on lines
+181
to
+202
| versions_mosdepth: | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - mosdepth: | ||
| type: string | ||
| description: The tool name | ||
| - "mosdepth --version | sed 's/mosdepth //g'": | ||
| type: string | ||
| description: The command used to generate the version of the tool | ||
|
|
||
| topics: | ||
| versions: | ||
| - versions.yml: | ||
| type: file | ||
| description: File containing software versions | ||
| pattern: "versions.yml" | ||
| ontologies: | ||
| - edam: http://edamontology.org/format_3750 # YAML | ||
| - - ${task.process}: | ||
| type: string | ||
| description: The process the versions were collected from | ||
| - mosdepth: | ||
| type: string | ||
| description: The tool name | ||
| - "mosdepth --version | sed 's/mosdepth //g'": | ||
| type: string | ||
| description: The command used to generate the version of the tool |
There was a problem hiding this comment.
The versions command fields under versions_mosdepth / topics.versions are declared as type: string, but they are shell expressions intended to be evaluated to obtain the version. Other updated modules in this PR use type: eval for these entries (e.g. samtools/index/meta.yml). Update these to type: eval to keep the meta.yml consistent and avoid schema/lint issues.
…ements and removing versioning where it is done with topic channels
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).nextflow run . -profile debug,test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).