Add NGSCheckMate #993

SPPearce · 2023-03-31T15:34:54Z

This PR adds the NGSCheckMate tool to the pipeline. I find this an essential tool and part of my initial QC for all sequencing we perform. This tool takes a bed file with a set of SNPs and tries to determine if samples come from the same individuals. Developed and generally used in humans (there are SNP bed files for hg19/hg38/GRCh37/GRCh38 available), but could be used in other species if a suitable set of common SNPs was provided.
This should be configured to run automatically on GRCh37/38, hg37/hg38 I think needs the files adding to igenomes - need to check.

This ensures that no sample swaps have occurred by checking:

Replicates come from the same individual
Paired samples (e.g. timepoints or tissue samples) come from the same individual
Non-paired samples didn't come from the same individual!
It works on human derived cell lines, but won't separate different treatments applied to the same cell line.

Testing.
I have used a bed file on the test datasets to get it to run on yeast, haven't tried the full tests yet.

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [X] If necessary, also make a PR on the nf-core/rnaseq branch on the nf-core/test-datasets repository.
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

github-actions · 2023-06-08T16:32:28Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit a4a946c

+| ✅ 145 tests passed       |+
#| ❔   6 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in WorkflowRnaseq.groovy: Optionally add in-text citation tools to this list.

❔ Tests ignored:

files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml
multiqc_config - multiqc_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-rnaseq_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-rnaseq_logo_light.png
files_exist - File found: docs/images/nf-core-rnaseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: lib/WorkflowRnaseq.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-rnaseq_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.14.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (268 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.10
Run at 2023-11-25 17:13:18

…o ngscheckmate

pinin4fjords

Inclusion probably a call for @drpatelh, but a couple of minor comments in the meantime

pinin4fjords · 2023-11-21T20:08:26Z

workflows/rnaseq.nf

@@ -768,6 +772,15 @@ workflow RNASEQ {
            ch_versions = ch_versions.mix(DUPRADAR.out.versions.first())
        }

+        if (params.ngscheckmate_bed) {
+            BAM_NGSCHECKMATE (


Is there any way the output could be baked into the MultiQC report?

That has been suggested over in sarek too, something I could consider.

pinin4fjords · 2023-11-21T20:09:10Z

docs/output.md

+
+</details>
+
+[NGSCheckMate](https://github.com/parklab/NGSCheckMate) is a tool to verify that samples come from the same individual, by examining a set of single nucleotide polymorphisms (SNPs). This calculates correlations between the samples, and then applies a depth-dependent model of allele fractions to call samples as being related or not. The principal output is a dendrogram, where samples that are .


Unfinished sentence

pinin4fjords · 2023-11-21T20:10:44Z

subworkflows/nf-core/bam_ngscheckmate/main.nf

+    ch_versions = ch_versions.mix(BCFTOOLS_MPILEUP.out.versions)
+
+    BCFTOOLS_MPILEUP
+    .out


Indent the ops (here and below)?

This is in the nf-core subworkflow, but I'll try and remember to do it when I convert to nf-test.

pinin4fjords · 2023-11-23T14:07:24Z

@SPPearce - @drpatelh is concerned that we take care to ensure that the new input bed file is compatible with other reference inputs (coordinates, 'chr' prefixes etc). Can we incorporate a check for those things?

SPPearce · 2023-11-23T14:17:51Z

@SPPearce - @drpatelh is concerned that we take care to ensure that the new input bed file is compatible with other reference inputs (coordinates, 'chr' prefixes etc). Can we incorporate a check for those things?

Hmm, we could check for the existence of chr mismatching, that would be relatively simple I guess?
A genome build mismatch (hg37/hg38) would probably just end up with most of the samples being considered as matching (the bed file would point to the wrong locations, the ~25% of which have the correct reference allele in the SNP bed file would then all probably be ref in all the samples so all would be considered homozygous)

Add NGSCheckMate

dc069ed

SPPearce changed the base branch from master to dev March 31, 2023 15:37

nf-core deleted a comment from github-actions bot Mar 31, 2023

SPPearce and others added 4 commits March 31, 2023 16:52

Move module config into if statement

fdbef43

Update modules.json

bb43740

Update to newer versions of modules

a129ebe

Merge branch 'dev' into ngscheckmate

38abdf8

SPPearce added 4 commits June 8, 2023 17:36

Add new parameter to config

289c3d3

Merge branch 'ngscheckmate' of https://github.com/SPPearce/rnaseq int…

9eef3f9

…o ngscheckmate

Update documentation and output location

2a25235

Update PR number

873087f

SPPearce requested a review from drpatelh June 9, 2023 09:33

SPPearce added the Ready for review label Jun 9, 2023

SPPearce and others added 7 commits June 9, 2023 10:35

Prettier

753e3e7

Merge branch 'dev' into ngscheckmate

2121eba

Update NGSCheckMate versions

f75835a

Add NGSCheckMate bed file path in igenomes

793c263

Fix metas and add to test config

2afbd2d

Remove the diff

1641a68

Merge branch 'dev' into ngscheckmate

2964132

SPPearce requested a review from pinin4fjords November 20, 2023 20:02

pinin4fjords reviewed Nov 21, 2023

View reviewed changes

SPPearce added 2 commits November 24, 2023 20:31

Finish sentene

ddf7d73

Merge branch 'dev' into ngscheckmate

a4a946c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add NGSCheckMate #993

Add NGSCheckMate #993

SPPearce commented Mar 31, 2023 •

edited

Loading

github-actions bot commented Jun 8, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

pinin4fjords left a comment

pinin4fjords Nov 21, 2023

SPPearce Nov 24, 2023

pinin4fjords Nov 21, 2023

pinin4fjords Nov 21, 2023

SPPearce Nov 24, 2023

pinin4fjords commented Nov 23, 2023

SPPearce commented Nov 23, 2023


		</details>

		[NGSCheckMate](https://github.com/parklab/NGSCheckMate) is a tool to verify that samples come from the same individual, by examining a set of single nucleotide polymorphisms (SNPs). This calculates correlations between the samples, and then applies a depth-dependent model of allele fractions to call samples as being related or not. The principal output is a dendrogram, where samples that are .

Add NGSCheckMate #993

Are you sure you want to change the base?

Add NGSCheckMate #993

Conversation

SPPearce commented Mar 31, 2023 • edited Loading

github-actions bot commented Jun 8, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

pinin4fjords left a comment

Choose a reason for hiding this comment

pinin4fjords Nov 21, 2023

Choose a reason for hiding this comment

SPPearce Nov 24, 2023

Choose a reason for hiding this comment

pinin4fjords Nov 21, 2023

Choose a reason for hiding this comment

pinin4fjords Nov 21, 2023

Choose a reason for hiding this comment

SPPearce Nov 24, 2023

Choose a reason for hiding this comment

pinin4fjords commented Nov 23, 2023

SPPearce commented Nov 23, 2023

SPPearce commented Mar 31, 2023 •

edited

Loading

github-actions bot commented Jun 8, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️