From aad5864fd688383d53fb95ae49e0162c9b75bf94 Mon Sep 17 00:00:00 2001 From: Carlos Gomes Date: Mon, 14 Jul 2025 16:29:15 -0300 Subject: [PATCH 1/3] fix: typo toc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 47834be..847250a 100755 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ NextFlow pipeline used by the Developmental Cognitive Neuroscience Lab (DCNL) to ## Table of Contents 1. [Getting Started](#getting-started) -1. [Pipeline paramters](#pipeline-parameters) +1. [Pipeline parameters](#pipeline-parameters) 1. [Pipeline output directory](#pipeline-output-directory) 1. [Examples](#examples) 1. [Useful links](#useful-links) -- GitLab From 7b87f64b9fa025ae36a18d6ad76b8a286856270a Mon Sep 17 00:00:00 2001 From: Carlos Gomes Date: Mon, 14 Jul 2025 16:50:21 -0300 Subject: [PATCH 2/3] up: clean-up procedure --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 847250a..dacad58 100755 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ NextFlow pipeline used by the Developmental Cognitive Neuroscience Lab (DCNL) to 1. [Getting Started](#getting-started) 1. [Pipeline parameters](#pipeline-parameters) 1. [Pipeline output directory](#pipeline-output-directory) +1. [Clean-up](#clean-up) 1. [Examples](#examples) 1. [Useful links](#useful-links) @@ -246,6 +247,18 @@ The pipeline also supports running [pre-configured parameter files](https://www. [top](#table-of-contents) +## Clean-up + +Each step will generate multiple files under the `work/` subdir. Those files are used to resume execution in case of an error but they are otherwise unnecessary. To prevent nextflow from using too much disk space, you should run `nextflow clean` after a successful execution. For more details, see [Cache & Resume](https://training.nextflow.io/2.0.1/basic_training/cache_and_resume/). + +For example, to clean temporary files but preserve logs (recommended), run the following: + +```sh +nextflow clean -f -k +``` + +Of note, if you're running the pipeline on an HPC environment managed by Slurm, you can leverage [Slurm's job dependency](https://slurm.schedmd.com/sbatch.html#OPT_dependency) to ensure that the clean-up will only take place after one or more jobs have finished running sucessfully. + ## Examples The following examples assume your current directory is the root directory of the project (`nanopore/`). -- GitLab From 1a7f33133de717408d21f40c502dc0da779c0474 Mon Sep 17 00:00:00 2001 From: Carlos Gomes Date: Tue, 15 Jul 2025 17:39:00 -0300 Subject: [PATCH 3/3] up: set cleanup to true in config --- README.md | 22 +++++----------------- src/nextflow.config | 3 +++ 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index dacad58..9aeb455 100755 --- a/README.md +++ b/README.md @@ -249,15 +249,7 @@ The pipeline also supports running [pre-configured parameter files](https://www. ## Clean-up -Each step will generate multiple files under the `work/` subdir. Those files are used to resume execution in case of an error but they are otherwise unnecessary. To prevent nextflow from using too much disk space, you should run `nextflow clean` after a successful execution. For more details, see [Cache & Resume](https://training.nextflow.io/2.0.1/basic_training/cache_and_resume/). - -For example, to clean temporary files but preserve logs (recommended), run the following: - -```sh -nextflow clean -f -k -``` - -Of note, if you're running the pipeline on an HPC environment managed by Slurm, you can leverage [Slurm's job dependency](https://slurm.schedmd.com/sbatch.html#OPT_dependency) to ensure that the clean-up will only take place after one or more jobs have finished running sucessfully. +Each step will generate multiple files under the `work/` subdir. Those files are used to resume execution in case of an error but they are otherwise unnecessary. To prevent nextflow from using too much disk space, the cleanup option is set to `true` in `src/nextflow.config`, which is equivalente to running `nextflow clean -f -k` after a successful run. the side effect of this is that you won't be able to resume a failed job, which is usually not an issue. however, if you do want to preserve the `-resume` option, make sure to comment out the config option or set it to `false`. For more details, see [Cache & Resume](https://training.nextflow.io/2.0.1/basic_training/cache_and_resume/). ## Examples @@ -290,8 +282,7 @@ The following examples assume your current directory is the root directory of th --basecall_compute "gpu" \ --basecall_demux "False" \ --queue_size 1 \ - --out_dir "$OUTPUT_DIR_NAME" \ - -resume + --out_dir "$OUTPUT_DIR_NAME" ``` 1. STEP 2A: Alignment Filtering and Quality Control from STEP 1 @@ -302,8 +293,7 @@ The following examples assume your current directory is the root directory of th --min_mapped_reads_thresh 500 \ --qscore_thresh 9 \ --mapq 10 \ - --step "2_from_step_1" \ - -resume + --step "2_from_step_1" ``` 1. STEP 2B (MinKNOW): Alignment Filtering and Quality Control from MinKNOW basecalling and alignment (bam files were generated by MinKNOW) @@ -315,8 +305,7 @@ The following examples assume your current directory is the root directory of th --is_barcoded "True" \ --qscore_thresh 9 \ --mapq 10 \ - --step "2_from_step_1" \ - -resume + --step "2_from_step_1" ``` 1. STEP 3: Methylation calling and MultiQC report @@ -325,8 +314,7 @@ The following examples assume your current directory is the root directory of th nextflow ./src/main.nf \ --steps_2_and_3_input_directory "./results/$OUTPUT_DIR_NAME/" \ --multiqc_config "./references/multiqc_config.yaml" \ - --step 3 \ - -resume + --step 3 ``` Alternatively, you can make use of an existing [parameter file](#parameter-files) to run one or more of the mentioned steps without the need to specify all parameters (each file is optimized for specific types of analysis). For example, the `parameters/human_blood/basecall.yaml` file runs the basecall analysis for a human blood sample, which can be executed from the project root directory as follows: diff --git a/src/nextflow.config b/src/nextflow.config index 41d2239..a93bda5 100755 --- a/src/nextflow.config +++ b/src/nextflow.config @@ -1,6 +1,9 @@ // MAIN CONFIGURATION FILE // see src/configs for other parameters +// clean the work/ dir after a successful run +cleanup = true + params { // Project name (used to identify which project you're working on) project_name = "default" -- GitLab