Glossary · Reproducibility and Workflows
Nextflow
The workflow engine that turns "it ran on my laptop" into "it ran the same way on 10,000 samples across three clusters."
By Ming "Tommy" Tang, Director of Bioinformatics in Big Pharma · Reviewed September 2026 · 3 min read
Also: nf-core
Definition
Nextflow is a workflow management system and domain-specific language for chaining command-line tools, scripts, and containers into a single reproducible pipeline. It is process-oriented: you define independent processes with declared inputs and outputs, and Nextflow wires them together with channels, asynchronous queues that stream data between steps without you managing intermediate files by hand. Parallelization falls out of the channel model automatically, and the same pipeline runs unchanged on a laptop, a SLURM cluster, or AWS/GCP/Azure by swapping an executor profile. nf-core is the community layer built on top: a curated, MIT-licensed collection of Nextflow pipelines that must pass CI testing and a fixed checklist before release.
You meet Nextflow the moment an RNA-seq analysis stops being "run FastQC, then STAR, then featureCounts by hand" and becomes "run this on 200 samples tonight without babysitting it." Most people meet it first as nf-core/rnaseq: you point nextflow run nf-core/rnaseq at a samplesheet and a profile, and it handles trimming, alignment, and quantification as one unit instead of five scripts stitched together with shell loops.
The decision Nextflow forces on you early is whether to write your own pipeline in its DSL or run an existing nf-core pipeline and only write the glue around it. For standard assays, RNA-seq, ChIP-seq, variant calling, use nf-core; don't rebuild what over 8,000 community members already tested. Reach for custom Nextflow only when your assay is genuinely non-standard.
Why it matters
Get this right and a failed job on sample 4,832 out of 10,000 costs you minutes: fix the input, rerun with -resume, and Nextflow picks up from the last successful step instead of redoing everything upstream. Get it wrong, a pipeline built from un-resumable shell scripts, and the same failure means restarting the whole batch, which at genome scale means redoing weeks of compute. Lesson 64 describes exactly this failure mode: processing 1,000 WGS samples at MD Anderson took three months on an HPC, and a single bad job with no workflow manager could stall the entire run.
The other consequence is portability. A pipeline written against Nextflow's executor abstraction runs unchanged on a laptop with -profile docker, on a SLURM cluster with -profile slurm, or on AWS Batch with -profile awsbatch. Hard-code paths and job-submission commands into a bash script instead, and every environment move becomes a rewrite.
Where people get it wrong
The mistake is treating "Nextflow" and "nf-core" as interchangeable, or assuming any Nextflow pipeline on GitHub has nf-core's guarantees. Nextflow is the engine: the DSL, the channel/process model, the executors. nf-core is a community standard built on top of it, MIT-licensed, Docker-containerized with versioned software, CI-tested, released under stable tags, and only pipelines that pass that checklist carry the nf-core name. A pipeline someone wrote in Nextflow and pushed to their own repo has none of those guarantees unless it's actually in the nf-core catalogue.
The second mixup is Nextflow vs Snakemake. Snakemake is file-oriented: a rule is defined by the file it produces, and the DAG is built by matching filenames. Nextflow is process-oriented: data moves through channels without you naming every intermediate file, and a process runs once per item a channel emits. People default to whichever their lab already uses, then get stuck translating a Snakemake "rule that outputs X.bam" into Nextflow's channel model, or vice versa, because the two frameworks build their DAGs from opposite starting points.
A concrete example
Running the nf-core/rnaseq test profile end to end, killing it partway through on purpose, then re-running with -resume shows Nextflow skip every process that already completed and start again only from the failed step forward, the behavior that saves a 10,000-sample run from a full restart.
nextflow run nf-core/rnaseq -profile docker,test --outdir results
# job dies partway through (bad node, OOM, bad input)
# fix the cause, then re-run the identical command with -resume:
nextflow run nf-core/rnaseq -profile docker,test --outdir results -resume
# completed processes are read from the cache; only downstream steps re-executeRelated terms
Questions people ask
- Is Nextflow the same thing as nf-core?
No. Nextflow is the workflow engine and DSL; nf-core is a community-maintained collection of pipelines built on Nextflow that must meet a fixed checklist (MIT license, Docker containers, CI testing, versioned releases) before they carry the nf-core name. You can write pipelines in Nextflow that are not nf-core, and none of the quality guarantees apply to those.
- Nextflow vs Snakemake: which should I learn first?
Learn whichever your lab or collaborators already use, since the value is in shared, reproducible pipelines, not the specific tool. If you're starting from zero and your work is mostly standard assays like RNA-seq or variant calling, Nextflow has the edge because of nf-core's large catalogue of pre-built, tested pipelines.
- Do I need to know Groovy to use Nextflow?
To run an existing nf-core pipeline, no, you only need the command line and a samplesheet. Nextflow's DSL is built on Groovy, and you'll need at least basic Groovy syntax once you start writing your own processes or modifying pipeline logic.
- What exactly does -resume skip?
Nextflow hashes each process's inputs, script, and parameters, and caches the outputs of every task that completes successfully. On a re-run with -resume, any task whose hash matches a cached entry is skipped entirely, and only tasks downstream of a changed or failed step re-execute.
- Can the same Nextflow pipeline run on a laptop and then on AWS without rewriting it?
Yes, if the pipeline uses Nextflow's executor and container abstractions instead of hard-coded paths or job-submission commands. Switching from a laptop to SLURM, LSF, or AWS Batch is a matter of changing the -profile or config file, not the pipeline code itself.
Related pages
Related reading on the blog
Sources
- Nextflow - A DSL for parallel and scalable computational pipelines — Process/channel model, executors, container support, resume behavior
- nf-core: What is nf-core? — nf-core community standards: MIT license, containers, CI testing, release tags
- nf-core/rnaseq RNA sequencing analysis pipeline — What nf-core/rnaseq does end to end, used in the worked example
- Seqera: nf-core rnaseq pipeline — Process-oriented vs file-oriented comparison with Snakemake; 83% deployment stat