What is a negative binomial distribution in RNA-seq analysis?

It's the statistical model DESeq2 and edgeR use to describe gene-level count data, where variance is allowed to exceed the mean (overdispersion) instead of being locked to it like a Poisson distribution. Each gene gets its own mean and dispersion estimate fit from the replicate counts.

Why not just use Poisson for RNA-seq counts?

Poisson forces variance to equal the mean, but real biological replicates always show more spread than that from sources like RNA degradation and library prep variation. Using Poisson underestimates that spread and inflates the number of false positive differentially expressed genes.

What is the dispersion parameter in DESeq2?

It's the phi (or size) parameter controlling how much variance exceeds the mean for a given gene, following Var = mu + mu^2/phi. DESeq2 estimates it per gene with estimateDispersions() and shrinks noisy per-gene estimates toward a fitted trend before testing.

Is single-cell RNA-seq data zero-inflated?

Generally no. Analyses of droplet scRNA-seq negative control data show the observed zero counts match what a plain negative binomial model predicts, so reaching for a zero-inflated negative binomial adds complexity without matching the actual data-generating process.

When should you avoid negative binomial models like DESeq2?

When your sample size runs into the hundreds, such as comparing large TCGA cohorts, DESeq2's negative binomial assumptions and dispersion shrinkage can break down and inflate the false discovery rate. In that setting use a Wilcoxon rank-sum test or limma-voom instead.

Glossary · RNA-seq

Negative binomial model

The two-parameter distribution that lets DESeq2 and edgeR tell real fold changes apart from ordinary replicate noise.

By Ming "Tommy" Tang, Director of Bioinformatics in Big Pharma · Reviewed September 2026 · 2 min read

Also: NB distribution, NB GLM

Definition

The negative binomial (NB) distribution models count data whose variance is larger than its mean, a property called overdispersion. It has two parameters: a mean mu and a dispersion parameter phi (sometimes written as size, or reported as 1/phi), related by Var = mu + mu^2/phi. As phi grows toward infinity, the NB distribution converges to the Poisson distribution, where variance is forced to equal the mean. RNA-seq tools such as DESeq2 and edgeR fit a negative binomial generalized linear model (NB GLM) to each gene's counts, estimating both the mean expression level and a gene-specific dispersion from the replicate data.

You meet the negative binomial model the moment you run DESeq() or estimateDispersions() in DESeq2, or open an edgeR dispersion plot. It's the statistical engine behind almost every bulk RNA-seq differential expression call you've ever trusted, and it decides how much variability between replicates gets treated as noise versus signal.

The choice isn't cosmetic. A gene's raw counts bounce around between replicates for reasons that have nothing to do with your treatment: RNA degradation, library prep, PCR duplication. The NB model is what lets DESeq2 separate that biological and technical noise from a real fold change, and getting the dispersion estimate wrong in either direction changes which genes land on your hit list.

Why it matters

If you used a Poisson model instead of NB, you'd be assuming variance equals the mean, which is never true for real biological replicates. Poisson underestimates the spread between samples, so genes that simply jitter a bit between replicates get flagged as significant. That's exactly why RNA-seq DE tools moved to NB: methods built on it control false positives far better than Poisson-based approaches because the extra dispersion parameter absorbs the variance Poisson can't.

The flip side matters too. DESeq2's NB machinery is tuned for the classic small-study design, roughly 3-vs-3 or similar, with n of 6 or more per group as the recommended floor. Push it to hundreds of samples, like comparing LUAD versus LUSC in TCGA, and the NB assumptions and dispersion shrinkage start to break down, inflating your false discovery rate. In that regime, switch to a Wilcoxon rank-sum test or limma-voom instead of forcing DESeq2 to scale.

Where people get it wrong

The mistake people make with single-cell data is assuming the abundance of zeros means the data is "zero-inflated" and reaching for a zero-inflated negative binomial (ZINB) model. It usually isn't. Once you fit a plain NB with the right mean and dispersion, the predicted probability of a zero, P(X=0) = (phi/(mu+phi))^phi, already matches the observed zero rate in droplet scRNA-seq. The zeros are just what NB sampling looks like at low counts and modest sequencing depth, not evidence of a separate dropout process that requires a fatter model.

A concrete example

Simulate counts from a negative binomial distribution with a fixed mean and dispersion, then check that variance exceeds the mean the way the NB formula predicts, compared against a Poisson draw with the same mean.

mu <- 50
size <- 5   # dispersion parameter (phi); smaller = more overdispersed

nb_counts <- rnbinom(1000, mu = mu, size = size)
pois_counts <- rpois(1000, lambda = mu)

var(nb_counts)     # ~550, matches mu + mu^2/size = 50 + 2500/5
var(pois_counts)   # ~50, variance pinned to the mean

Related terms

Questions people ask

What is a negative binomial distribution in RNA-seq analysis?: It's the statistical model DESeq2 and edgeR use to describe gene-level count data, where variance is allowed to exceed the mean (overdispersion) instead of being locked to it like a Poisson distribution. Each gene gets its own mean and dispersion estimate fit from the replicate counts.
Why not just use Poisson for RNA-seq counts?: Poisson forces variance to equal the mean, but real biological replicates always show more spread than that from sources like RNA degradation and library prep variation. Using Poisson underestimates that spread and inflates the number of false positive differentially expressed genes.
What is the dispersion parameter in DESeq2?: It's the phi (or size) parameter controlling how much variance exceeds the mean for a given gene, following Var = mu + mu^2/phi. DESeq2 estimates it per gene with estimateDispersions() and shrinks noisy per-gene estimates toward a fitted trend before testing.
Is single-cell RNA-seq data zero-inflated?: Generally no. Analyses of droplet scRNA-seq negative control data show the observed zero counts match what a plain negative binomial model predicts, so reaching for a zero-inflated negative binomial adds complexity without matching the actual data-generating process.
When should you avoid negative binomial models like DESeq2?: When your sample size runs into the hundreds, such as comparing large TCGA cohorts, DESeq2's negative binomial assumptions and dispersion shrinkage can break down and inflate the false discovery rate. In that setting use a Wilcoxon rank-sum test or limma-voom instead.

Guide · How to Log-Transform Counts Without Fooling Yourself in Single-Cell RNA-seq

Sources

DESeq2: Differential expression of RNA-seq data using the Negative Binomial — NB GLM implementation, dispersion and log-fold-change prior estimation
Deseq2: RNA-seq and Negative binomial distribution — Two-parameter NB estimation per gene and the n>=6 per group recommendation
Negative binomial distribution in single-cell RNA-seq — Dispersion parameter estimate and the zero-count match showing scRNA-seq isn't zero-inflated
An evaluation of RNA-seq differential analysis methods — NB-based methods control false positives better than Poisson-based approaches