Should I remove an outlier sample from RNA-seq before running DESeq2?

Only if independent QC evidence, not just the PCA plot, backs it up: low mapped reads, low correlation with replicates, low RNA input, or a documented prep problem. Removing a sample purely because it improves your p-values is p-hacking. Confirm the technical cause first, then rerun the full pipeline without it.

How far from the cluster counts as an outlier in a bulk RNA-seq PCA plot?

A practical rule used in the field is a sample sitting more than 3 standard deviations from the group mean on a given PC, after converting PC coordinates to Z-scores. Treat that as a trigger to investigate with QC metrics, not as automatic grounds for exclusion.

What sample correlation indicates a failed RNA-seq library?

Replicates within a condition typically correlate above 0.80 on variance-stabilized counts. Published QC benchmarks put failing samples closer to a median correlation below 0.75 with the rest of the cohort, usually paired with under 25 million reads mapped to gene regions.

Can DESeq2 automatically remove outlier samples for me?

DESeq2 flags per-gene outliers using Cook's distance and can replace flagged values with replaceOutliers(), but that only triggers automatically for groups with 7 or more replicates by default (minReplicatesForReplace). It operates gene-by-gene, not sample-by-sample, so it won't decide whether to drop an entire sample for you.

Is it better to downweight a suspect sample than to drop it entirely?

If you can't confirm a clear technical failure and don't have a replicate to spare, yes. limma's voomWithQualityWeights() lets the model down-weight a noisy sample instead of discarding its data outright. Reserve outright removal for samples with confirmed QC failures across multiple independent metrics.

Sanity check · Bulk RNA-seq

How to Handle Outlier Samples in PCA in Bulk RNA-seq

PCA can't tell you whether a stray point is a dead library or your best result; your QC metrics and metadata have to do that job for it.

By Ming "Tommy" Tang, Director of Bioinformatics in Big Pharma · Reviewed September 2026 · 4 min read

You ran plotPCA() on your variance-stabilized counts, expecting three tidy clusters, and instead one point sits off by itself, closer to nothing than it is to its own replicates. Now you're staring at it wondering if it's a failed library, a swapped tube, or the most interesting sample in the experiment. The plot alone won't tell you. It never will: PCA summarizes total variance, and it has no way of knowing whether that variance is biology or a bad RNA extraction.

What's at stake is bigger than one dot on a plot. Drop the sample because it makes your p-values look better and you've p-hacked your differential expression call, whether you meant to or not. Keep a genuinely failed library in the model and you inflate dispersion estimates for every gene, which costs power for the whole experiment, not just that one sample's comparisons.

This page gives you an ordered set of checks, cheapest first, to turn "that point looks weird" into a documented, defensible decision, plus the fixes that follow from each answer and the one case where the right move is to leave the outlier exactly where it is.

What it looks like when it's happening

plotPCA() shows one sample sitting well outside the cluster formed by its own replicates, sometimes closer to the opposite condition than to its group
The sample-to-sample distance heatmap has one row/column that reads darker (larger distance) against every other sample, including its own replicates
Pairwise Pearson or Spearman correlation between the sample and its replicates on variance-stabilized counts is well under 0.80
DESeq2's per-sample Cook's distance boxplot is shifted noticeably higher for one sample across many genes
The alignment or counting QC report shows far fewer mapped reads or detected genes for that sample than the rest of the batch
Dropping the sample and rerunning the model roughly doubles the DE gene count or flips a borderline p-value
The sample also stands out on the wet-lab sheet: a lower RIN, lower RNA concentration, or lower library Qubit value than the rest of the cohort

Why it happens

plotPCA() by default runs on the 500 most variable genes in your variance-stabilized or rlog-transformed counts. That's a summary of total transcriptome-wide variance, technical and biological together, with no label attached to either source. A technical failure, degraded RNA, low input, over- or under-amplification during library prep, shifts expression broadly across thousands of genes at once, not the focused set of a hundred or so genes your actual biology usually moves. That broad, non-specific shift is exactly what dominates a principal component and pulls the point away from the rest of the cloud.

The trouble is that real biology produces the same signature. A different tissue region, an unusually aggressive tumor, a strong treatment responder, or a contaminating-but-real cell population all move expression broadly too, and they land in the same "far from the cluster" spot in PCA space. The plot cannot distinguish "this library was garbage" from "this sample is genuinely different," because both produce the same geometric symptom: large overall distance from its neighbors.

Depth compounds the problem. Uneven sequencing depth changes which low-expression genes are even detected across samples, and that alone can shift a sample's position on a PC axis before you've touched biology or QC (see [[364]]). And if library prep date or sequencing lane happens to align with your experimental groups, you don't get one stray outlier, you get systematic separation: PC1 splits by batch instead of phenotype, and no amount of sample-dropping fixes a confounded design (see [[384]]).

The checks

Run them in order. Each one tells you what healthy looks like and what the problem looks like.

0/9 checked · saved in this browser

01Plot PCA on transformed counts first, before any testing
Transform counts with vst() (datasets over ~20 samples) or rlog() (smaller n), then run plotPCA() with intgroup set to your experimental factor, before you fit any DE model.
r
```
vsd <- vst(dds, blind = TRUE)
plotPCA(vsd, intgroup = "condition", ntop = 500)
```
Healthy
Samples cluster tightly by condition/replicate group; no point sits noticeably outside its own group's cloud.
Red flag
One sample sits far from its replicates on PC1 or PC2, sometimes nearer the other condition's cluster than its own.
02Build a sample-to-sample distance heatmap
Compute Euclidean distance on the transformed count matrix and visualize it as a heatmap, ordered by sample.
r
```
sampleDists <- dist(t(assay(vsd)))
pheatmap::pheatmap(as.matrix(sampleDists))
```
Healthy
Block-diagonal pattern: samples within a condition are close (dark/low distance) to each other and further from other conditions.
Red flag
One sample's row/column reads uniformly far from every other sample, including its own replicates.
03Quantify how far is far: Z-score the flagged PC
Pull PCA coordinates with returnData=TRUE, convert the suspect PC to Z-scores, and check whether any sample exceeds 3 SD from the mean.
r
```
pca_data <- plotPCA(vsd, intgroup = "condition", returnData = TRUE)
z <- scale(pca_data$PC1)
pca_data$name[abs(z) > 3]
```
Healthy
No sample exceeds |Z| > 3 on the PC that separates it from the rest.
Red flag
A sample sits beyond 3 standard deviations from the group mean on that PC.
04Check pairwise correlation against replicates
Compute Pearson (or Spearman) correlation across all samples on the transformed matrix and pull out the suspect sample's correlation with its own replicates specifically.
r
```
cor_mat <- cor(assay(vsd), method = "pearson")
cor_mat["suspect_sample", ]
```
Healthy
Correlation with its own replicates above 0.80.
Red flag
Correlation with replicates below 0.80, and closer to 0.75 or lower matches published thresholds for QC-failed samples.
05Pull mapped reads and detected gene counts from the alignment QC report
Check the STAR Log.final.out, featureCounts summary, or MultiQC report for total reads assigned to gene features and the number of genes detected at TPM > 4 for that sample versus the rest of the batch.
Healthy
Roughly in line with the batch: published QC benchmarks use ≥25 million reads mapped to gene regions and ≥11,400 detected genes at TPM > 4 as passing thresholds.
Red flag
Mapped reads under 25M or detected genes under 11,400 for that sample specifically, while the rest of the batch clears both.
06Cross-check pre-sequencing lab metrics and batch metadata
Look up the sample's RIN, RNA concentration, and library Qubit value on the wet-lab QC sheet, and compare its prep date, extraction batch, and operator against the rest of the cohort.
Healthy
Metrics and batch assignment consistent with the passing samples in the cohort.
Red flag
Noticeably lower RNA concentration or Qubit value than passing samples (published FFPE data shows failing samples at a median ~18.9 ng/uL input vs ~40.8 ng/uL passing, and ~2.08 vs 5.82 ng/uL on library Qubit), or a prep date/batch that also separates on PC1.
07Rule out a sample swap with sex-linked marker genes
If a swap is even plausible, plot XIST expression against a Y-linked gene (RPS4Y1 or DDX3Y) per sample and compare the pattern to the donor sex recorded in metadata.
Healthy
XIST/Y-gene expression pattern matches the recorded sex for every sample.
Red flag
The outlier sample's sex-linked expression contradicts its recorded metadata, pointing to a mislabeled tube at the core.
08Inspect per-sample Cook's distance from the DESeq2 fit
After fitting the model, pull the Cook's distance matrix and boxplot it per sample on a log scale; DESeq2's internal outlier flag uses the 99% quantile of the F(p, m-p) distribution.
r
```
boxplot(log10(assays(dds)[["cooks"]]), range = 0, las = 2)
# threshold reference: qchisq(0.99, df = p) as an approximation
```
Healthy
Cook's distances distributed similarly across all samples.
Red flag
One sample's boxplot sits clearly above the rest, meaning it's driving fit residuals for an unusually large number of genes.
09Rerun the DE analysis with and without the sample
Fit the full DESeq2 or edgeR model twice with identical design and thresholds, once with the suspect sample included and once excluded, and compare the resulting gene lists and top hits.
Healthy
Broadly similar results whether the sample is in or out.
Red flag
Significant gene count roughly doubles, or a key hit appears/disappears, purely based on that one sample's presence.

What to do about it

Drop the sample after confirming a technical failure

When: QC metrics you check independent of the DE result, low mapped reads, low detected gene count, correlation with replicates below 0.75, low RNA input, or a documented prep problem, agree with the PCA outlier call.

Exclude the sample from the count matrix and colData before building the DESeq2/edgeR object, rerun the entire pipeline (normalization, dispersion estimation, testing) from scratch, and record the excluded sample and the specific metric that failed in your methods.

Caveat: You lose a replicate. At n=3 per group that leaves one condition at n=2, and dispersion estimates get noisier. Check the study still has power to answer the question before committing.

Downweight instead of dropping when the call is borderline

When: The sample fails one or two QC checks but not decisively, or you can't afford to lose a replicate.

Use limma::voomWithQualityWeights() to let the model estimate a per-sample weight from the data, or run DESeq2's replaceOutliers() deliberately (it only auto-triggers for groups with 7+ replicates via minReplicatesForReplace, so set that parameter explicitly for smaller designs).

Caveat: Automatic weighting absorbs extra variance, it won't catch a full sample swap or a genuine biological subgroup.

Verify identity before deciding anything, if a swap is plausible

When: The outlier looks nothing like its assigned condition, or workflow records suggest tubes could have been mixed up at the sequencing core.

Check sex-linked marker genes (XIST vs RPS4Y1/DDX3Y) against recorded donor sex, or genotype concordance against matching SNP/DNA-seq data if available.

Caveat: This only catches swaps that cross a detectable label. A swap between two same-sex donors in the same condition will look clean on this test.

Fix the design, not the sample, when it's really a batch effect

When: PC1 or PC2 separates prep date, lane, or operator rather than isolating one sample, per [[384]].

Add batch as a covariate in the design formula (~ batch + condition) and let the GLM absorb it, rather than pre-correcting the expression matrix with removeBatchEffect() before testing.

Caveat: Only works if batch and condition aren't fully confounded. If every condition-A sample was prepped on one date and every condition-B sample on another, no formula term fixes that; you need to re-sequence with a randomized design.

Report both results when the conclusion depends on the sample

When: Removing the sample changes the outcome meaningfully: the gene list roughly doubles, or a key hit appears or disappears.

Run the analysis with and without the sample and present both, flagging the finding as sensitive to that one data point rather than reporting whichever version you prefer.

Caveat: More honest than picking a side, but it doesn't resolve the question. Follow up with additional replicates or a repeat library if the sample can be resequenced.

When not to "fix" it

Sometimes the outlier is the answer, not the noise. In TCGA lung cancer cohorts, some LUSC samples mix with the LUAD cluster on PCA, and that overlap reflects real, previously documented ambiguity between the two histological calls, not a bad library. If your outlier is a strong treatment responder, an unusually aggressive tumor, or a contaminating-but-real cell population, dropping it to tighten the PCA plot removes the exact signal you set out to find. Before excluding anything, ask what result you're protecting: if the only problem with the sample is that it makes your p-value worse, that's not a QC failure, that's discarding data until the test agrees with you.

Five things experienced analysts do here

Run plotPCA() before you fit a single differential expression model, not after you've already seen a p-value you don't like. Deciding on outliers post-hoc is p-hacking with extra steps.
Keep pre-sequencing QC (RIN, RNA concentration, library Qubit) attached to your sample sheet from day one. Without it, an outlier is a mystery instead of a five-minute lookup.
Always rerun the entire pipeline after dropping a sample, normalization, dispersion estimation, and testing all shift when sample count changes. Don't just re-filter an existing results table.
Document every excluded sample in the methods with the specific metric that failed. 'Removed sample X: 18M mapped reads, r=0.62 with replicates' survives review; 'removed an outlier' does not.
When in doubt, run the DE analysis both ways. If your conclusion depends on one sample being in or out, that's telling you about your study's power, not just about that sample.

Questions people ask

Should I remove an outlier sample from RNA-seq before running DESeq2?: Only if independent QC evidence, not just the PCA plot, backs it up: low mapped reads, low correlation with replicates, low RNA input, or a documented prep problem. Removing a sample purely because it improves your p-values is p-hacking. Confirm the technical cause first, then rerun the full pipeline without it.
How far from the cluster counts as an outlier in a bulk RNA-seq PCA plot?: A practical rule used in the field is a sample sitting more than 3 standard deviations from the group mean on a given PC, after converting PC coordinates to Z-scores. Treat that as a trigger to investigate with QC metrics, not as automatic grounds for exclusion.
What sample correlation indicates a failed RNA-seq library?: Replicates within a condition typically correlate above 0.80 on variance-stabilized counts. Published QC benchmarks put failing samples closer to a median correlation below 0.75 with the rest of the cohort, usually paired with under 25 million reads mapped to gene regions.
Can DESeq2 automatically remove outlier samples for me?: DESeq2 flags per-gene outliers using Cook's distance and can replace flagged values with replaceOutliers(), but that only triggers automatically for groups with 7 or more replicates by default (minReplicatesForReplace). It operates gene-by-gene, not sample-by-sample, so it won't decide whether to drop an entire sample for you.
Is it better to downweight a suspect sample than to drop it entirely?: If you can't confirm a clear technical failure and don't have a replicate to spare, yes. limma's voomWithQualityWeights() lets the model down-weight a noisy sample instead of discarding its data outright. Reserve outright removal for samples with confirmed QC failures across multiple independent metrics.

Guide · How to Detect Batch Effects in Bulk RNA-seq
Guide · How to Handle Multiple Testing and FDR in Bulk RNA-seq
Guide · How to Log-Transform Counts Without Fooling Yourself in Bulk RNA-seq
Guide · Why You Must Not Use TPM for Differential Expression in Bulk RNA-seq
Compare · DESeq2 vs edgeR: Which One Should You Use?
Glossary · Count matrix
Glossary · Heatmap
Glossary · Z-score

Sources

DESeq2 Vignette — vst/rlog transformation guidance and plotPCA usage
QC Methods for DE Analysis using DESeq2 — Sample distance heatmap construction and the >0.80 replicate correlation guideline
plotPCA: Sample PCA plot for transformed data in DESeq2 — ntop=500 default and returnData option used in the Z-score check
replaceOutliers: Replace outliers with trimmed mean in DESeq2 — minReplicatesForReplace behavior cited in the downweighting fix
Deseq2 default Cooks distance threshold — Cook's distance threshold and per-sample boxplot check
Outlier detection of RNAseq samples (Bioconductor Support) — Distance-based outlier detection and voomWithQualityWeights alternative
Robust principal component analysis for accurate outlier sample detection in RNA-Seq data — PcaGrid/PcaHubert performance versus classical PCA
Identifying outliers in RNA-Seq Data... To Remove or Not to Remove? — Z-score >3SD rule and decision framework for technical vs biological outliers
Quality control recommendations for RNASeq using FFPE samples — RNA concentration and library Qubit thresholds for QC-failed samples
Integration of Bulk RNA-seq Pipeline Metrics for Assessing Low-Quality Samples — Mapped read and detected gene count thresholds

Part of the Outlier samples in PCA series.