LiftSignals is live. Every plan starts with a 30-day free trial — no credit card required. Start free →
Free tool · coming soon

Holdout sample size calculator

Coming soon: a free calculator that tells you how big a holdout group your campaign test needs. Until it’s live, here’s the sample size formula, a worked holdout example and a spreadsheet version you can use before your next send.

One email when it launches. No newsletter.

PREVIEW · EXAMPLE NUMBERS

Coming soon
4.0%
5.9%
10%
95% · 80%
Holdout needed≈ 970

The formula

The holdout sample size formula.

n = (zα/2 + zβ)² × ( p₁(1−p₁) + p₂(1−p₂) ÷ k ) ÷ (p₂ − p₁)²

p₁ and p₂
The conversion rate without the campaign (holdout) and the rate you hope to see with it.
z values
1.96 for 95% confidence and 0.84 for 80% power. Together, (1.96 + 0.84)² ≈ 7.84.
k
Customers sent per customer held out. A 10% holdout means k = 9.
n
Customers needed in the holdout. The full audience is n ÷ holdout share.

Worked example

A holdout test, sized step by step.

Uses the rates from the illustrative holdout test on our homepage: 4.0% without the email, 5.9% with it.

01

Variance term0.04 × 0.96 + (0.059 × 0.941) ÷ 9

0.0446

02

Effect size, squared(0.059 − 0.040)²

0.000361

03

Holdout needed7.84 × 0.0446 ÷ 0.000361

≈ 970

04

Full audience at a 10% holdout970 ÷ 0.10

≈ 9,700

Use it today

Holdout sample size in a spreadsheet.

Change the rates and holdout share in column B; the sizes recalculate.

AB
1Rate without campaign0.04
2Rate hoped for0.059
3Holdout share0.1
4Holdout needed=7.84*(B1*(1-B1)+B2*(1-B2)/((1-B3)/B3))/(B2-B1)^2
5Full audience=B4/B3

Works the same in Excel and Google Sheets. Formulas in orange.

Where the simple version breaks

Where hand-sized tests go wrong.

Small lifts need big audiences

Halve the lift you expect and you need about four times as many customers. Many email lists are simply too small for subtle tests.

Peeking ruins the math

Checking results daily and stopping when they look good inflates false wins. Decide the window before you send.

One test isn’t a program

Sizing, randomizing, excluding the holdout everywhere and reading the result for every campaign is where manual testing stops.

LiftSignals keeps a random holdout on every audience and reports caused revenue with the confidence interval. Incrementality testing →

FAQ · holdout sample size calculator

Holdout sample size questions, answered.

Why can’t I just hold out 100 customers?

Because random noise in 100 customers is larger than most campaign effects. Too small a holdout gives results that can’t be told apart from chance.

Is a bigger holdout always better?

A bigger holdout reaches a clear answer sooner but withholds the campaign from more customers. Between 10% and 20% is a common balance.

What do 95% confidence and 80% power mean?

95% confidence limits false wins to about 1 in 20. 80% power means a real effect of the size you expect will be detected 4 times in 5.

When will the holdout calculator be available?

It’s on the way. Leave your email above and we’ll send one note when it launches.

Is this the same as an A/B test sample size?

Yes, it’s the same math. A holdout test is an A/B test where version B is “no message”.