Holdout sample size calculator
Coming soon: a free calculator that tells you how big a holdout group your campaign test needs. Until it’s live, here’s the sample size formula, a worked holdout example and a spreadsheet version you can use before your next send.
One email when it launches. No newsletter.
The formula
The holdout sample size formula.
n = (zα/2 + zβ)² × ( p₁(1−p₁) + p₂(1−p₂) ÷ k ) ÷ (p₂ − p₁)²
- p₁ and p₂
- The conversion rate without the campaign (holdout) and the rate you hope to see with it.
- z values
- 1.96 for 95% confidence and 0.84 for 80% power. Together, (1.96 + 0.84)² ≈ 7.84.
- k
- Customers sent per customer held out. A 10% holdout means k = 9.
- n
- Customers needed in the holdout. The full audience is n ÷ holdout share.
Worked example
A holdout test, sized step by step.
Uses the rates from the illustrative holdout test on our homepage: 4.0% without the email, 5.9% with it.
Variance term0.04 × 0.96 + (0.059 × 0.941) ÷ 9
0.0446
Effect size, squared(0.059 − 0.040)²
0.000361
Holdout needed7.84 × 0.0446 ÷ 0.000361
≈ 970
Full audience at a 10% holdout970 ÷ 0.10
≈ 9,700
Use it today
Holdout sample size in a spreadsheet.
Change the rates and holdout share in column B; the sizes recalculate.
| A | B | |
|---|---|---|
| 1 | Rate without campaign | 0.04 |
| 2 | Rate hoped for | 0.059 |
| 3 | Holdout share | 0.1 |
| 4 | Holdout needed | =7.84*(B1*(1-B1)+B2*(1-B2)/((1-B3)/B3))/(B2-B1)^2 |
| 5 | Full audience | =B4/B3 |
Works the same in Excel and Google Sheets. Formulas in orange.
Where the simple version breaks
Where hand-sized tests go wrong.
Small lifts need big audiences
Halve the lift you expect and you need about four times as many customers. Many email lists are simply too small for subtle tests.
Peeking ruins the math
Checking results daily and stopping when they look good inflates false wins. Decide the window before you send.
One test isn’t a program
Sizing, randomizing, excluding the holdout everywhere and reading the result for every campaign is where manual testing stops.
LiftSignals keeps a random holdout on every audience and reports caused revenue with the confidence interval. Incrementality testing →
FAQ · holdout sample size calculator
Holdout sample size questions, answered.
Why can’t I just hold out 100 customers?
Because random noise in 100 customers is larger than most campaign effects. Too small a holdout gives results that can’t be told apart from chance.
Is a bigger holdout always better?
A bigger holdout reaches a clear answer sooner but withholds the campaign from more customers. Between 10% and 20% is a common balance.
What do 95% confidence and 80% power mean?
95% confidence limits false wins to about 1 in 20. 80% power means a real effect of the size you expect will be detected 4 times in 5.
When will the holdout calculator be available?
It’s on the way. Leave your email above and we’ll send one note when it launches.
Is this the same as an A/B test sample size?
Yes, it’s the same math. A holdout test is an A/B test where version B is “no message”.
MORE FREE TOOLS ON THE WAY
Coming soon
CLV calculator →
Predict what a customer is worth over their lifetime.
Coming soon
Churn rate calculator →
Churn and retention rate for any period.
Coming soon
RFM analysis tool →
Score customers on recency, frequency and spend.
Coming soon
LTV:CAC calculator →
Is each customer worth what you paid for them?
Get a holdout on every campaign, sized for you.
LiftSignals runs the test and reports what each campaign caused. 30 days free.