Email holdout test: how to prove what a Klaviyo flow actually earns
Your email platform says the flow made $9,000. A coin flip and a little patience will tell you whether it made $2,600.
Key takeaways
- An email holdout test sends a flow to a random group and withholds it from another, so the gap in orders is what the email caused.
- Klaviyo's attributed revenue gives credit to the last eligible open or click within a window, five days by default, whether or not the email changed anything.
- Klaviyo's global holdout groups need at least 400,000 profiles; smaller stores can test one flow with a random sample split.
- Compare all orders by group, not attributed orders, opens or clicks.
- A holdout too small to read is the worst option: you give up sales and still learn nothing.
An email holdout test tells you how much revenue an email or flow actually caused. You split the people who would get it at random, send it to one group, withhold it from the other, then compare orders. Because the groups were picked by chance, the held-out group shows what would have happened anyway. This guide shows how to run an email holdout test in Klaviyo, whether or not your account qualifies for a global holdout group, how big the holdout has to be, and how to read the result without fooling yourself.
Why your email platform's revenue number runs high
Klaviyo's revenue figures are attribution, not proof. Its message attribution guide says 100% of conversion credit goes to the last eligible touchpoint before the conversion. For email, a touchpoint is an open or a click, and by default the window for both is five days (checked Nov 18, 2026).
So if a customer opens your abandoned checkout email and buys within five days, the email gets the sale. It gets the sale if they were always going to come back after payday. It gets the sale if they only opened it to delete it. Attribution answers "what came last?", not "what made the difference?"
Opens make this worse. Apple Mail Privacy Protection can register opens that no human made, which Klaviyo's own flow testing guide warns may lead to inflated open rates. Klaviyo's attribution settings include an option to remove Apple Privacy Protection opens from attribution, and they note this does not remove them from reporting. It's worth switching on, but it doesn't turn attribution into causation.
This isn't a Klaviyo flaw. Any last-touch model hands out credit this way, and it's useful for comparing one email with another. It's just not the number to put in a board deck as "revenue email generated", unless you enjoy being asked why the channels add up to 140% of sales.
Treat attributed revenue as a ranking tool. When you need to know what a flow is worth, run a holdout.
Option 1: Klaviyo global holdout groups
If your account is big enough, Klaviyo has a built-in way to measure email incrementality across your whole program. According to Klaviyo's global holdout groups guide (updated Oct 29, 2026):
| What | How it works |
|---|---|
| Eligibility | At least 400,000 profiles in total |
| What's held back | Campaigns and flows on every channel, except transactional flow messages and anything you override |
| Holdout size | Any percentage above 0 and below 100; Klaviyo recommends about 5% of marketable profiles. It can't be changed once active |
| Duration | Klaviyo recommends about three months |
| Results | Conversion rates, lift and a win probability, counting conversions whether or not they were attributed to a message |
| Limit | One active holdout group at a time |
That last results line matters. A global holdout counts every order, attributed or not, which is exactly what an honest comparison needs.
The catch is scope. A global holdout measures your whole email and SMS program. The guide doesn't describe a way to hold out just one flow; overrides send a flow to the holdout as normal, and Klaviyo notes that overrides can reduce the lift you see. It also suggests thinking about whether the test window overlaps big events like Black Friday.
If you qualify, run a global holdout once a year to measure the whole program. Use the next option to test individual flows.
Option 2: How to run an email holdout test on one flow
Most stores have far fewer than 400,000 profiles, and most of the useful questions are about one flow anyway. Is the browse abandonment flow worth its discount? Does the second win-back email earn anything? You can answer these with a random split.
- Pick one flow and one question. For example: "Does the abandoned checkout flow cause orders, beyond the customers who'd come back anyway?"
- Add a random sample split at the top of the flow. Klaviyo's guide to A/B testing flow branches describes it: add a split, choose Random sample as the path criteria, and set the percentage for the first path. Everyone else goes down the other path. The split chooses a branch per profile at random, so the groups won't be perfectly even, but they'll be close.
- Make one path the holdout. Leave the emails on the test path. Give the holdout path no messages, so people on it simply receive nothing from this flow.
- Tag both paths. Add an update profile property action on each path that sets a property such as
checkout_testto "sent" or "holdout". Without a tag, the holdout group is invisible later, because nobody received anything you can report on. - Fix the end date and the measurement window before you start. For example: everyone who enters over eight weeks, measured on orders placed within seven days of entering.
- Compare all orders per group. Export both tagged groups and match them to your store's orders. Count every order placed in the window, whether Klaviyo attributed it or not.
Two rules keep the test honest. Change nothing else in the flow while it runs, because Klaviyo's own testing guide warns that testing more than one variable at a time can skew results. And don't compare open or click rates. The holdout group didn't get an email, so it has none.
Set up the tag before you set up the emails. A holdout you can't identify afterward is just a group of customers you ignored.
How big does the holdout need to be?
This is where most email holdout tests go wrong. People hold back 10% because it feels safe, then wait for an answer that never comes.
Here's a worked example on an illustrative store with an $86.80 average order. Its abandoned checkout flow gets 2,000 entries over eight weeks. Assume, for the example, that customers who get the flow order at 12% and customers who don't order at 9%. That's a real 3-point gap.
The test can only see that gap if it's clearly bigger than the random wobble between two groups. A common rule is that the gap should be at least about twice its standard error:
Standard error of the gap = √( p₁ × (1 − p₁) ÷ n₁ + p₂ × (1 − p₂) ÷ n₂ )
Here p is each group's order rate and n is the number of people in it.
| Split | Sent / held out | Orders | Standard error | Twice the error | Readable? |
|---|---|---|---|---|---|
| 90 / 10 | 1,800 / 200 | 216 / 18 | 2.2 points | 4.3 points | No: the 3-point gap is inside the noise |
| 50 / 50 | 1,000 / 1,000 | 120 / 90 | 1.4 points | 2.7 points | Yes, just |
The math for the 90/10 split: √(0.12 × 0.88 ÷ 1,800 + 0.09 × 0.91 ÷ 200) = √0.000468 = 0.0216, or 2.2 points.
Now the cost. In the 50/50 test, the 1,000 held-out customers place about 30 fewer orders than they would have with the flow, or about $2,604 in revenue (30 × $86.80). In the 90/10 test, the 200 held-out customers give up about 6 orders, or $520.80, and the store learns nothing it can act on.
A holdout too small to read is the most expensive kind: you still give up the sales, and you get no answer for them.
The fix is to size the test before you launch it. The holdout sample size calculator page shows the formula and a spreadsheet version. On a low-volume flow, run a larger split for a few weeks, get the answer, then shrink the holdout to a small permanent one that keeps checking the result.
Work out how many entries your flow gets per week before choosing the split. If even 50/50 won't be readable in three months, test a bigger question or a busier flow.
How to read the result
Say the 50/50 test above finishes as expected. Here's how it looks next to attribution:
| Number | Revenue | |
|---|---|---|
| Orders from the sent group | 120 | $10,416.00 |
| Orders Klaviyo attributed to the flow (assumed: those with an open or click within five days) | 105 | $9,114.00 |
| Orders the holdout says the sent group would have placed anyway (9% of 1,000) | 90 | $7,812.00 |
| Orders the flow caused | 30 | $2,604.00 |
The attributed figure is about 3.5 times the caused figure ($9,114 ÷ $2,604). Neither number is wrong; they answer different questions. The incrementality vs attribution guide explains the difference in more depth, and the incrementality testing page covers holdouts on every audience you send.
A caused figure of $2,604 over eight weeks can still make the flow well worth running. What it changes is the decisions around it. If the flow includes a discount, you can now check whether the discount pays for itself. The discount addiction guide shows how quickly a code given to people who'd buy anyway eats the margin. If the caused figure is near zero, the flow is mostly sending receipts in advance.
Report caused orders next to attributed ones, and make decisions about discounts and send volume on the caused number.
Which flows to test first
You can't test everything at once, and you shouldn't. Start where the attributed number is big and the doubt is real:
- Abandoned checkout and browse abandonment. These go to people who were already shopping, so a lot of their attributed revenue may have happened anyway.
- Flows that carry a discount. The holdout tells you whether the code is buying orders or just discounting them.
- Win-back flows. Some lapsed customers return on their own. The lapsed customers guide covers timing and the sunset step.
- Post-purchase cross-sell. Useful to test because it can quietly pull a reorder forward rather than add one.
Welcome flows are harder to hold out, because new subscribers often expect a promised first-order code. Keep any code you promised and test the follow-up emails instead.
When an email holdout test won't tell you much
Be honest about the limits before you start.
- Small flows. If a flow gets a few dozen entries a week, even a 50/50 split may take months to read. Combine similar flows into one test, or test a bigger change.
- Seasonal swings. A test that straddles Black Friday measures Black Friday. Klaviyo makes the same point about global holdouts. Run tests in steadier weeks, or long enough to cover a full cycle.
- Other channels stepping in. If held-out customers then get the same message by SMS or a retargeting ad, you're measuring the difference between channels, not email versus nothing.
- Very long buying cycles. For products bought once or twice a year, the order window may need to be months long. Measure a closer signal in the meantime, but don't call it revenue.
When a test can't be read, say so, and don't round the result up to a win.
In LiftSignals
LiftSignals scores every customer nightly and syncs audiences to Klaviyo and your other email and ad tools with a random holdout kept back on each send. So each send shows the orders it caused, and you don't have to build the split, the tags and the order matching by hand.
What to do next
You now know how to run an email holdout test on a single flow, how to size it and how to read it. The hard part is doing it on every list, every week, without a spreadsheet of tagged profiles. The Klaviyo integration sends scored audiences into Klaviyo with a random holdout kept back on each, so every send shows the orders it caused as a matter of routine. See it on your own flows during the 30-day free trial. No card, and no more arguing about which number goes in the deck.
Sources
- "Understanding Klaviyo message attribution," Klaviyo Help Center, updated Oct 2026 (checked Nov 18, 2026). https://help.klaviyo.com/hc/en-us/articles/1260804504250
- "How to change your attribution model," Klaviyo Help Center, updated Oct 2026 (checked Nov 18, 2026). https://help.klaviyo.com/hc/en-us/articles/11118357030555
- "Getting started with global holdout groups," Klaviyo Help Center, updated Oct 2026 (checked Nov 18, 2026). https://help.klaviyo.com/hc/en-us/articles/18138290642971
- "How to A/B test flow branches," Klaviyo Help Center, updated Oct 2026 (checked Nov 18, 2026). https://help.klaviyo.com/hc/en-us/articles/360049849432
- "How to add an update profile property action to a flow," Klaviyo Help Center, updated Oct 2026 (checked Nov 18, 2026). https://help.klaviyo.com/hc/en-us/articles/360001768432
Email holdout test questions
What percentage of a Klaviyo flow should I hold out?
Enough that the result will be readable in the time you have. On a low-volume flow, a 10% holdout can take months to show anything, so a larger split for a few weeks is often the better trade. Size it before you start, then shrink the holdout once you have your answer.
Can I run a holdout test in Klaviyo without a global holdout group?
Yes, for a single flow. Add a random sample split near the start of the flow, put the emails on one path and nothing on the other, and tag each path so you can tell the groups apart later. Global holdout groups need an account with at least 400,000 profiles.
Why is Klaviyo's attributed revenue higher than my holdout result?
Attribution credits an email for any order placed within the attribution window after an eligible open or click, including orders the customer was going to place anyway. A holdout subtracts those. The gap between the two numbers is the revenue that would have happened without the email.
How long should an email holdout test run?
Long enough for both groups to finish their buying window and for the gap to be clearly bigger than the noise. Klaviyo recommends about three months for its global holdout groups. Decide the end date before you start, and don't stop early because the numbers look good.
Put it to work: Klaviyo integration → · Incrementality testing → · Holdout sample size calculator →
Keep reading
All email & klaviyo guides →Start here · Incrementality & measurement
Incrementality vs attribution: what each one tells you, and when to use which
Attribution says who touched the sale. Incrementality says whether the sale would have happened anyway. They answer different questions.
4 min read · Updated Nov 15, 2026
Start here · Win-back
Lapsed customers: how to spot them by their own rhythm and win them back
A customer who buys monthly is gone by day 90. A customer who buys twice a year is just getting started. Your win-back flow should know the difference.
11 min read · Updated Nov 17, 2026
Start here · Discounts & pricing
Discount addiction: how online stores train customers to wait for a sale, and how to stop
The coupon that pays for itself on the dashboard and quietly costs you on the P&L. Here's how to tell the difference, and how to stop.
12 min read · Updated Nov 17, 2026
Put a holdout on every Klaviyo list
Send scored audiences to Klaviyo with a random holdout kept back on each, so every flow reports the orders it caused.