Platform · Incrementality testing
Incrementality testing on every campaign, with a holdout group built in.
Your email platform says the campaign made $41,490. Some of those customers were going to buy anyway. LiftSignals holds back a random slice of every audience you send and shows what the campaign actually caused, in dollars, with the confidence interval next to it.
No credit card · Read-only access · Works with Shopify, WooCommerce, BigCommerce, Amazon, Square or a CSV
Definition
What is incrementality testing?
Incrementality testing measures the revenue a campaign caused, as opposed to the revenue that happened near it. You split an audience at random, send to one group and hold the other back. The difference between the two is the incremental lift. Attribution assigns credit; incrementality measures cause.
THE LIFT FORMULA
Incremental revenue = (treated conversion rate − holdout conversion rate) × treated customers × average order value
Example: 478 orders from 8,100 treated customers, against a 4.0% holdout rate that predicts 324 anyway. 154 extra orders × $86.80 ≈ $13,370 caused, not the $41,490 claimed.
| Platform attribution | LiftSignals incrementality | |
|---|---|---|
| Question | Who touched the order? | Would the order have happened anyway? |
| Method | Click and open windows | Randomized holdout group |
| Double counting | Email, SMS and ads can each claim the same order | One test, one answer |
| Output | Attributed revenue | Caused revenue, with a confidence interval |
Why attribution overstates
Every channel takes credit for the same order, and nobody checks.
01
Your loyal customers inflate every report.
Customers who buy every month will open your email and buy. Attribution counts that as the email’s revenue, even though they were coming back regardless.
02
Three platforms claim one sale.
Klaviyo, Meta and Google each saw a touch. Add up their dashboards and you’ve “made” more than your store did.
03
Holdouts are reserved for big lists.
Most tools offer control groups only to very large senders, or make you build them by hand. So nobody runs them.
How LiftSignals does it
A holdout test that runs itself on every send.
Holdout
Every audience you sync keeps a random holdout, 10% by default, that you can change per audience. Holdout customers are excluded at the destination.
Window
Results are measured over a fixed window, 30 days by default, from the first send.
Statistics
A significance test on the difference between groups, with the confidence interval shown. If the result isn’t significant yet, the app says “not proven yet” rather than rounding up.
Revenue
Lift is reported in orders and in dollars, using your own order values.
Sample size
Before you send, the app estimates whether the audience is big enough to detect a meaningful lift.
LIFT REPORT · WIN-BACK, 30 DAYS
Illustrative read-out. The same example runs through the homepage. Your results are computed from your own orders.
What to do with it
Four questions a holdout answers that a dashboard can’t.
01
Is my win-back flow worth the discount?
Run the win-back with a holdout. If the people who didn’t get the 20% off came back at nearly the same rate, the coupon is giving away margin.
02
Are VIP perks driving more orders?
Hold back a slice of your top tier from early access. If their spending doesn’t move, the perk is a cost, not a lever.
03
Is retargeting reaching buyers or just paying for them?
Hold out part of a Meta or Google audience. Compare order rates and see what the spend actually added.
04
What should I tell the board?
Report caused revenue with a confidence interval instead of platform-reported revenue. It’s a smaller number you can defend.
Where the holdouts add up
Sum every test and you get one honest number.
Each test answers one question. Together they show how much of your revenue your marketing caused this quarter, channel by channel, with nothing counted twice.
Is it right for you?
Who incrementality testing is for.
A good fit if you
- Send email, SMS or paid audiences every week
- Want to know which campaigns are worth their discount or spend
- Report marketing results to a founder, board or client
- Have audiences in the thousands, not the dozens
Not the right tool if you
- Want multi-touch attribution paths for every order
- Can’t accept withholding a campaign from anyone
- Send very small audiences where no test can reach significance
- Need geo-lift tests for TV or out-of-home media
FAQ · incrementality testing
Incrementality testing questions, answered.
New to the idea? Read incrementality vs attribution, explained.
What is a holdout group?
A random set of customers from your audience who don’t receive the campaign. Because they were chosen at random, their behavior shows what everyone would have done without it.
How is incrementality different from attribution?
Attribution decides which touchpoint gets credit for an order. Incrementality asks whether the order would have happened at all without the campaign. Only a holdout can answer that.
How big should the holdout be?
LiftSignals defaults to 10%. Larger holdouts reach a clear answer faster but withhold more customers. The app estimates the detectable lift for your audience size before you send.
What if the result isn’t statistically significant?
The app shows “not proven yet” with the range of plausible results. It doesn’t round a hopeful number up to a win.
Does it work with Klaviyo flows?
Yes. Audiences synced to Klaviyo exclude their holdout, so both campaigns and flows built on those audiences are measured.
Can I test ads, not just email?
Yes. Audiences sent to Meta and Google carry their holdouts too, so you can measure whether retargeting a group added orders.
Isn’t withholding a campaign lost revenue?
Only if the campaign works. If it does, the test proves it and you keep sending. If it doesn’t, the holdout just saved you the discount.
RELATED CAPABILITIES
Audience sync →
Send scores to Klaviyo, Meta, Google and more.
Churn prediction →
See who’s drifting away before they’re gone.
Marketing mix modeling →
Where the next ad dollar should go.
Find out what your last campaign really made.
Connect your store this afternoon. Every audience you send carries a holdout from day one.