LiftSignals is live. Every plan starts with a 30-day free trial — no credit card required. Start free →

Platform · Incrementality testing

Incrementality testing on every campaign, with a holdout group built in.

Your email platform says the campaign made $41,490. Some of those customers were going to buy anyway. LiftSignals holds back a random slice of every audience you send and shows what the campaign actually caused, in dollars, with the confidence interval next to it.

No credit card · Read-only access · Works with Shopify, WooCommerce, BigCommerce, Amazon, Square or a CSV

Two identical clay seedlings in matching pots; the right one has one extra orange leaf

Definition

What is incrementality testing?

Incrementality testing measures the revenue a campaign caused, as opposed to the revenue that happened near it. You split an audience at random, send to one group and hold the other back. The difference between the two is the incremental lift. Attribution assigns credit; incrementality measures cause.

THE LIFT FORMULA

Incremental revenue = (treated conversion rate − holdout conversion rate) × treated customers × average order value

Example: 478 orders from 8,100 treated customers, against a 4.0% holdout rate that predicts 324 anyway. 154 extra orders × $86.80 ≈ $13,370 caused, not the $41,490 claimed.

Platform attributionLiftSignals incrementality
QuestionWho touched the order?Would the order have happened anyway?
MethodClick and open windowsRandomized holdout group
Double countingEmail, SMS and ads can each claim the same orderOne test, one answer
OutputAttributed revenueCaused revenue, with a confidence interval

Why attribution overstates

Every channel takes credit for the same order, and nobody checks.

01

Your loyal customers inflate every report.

Customers who buy every month will open your email and buy. Attribution counts that as the email’s revenue, even though they were coming back regardless.

02

Three platforms claim one sale.

Klaviyo, Meta and Google each saw a touch. Add up their dashboards and you’ve “made” more than your store did.

03

Holdouts are reserved for big lists.

Most tools offer control groups only to very large senders, or make you build them by hand. So nobody runs them.

How LiftSignals does it

A holdout test that runs itself on every send.

Holdout

Every audience you sync keeps a random holdout, 10% by default, that you can change per audience. Holdout customers are excluded at the destination.

Window

Results are measured over a fixed window, 30 days by default, from the first send.

Statistics

A significance test on the difference between groups, with the confidence interval shown. If the result isn’t significant yet, the app says “not proven yet” rather than rounding up.

Revenue

Lift is reported in orders and in dollars, using your own order values.

Sample size

Before you send, the app estimates whether the audience is big enough to detect a meaningful lift.

LIFT REPORT · WIN-BACK, 30 DAYS

GROUPCUSTOMERSORDERSRATE
Sent8,1004785.9%
Holdout900364.0%
Expected anyway—3244.0%
Caused by campaign—154+1.9 pts
Attributed by email platform$41,490
Caused, p ≈ 0.02$13,370

Illustrative read-out. The same example runs through the homepage. Your results are computed from your own orders.

What to do with it

Four questions a holdout answers that a dashboard can’t.

01

Is my win-back flow worth the discount?

KlaviyoAttentive

Run the win-back with a holdout. If the people who didn’t get the 20% off came back at nearly the same rate, the coupon is giving away margin.

Churn prediction →

02

Are VIP perks driving more orders?

KlaviyoShopify

Hold back a slice of your top tier from early access. If their spending doesn’t move, the perk is a cost, not a lever.

Customer lifetime value →

03

Is retargeting reaching buyers or just paying for them?

MetaGoogle Ads

Hold out part of a Meta or Google audience. Compare order rates and see what the spend actually added.

Audience sync →

04

What should I tell the board?

Reporting

Report caused revenue with a confidence interval instead of platform-reported revenue. It’s a smaller number you can defend.

Incrementality vs attribution, explained →

Where the holdouts add up

Sum every test and you get one honest number.

Each test answers one question. Together they show how much of your revenue your marketing caused this quarter, channel by channel, with nothing counted twice.

Two glass columns of gray sand at the same level; the right column carries an extra layer of orange sand on top

Is it right for you?

Who incrementality testing is for.

A good fit if you

  • Send email, SMS or paid audiences every week
  • Want to know which campaigns are worth their discount or spend
  • Report marketing results to a founder, board or client
  • Have audiences in the thousands, not the dozens

Sounds like you? Start free

Not the right tool if you

  • Want multi-touch attribution paths for every order
  • Can’t accept withholding a campaign from anyone
  • Send very small audiences where no test can reach significance
  • Need geo-lift tests for TV or out-of-home media

FAQ · incrementality testing

Incrementality testing questions, answered.

New to the idea? Read incrementality vs attribution, explained.

What is a holdout group?

A random set of customers from your audience who don’t receive the campaign. Because they were chosen at random, their behavior shows what everyone would have done without it.

How is incrementality different from attribution?

Attribution decides which touchpoint gets credit for an order. Incrementality asks whether the order would have happened at all without the campaign. Only a holdout can answer that.

How big should the holdout be?

LiftSignals defaults to 10%. Larger holdouts reach a clear answer faster but withhold more customers. The app estimates the detectable lift for your audience size before you send.

What if the result isn’t statistically significant?

The app shows “not proven yet” with the range of plausible results. It doesn’t round a hopeful number up to a win.

Does it work with Klaviyo flows?

Yes. Audiences synced to Klaviyo exclude their holdout, so both campaigns and flows built on those audiences are measured.

Can I test ads, not just email?

Yes. Audiences sent to Meta and Google carry their holdouts too, so you can measure whether retargeting a group added orders.

Isn’t withholding a campaign lost revenue?

Only if the campaign works. If it does, the test proves it and you keep sending. If it doesn’t, the holdout just saved you the discount.

RELATED CAPABILITIES

Find out what your last campaign really made.

Connect your store this afternoon. Every audience you send carries a holdout from day one.