Is this A/B lift real enough to ship?

Enter visitors and conversions for control and variant. Get confidence, lift, and a clear call on whether to ship, keep testing, or hold.

Control (A)

Total number of users who saw version A
Enter the total number of users who saw version A
Number of users who completed the desired action in version A
Enter the number of users who completed the desired action in version A

Variant (B)

Total number of users who saw version B
Enter the total number of users who saw version B
Number of users who completed the desired action in version B
Enter the number of users who completed the desired action in version B

What this helps you decide

Significance shows whether the lift is likely real so you can ship, keep testing, or protect the control without guessing from a noisy sample.

  • Confidence level for the observed difference
  • Conversion rates for A and B
  • Relative lift between variants
  • Shareable link for team review

Need a cleaner test plan?

fisagency helps values-led brands design experiments that answer a real growth question, not just a vanity lift.

Talk with fisagency →
Decisions

A/B questions founders ask

Use these before you ship a variant or kill a test early.

When is 95% confidence enough to ship?

When the lift is also practically meaningful for margin and ops, and the sample covers a full business cycle. Statistical significance without practical impact is still a weak ship decision.

Results look promising but under 95%. What next?

Keep the test running until you hit sample size and at least one to two business cycles. Peeking early and shipping on 90% confidence often reverses later.

How long should the test run?

Long enough to reach required sample and cover weekday and weekend patterns, usually at least two weeks for most DTC and lead-gen sites. Never stop solely because early results look good.

What if the test is inconclusive?

Treat it as learning. Check sample size, segment effects, and whether the change was too subtle. Plan a sharper follow-up test instead of forcing a ship or kill call.

Should I watch only conversion rate?

No. Pair primary conversion with revenue per visitor, AOV, and retention signals. A higher conversion rate that lowers order value can still hurt the business.