How do you determine the duration and scope of A/B tests?

Determining the duration and scope of A/B tests is all about balancing statistical rigor with business urgency—you want enough data to make confident decisions without wasting time or budget. Here's how I typically approach it:

🧮 1. Calculate Required Sample Size Upfront
Before launching a test, use an A/B test sample size calculator to determine:

How many impressions, clicks, or conversions you need

How long it'll take to get there, based on your traffic volume

You'll typically input:

Baseline conversion rate (from historical data)

Minimum detectable effect (MDE) – e.g., a 10% lift

Statistical significance level (usually 95%)

Statistical power (usually 80%)

🧠 Tools I use:

Optimizely Calculator

VWO Calculator

📌 Tip: If your site or campaign has low volume, test larger changes (to detect a bigger effect), or allow more time.

📆 2. Set a Minimum Duration to Account for Variability
Even if you hit significance quickly, let it run for at least 1–2 full buying cycles to capture day-of-week, weekend, and campaign seasonality.

Low-volume tests: 2–4 weeks

Medium to high volume: 7–14 days is often enough

Avoid running less than 7 days unless you have very high volume and confidence

📅 Avoid launching during holidays or seasonal spikes, unless you're specifically testing for those periods.

🛠� 3. Define the Scope: What's Being Tested + Where
Scope is all about how big the test reaches. Define:

What's being tested (e.g., headline copy, bid strategy, landing page layout)

Where it's tested:

One campaign or all?

One device type? (e.g., mobile-only)

Geographic or audience segmentation?

🎯 Keep it narrow when starting to minimize risk—you can scale up once a clear winner is found.

🔄 4. Test One Variable at a Time (If Possible)
To isolate impact:

Change only one major element per test

e.g., headline copy A vs. B

OR manual bids vs. target CPA bidding

If you're testing multiple variables, consider multivariate testing—but only with high traffic

🔬 More variables = more complexity = more data required

⚖️ 5. Decide the Success Criteria
Set your primary KPI ahead of time, and don't move the goalposts mid-test:

Click-through rate (CTR)?

Conversion rate (CVR)?

Cost per acquisition (CPA)?

Return on ad spend (ROAS)?

Also set what "success" means numerically, like:

"We'll consider this test a win if Variant B improves CVR by at least 12% without increasing CPA."

⏸️ 6. Know When to End the Test Early (or Extend It)
✅ End early if:

You hit statistical significance well above your threshold

Performance is consistently superior across all segments

🚫 Extend or pause if:

Results are flat or unclear after expected timeframe

There's unusual traffic fluctuation (e.g., big spike or drop due to an unrelated event)

You didn't get enough sample size to draw conclusions

📈 7. Monitor Performance, But Don't Optimize Mid-Test
Avoid tweaking the test while it's running—it can corrupt your results. Wait until the test is complete before making changes.

📌 Exception: If it's clearly tanking, and performance is way off baseline (e.g., CPA spikes 200%), pause the test and review.

🧠 Final Framework:

Question   Answer
What's the hypothesis?   Clearly defined and measurable
What KPI will determine success?   Primary + secondary metrics
How long will it run?   Based on volume + sample size calc
How big is the scope?   Narrow enough to control variables
What happens next?   Pre-decided actions for win, loss, or inconclusive