How long should you run an A/B test on a landing page?

You launched your first test following our guide to A/B testing a landing page. Three days later, variant B is ahead, your tool shows a pleasing green percentage, and the temptation to declare a winner and move on is enormous. That is exactly the moment most tests sabotage themselves. The duration of an A/B test isn't an implementation detail: it's what decides whether your result is information or a mirage. This article walks through the four mistakes that artificially shorten tests, then the practical rules for setting a duration and a sample size before launching anything.

The number one mistake: stopping the test as soon as one variant pulls ahead

This reflex has a name: peeking — watching results continuously and stopping at the first moment the gap looks significant. The problem is mechanical: early in a test, results swing wildly, simply because each individual conversion weighs heavily on a small sample. A variant can show a spectacular lead on Tuesday and have lost it entirely by Friday, without anything changing on the page. By checking every day and stopping at the first flattering gap, you are selecting precisely the moments when randomness tells you the story you wanted to hear.

This isn't a theoretical concern. A study by Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu, published at the KDD conference in 2012, reviews real experiments run at Microsoft and on Bing and documents a series of puzzling outcomes: tests that look like winners at first and then reverse over time, effects that fade day after day, conclusions that flip entirely depending on when you look. The authors' central lesson is simple: an online experiment must run long enough for these transient effects to dissipate — otherwise you are making decisions on noise.

The novelty effect: when the variant wins just because it's new

Among the phenomena Kohavi and colleagues describe is the novelty effect: your returning visitors notice the change — a button in a new color, a block that moved — and interact with it more, simply because it's new. That enthusiasm fades once the habit settles in, and the advantage measured in the first days fades with it. The reverse also exists: a genuinely better change can underperform at first, while regulars readjust to it. Either way, the conclusion is the same — the first days of a test are the least reliable, and that is exactly when peeking makes you decide.

Think in full weeks, never in days

Landing page traffic isn't homogeneous across the week: the Monday-morning visitor comparing offers from their desk doesn't behave like the Saturday-evening visitor on their phone. If your test starts on a Wednesday and stops on a Monday, you've compared your variants on an unbalanced mix of those populations. The practical rule is simple: test duration is counted in full weeks — Monday through Sunday — so each variant sees every type of day the same number of times. One to two full weeks is the reasonable floor for most landing pages; below one complete weekly cycle, your sample is structurally biased no matter how large it is.

Sample size is computed before launch, not during the test

There is no universal magic number — "you need X visitors" is a myth, because the required volume depends on your baseline conversion rate and on the size of the effect you're trying to detect. Detecting a small improvement on a page that already converts poorly takes enormous traffic; detecting a large gap on a high-converting page takes far less. The good practice: run a sample size calculator before launching the test (most A/B testing tools include one), enter your current conversion rate and the minimum improvement that would justify the change, and note the number of visitors required per variant. That figure, fixed in advance, becomes your stopping condition — not today's curve. If you don't know your baseline, start by measuring it properly: our article on the average landing page conversion rate explains how to benchmark yours.

This discipline — hypothesis, sample size, duration, and only then reading the results — is not academic fussiness. In a second paper published at KDD in 2013, "Online Controlled Experiments at Large Scale", Kohavi and colleagues describe how large platforms industrialized their online testing: what makes results usable at scale isn't tool sophistication, it's the statistical rigor imposed on every experiment — stopping conditions defined up front, durations respected, surprising results re-verified before being believed. A solo marketer doesn't have their volumes, but the method transfers as is.

The practical rules, in order

1. State the hypothesis and pick one single change to test (see the complete A/B testing guide).
2. Estimate the required sample size before launch, with a statistical test calculator, from your current conversion rate and the minimum effect you care about.
3. Set the duration in full weeks: at least one complete weekly cycle, ideally two, even if the sample is reached earlier.
4. Don't check the results every day — or, if you can't help it, decide nothing before the planned end date.
5. Distrust the first days: an immediate spectacular gap is more often a novelty effect than a real win.
6. If the final result is surprising, re-verify it (rerun the test) rather than taking it at face value.

What if you don't have enough traffic?

This is the real situation of many coaches, trainers and small agencies: a few hundred visitors a month, not tens of thousands. At that volume, detecting a modest improvement would take months of testing — let's be honest, that's not a reasonable use of your time. You have three options, in order of preference.

• Test bold changes rather than tweaks. A large gap between variants (a fully rewritten value proposition, a different page structure) can be detected with far less traffic than a button color change. Low traffic = radical tests or no test.
• Fix the obvious problems first, without a test. A promise nobody understands in 5 seconds, a nine-field form, a page unreadable on mobile don't need an A/B test to get fixed: the 5-second test and our list of levers to increase your conversion rate are enough to prioritize.
• Replace the quantitative test with qualitative observation. A heatmap and a few session recordings show where visitors get stuck, with a tenth of the traffic an A/B test requires.

Recap: the four traps and their countermeasures

Duration isn't the problem — impatience is

A well-run A/B test isn't long because the method is heavy: it's long because reality takes time to separate itself from noise. Fix the sample size up front, count in full weeks, distrust day-three victories — and accept that with low traffic, direct improvement often beats experimentation. That's also why starting from a proven structure changes the game: the 10 LanderKit templates ($89 each, $229 for the full bundle) build in the fundamentals from the start — visible promise, short form, social proof in the right place — so your tests can focus on real hypotheses instead of fixing obvious flaws.