A/A testing before A/B testing: making sure your testing tool isn't lying to you

You run an A/B test on your landing page, variant B wins with 8% more conversions, and you roll out the change. Reasonable — unless the tool that measured that number has its own bias: traffic split unevenly between the two versions, tracking that misses visitors on one branch, cache serving the wrong variant to part of your audience. In that case, the "winner" didn't win anything: the tool lied. The A/A test exists precisely to catch this, upstream, before you act on a real A/B test result. It's plumbing maintenance, not a marketing experiment — which is exactly why it's almost always skipped.

What is an A/A test?

An A/A test works like a standard A/B test — two visitor groups, random assignment, conversion rate measured — except both groups see the exact same page. No content, button, or layout change. In theory, both groups should converge on the same conversion rate, within normal statistical noise. If the measured gap goes well beyond what chance explains, or if the group sizes themselves are unbalanced when the split was supposed to be 50/50, the problem isn't the page — it's the instrumentation. It's the only way to tell a real effect apart from a measurement artifact; see our landing page A/B testing guide for the mechanics of a standard test.

The problem it uncovers: sample ratio mismatch

This split imbalance has a name in the online-experimentation literature: sample ratio mismatch (SRM), a gap between the traffic proportion expected per group (50/50, say) and what's actually observed. A study by Fabijan, Gupchup, Gupta, Omhover, Qin, Vermeer, and Dmitriev, presented at the KDD conference in 2019 ("Diagnosing Sample Ratio Mismatch in Online Controlled Experiments"), analyzed experiments run across four large tech companies and built a taxonomy of SRM causes: redirects that load slower on one branch and lose impatient visitors along the way, bot filtering that doesn't act identically on both branches, caching that serves an outdated page to part of the audience assigned to the other variant, or a bug in the hashing function that assigns visitors to a group. The study's central point: an SRM, even a small one, is enough to fully invalidate a test result — not just make it less precise.

• A redirect or test script that loads more slowly on one variant, losing impatient visitors before assignment completes
• Bot and non-human traffic filtering that doesn't behave identically across both branches
• CDN or browser caching serving an old version to visitors who should have seen the other variant
• A bug in the hashing function or assignment cookie that skews the split for certain segments (mobile, specific browsers, specific regions)
• Extra HTTP redirects on one variant that trigger a drop-off before the visitor even reaches the page

Why it matters before running a real A/B test

The idea isn't new: as far back as 2009, Crook, Frasca, Kohavi, and Longbotham, in a paper that remains a reference at the KDD conference ("Seven Pitfalls to Avoid when Running Controlled Experiments on the Web"), already recommended running A/A tests regularly — what they also call "null tests" — to exercise the experimentation system itself and confirm it isn't producing false positives. Their finding: without this check, a team can spend months optimizing a landing page based on results that measure nothing real, simply because the underlying tooling (the A/B testing script, the analytics integration, the CDN) introduces a systematic bias. An A/A test doesn't replace methodological care on each individual experiment, but it catches structural flaws nothing else reveals — because it removes the one variable a standard A/B test can never isolate: the content itself.

When to run one

An A/A test doesn't need to run all the time. It matters most at three points: when switching A/B testing tools (new platform, new integration, migration), before a high-stakes series of tests where a decision worth thousands of euros in ad spend will hinge on the result, and periodically if traffic volume allows, to confirm no technical regression (a site update, a new CDN, a change to cookie consent) has silently introduced a bias.

How to run an A/A test on your landing page

1. Duplicate your current page exactly as-is in your testing tool, with no change to content, styling, or scripts
2. Split traffic 50/50 exactly as you would for a real A/B test, using the same assignment mechanism
3. Let it run until you reach the conversion volume you'd normally use for a real test — see how many users a reliable test needs and how long an A/B test should run
4. Check group sizes first: a simple chi-squared test tells you whether the split gap (say 51%/49% instead of 50/50) is consistent with chance or signals an SRM
5. Then compare conversion rates between the two groups: a significant gap between two identical pages is proof the problem lies in the instrumentation, not the marketing

If the A/A test reveals a gap: where to look

A detected SRM doesn't immediately say where the bug is, but it narrows the search. Start with where the test script sits in the <head> — a script loaded too late gives a visitor time to see a flash of content or bounce before assignment, often asymmetrically between variants. Then check whether caching (CDN, browser, ad-blocking extension) is freezing one variant for certain visitor segments, whether tracking (GA4, Meta pixel — see our GA4 tracking guide) fires identically on both branches, and finally whether the cookie consent banner blocks measurement differently depending on which variant is shown. An SRM is almost always an infrastructure problem, never a content problem — which is exactly what makes it hard to spot without a dedicated A/A test.

What if your traffic can't support a real A/A test?

On a low-traffic landing page, a formal A/A test with a statistical threshold often isn't practical — the same volume problem as any A/B test, covered in our test prioritization guide. In that case, a lighter check is still worth doing before running a series of important tests: manually verify, across several browsers and devices, that the tool serves the same page to both groups, that conversion events fire on both sides in your analytics tool, and that assignment doesn't depend on a parameter that varies by traffic source (UTM, device, region). Starting from a clean page structure already limits some of this risk: the 10 LanderKit templates (€89 each, €229 for the full pack) ship as static Next.js, with no hidden third-party dependency that could interfere with a testing script, from the SaaS waitlist template to the coaching booking template.