What to test first on a landing page? Prioritizing your A/B tests

Every A/B test consumes a finite resource: weeks of traffic. A site receiving 10,000 monthly visitors on its landing page can statistically run a handful of conclusive tests per year — we walk through that math in how long an A/B test should run. So the question isn't "what could we test?" (everything) but "in what order do we spend a budget of five tests?". And on that point, the data accumulated by experimentation platforms tells an uncomfortable story: the vast majority of tests produce nothing measurable, and most of the gains come from a small minority of ideas. Prioritizing isn't a methodological refinement — it's what separates profitable testing programs from statistical busywork.

What research says: gains are very unequally distributed

A study by Azevedo, Deng, Montiel Olea, Rao, and Weyl published in 2020 in the Journal of Political Economy ("A/B Testing with Fat Tails") analyzed thousands of experiments run on the Bing search engine's platform and asked the question as economists: how should an experimentation budget be allocated when you don't know which ideas will win? Their empirical finding: the distribution of gains is "fat-tailed" — most tested ideas have a tiny effect, and a small fraction produces outsized ones. Their practical conclusion: in such a regime, it's better to test more genuinely different ideas, even less finely, than to endlessly refine micro-variations. For a landing page, the translation is direct: five tests on five radically different hypotheses (promise, offer, structure, proof, form) are far more likely to contain a major winner than five variations of the same button. The field's pioneers at Microsoft had already reached this humbling conclusion in their reference guide published in 2009 in Data Mining and Knowledge Discovery ("Controlled Experiments on the Web: Survey and Practical Guide", Kohavi et al.): the majority of tested ideas, including experts', fail to improve the metrics they targeted.

The hierarchy of elements: test from the top down

In the absence of data specific to your page, the hierarchy by potential impact is well established — it runs from the elements that change what the page says to those that change how it says it:

1. The offer itself — guarantee, trial, entry price, bonus: the most powerful and least tested lever, because it requires a business decision, not just a page change.
2. The value proposition and the headline — what the visitor understands in five seconds; the headline and the value proposition condition everything that follows.
3. The structure and section order — which arguments, which proof, in what order (see the anatomy of a page that converts).
4. The form and the CTA — number of fields, button wording, journey friction.
5. Cosmetic elements — colors, images, micro-phrasings: to test last, once the rest is solid; the button color is the canonical example of the popular low-stakes test.

ICE and PIE: useful frameworks, if you stay honest

The classic prioritization frameworks score each test idea on three criteria. ICE: expected Impact, Confidence in the hypothesis, Ease of implementation. PIE: the page's Potential, the Importance of the traffic it receives, Ease of execution. Score each criterion out of 10, average, sort. Their real value isn't the pseudo-precision of the score — it's forcing three explicit questions before every test. Two guardrails make them genuinely useful: first, anchor the "confidence" score in concrete evidence (session recordings, heatmaps, customer feedback, results of a 5-second test) rather than in the idea author's enthusiasm; second, remember the fat-tail lesson: bold ideas deserve an impact bonus, not a confidence penalty — it's precisely because their effect is uncertain that they can pay off big.

The low-traffic case: prioritize even harder

Below a few thousand monthly visitors, most tests will never reach significance — our article on A/B testing without traffic covers the alternatives (sequential tests, user research, bold redesigns measured before/after). The prioritization logic stays the same, taken to the extreme: only test changes whose expected effect is big enough to be detectable on your volume — levels 1 to 3 of the hierarchy above. A low-traffic site testing a button color has doomed itself in advance to learning nothing.

In practice: a testing pipeline, not isolated tests

Keep a single list of hypotheses, scored with ICE or PIE, re-evaluated at every result: a winning headline test re-ranks the following tests (the new promise changes what the social proof must prove), and a losing test is information that recalibrates the confidence scores of neighboring hypotheses. And start from an already proven structure rather than a page to invent: the ten LanderKit templates (€89 each, €229 for the bundle) provide the standard anatomy — promise, proof, offer, form — so your tests focus on the content, where research locates the big wins, rather than on structural questions already settled by practice. The full playbook for execution lives in our landing page A/B testing guide.