Multi-armed bandit or classic A/B test: which method for a landing page fed by paid ads

A classic A/B test splits traffic 50/50 between two variants from start to finish, even if one of them turns out to be clearly worse within the first few days. That's a deliberate design choice, not a flaw: a fixed split guarantees a clean statistical comparison at the end. But it comes at a direct cost when the tested traffic comes from a Google Ads or Meta Ads campaign paid by the click — every visitor sent to the losing variant during the three or four weeks the test runs is a lost conversion, paid at full price. The multi-armed bandit algorithm starts from a different problem: instead of waiting for the test to end before acting, it gradually shifts traffic toward the winning variant while the test is still running.

The analogy behind the name

The name comes from a casino gambler facing several slot machines (the original "one-armed bandits"), each with a different, unknown payout rate. Every time a coin is inserted, the gambler must choose: play the machine that has already paid well (exploit what's known), or try another machine to check whether it might pay better (explore the unknown). Too much early exploitation risks missing the best machine; too much exploration wastes coins on machines already known to be poor. A landing page with two or three variants faces exactly the same trade-off: every visitor sent to a variant is a "coin" that either converts or doesn't, and the question is how to split the next visitors in real time between what already looks like it's winning and what still needs to be checked.

What a bandit algorithm changes compared to a fixed-split A/B test

A classic A/B test freezes the traffic split (50/50, or any chosen ratio) for the entire duration of the test, and only decides at the end, once a statistical significance threshold is reached — the full logic is covered in our guide to statistical significance. A bandit algorithm instead recalculates continuously — after every handful of visitors, sometimes after every conversion — the probability that each variant is the best one, and adjusts the share of traffic sent to each accordingly. Steven Scott, a statistician at Google, describes in "A Modern Bayesian Look at the Multi-Armed Bandit," published in 2010 in Applied Stochastic Models in Business and Industry (see on Google Scholar), the most widespread method for this trade-off: Thompson sampling (or randomized probability matching), which randomly draws an estimated conversion rate for each variant based on what has already been observed, and sends the next visitor to whichever variant's draw comes out highest. A variant that disappoints gradually sees its traffic share shrink, without ever dropping abruptly to zero as long as uncertainty remains — the mechanism stays probabilistic, not a binary cutoff.

The "learn-and-earn" trade-off in an advertising context

Eric Schwartz, Eric Bradlow and Peter Fader, in "Customer Acquisition via Display Advertising Using Multi-Armed Bandit Experiments," published in 2017 in Marketing Science (see on Google Scholar), study exactly this ground: advertisers who must split ad impressions across several creatives without knowing in advance which one acquires the most customers, while trying simultaneously to learn and to not lose too much while learning — the so-called "learn-and-earn" trade-off. Their work shows that a properly calibrated bandit approach reduces acquisition cost compared to a fixed-split test, precisely because it limits the volume sent to creatives that turn out to be poor early in the test. That's exactly the situation of a landing page fed by a Meta Ads or Google Ads campaign: the ad budget is the scarce resource, and every visitor sent to the losing variant during the test phase carries a real cost, not just a missed conversion.

When the bandit has the edge over a classic A/B test

• A time-limited campaign: a launch, an offer with a fixed deadline, a webinar whose registration closes in two weeks — the time available to learn is short, and every misallocated visitor can't be made up for afterward. Our webinar & masterclass template (demo) illustrates this case well: the registration window closes, and there's no "next time" to recover traffic misallocated during the test.
• Traffic paid by the click: when every visitor has a direct cost (Google Ads, Meta Ads), reducing the volume sent to the losing variant during the test translates directly into saved ad budget, as shown by Schwartz, Bradlow and Fader.
• More than two variants to compare: a classic A/B test quickly loses statistical power as more variants are added (traffic gets diluted accordingly), whereas a bandit keeps converging toward the best ones without needing a separate sample plan per variant.
• Frequent, low-stakes decisions: continuously testing several ad visuals or headlines on a high-volume page, where a slightly suboptimal decision today corrects itself tomorrow.

When a classic A/B test remains the right choice

• A permanent, structural decision: permanently changing the page's architecture, its displayed price, or its core pitch deserves a clean statistical conclusion rather than a probabilistic convergence — see our complete A/B test guide.
• Traffic that varies over time: most bandit algorithms assume a stable conversion rate throughout the test; if the traffic changes in nature partway through (a new source, seasonality, weekday vs. weekend), the bandit can over-allocate a variant that was winning temporarily for the wrong reasons — a bias that the classic statistical framework handles better by fixing the duration in advance.
• Little traffic available: below a few thousand monthly visitors, neither the bandit nor a classic A/B test has enough data to converge quickly — the qualitative alternatives covered in our article on A/B testing without traffic remain more cost-effective.
• Needing to justify a decision after the fact: an A/B test report with a p-value and a confidence interval is easy to document and defend to management or a client; a bandit's convergence is harder to sum up in one sentence.

What this means in practice for a small team

Few consumer-grade tools offer a real, configurable multi-armed bandit for a standalone landing page — most of the experimentation platforms historically accessible to small teams (Google Optimize chief among them) have shut down, and the ones that remain (VWO, Convert) often reserve dynamic allocation for their pricier tiers. Two realistic options exist regardless. The first: rely on the optimization already built into the ad platforms themselves — Meta Ads and Google Ads automatically reallocate budget across several ad creatives following a principle very close to a bandit, which already covers part of the problem without any extra tool on the landing page side. The second: for the page itself, stick to a properly sized classic A/B test (see how many visitors you need and how long to run it), and limit the number of variants tested at once so as not to dilute traffic that, in any case, doesn't yet justify the infrastructure of a custom bandit.

Whatever the method, it never makes up for a poorly designed variant to begin with. Our 10 LanderKit templates (€89 each, €229 for the full pack) start from an already proven conversion structure, so that the test — bandit or classic — bears on real decision points (angle, social proof, offer) rather than on structural flaws that any test would have eventually revealed anyway. That's especially true on a single-product page like e-commerce product (demo), typically fed by paid ad traffic where every misallocated visitor during a test carries a direct cost.