Should you deindex a paid landing page? noindex, robots.txt, and real use cases

Published on 23 August 2026 · 8 min read

Every campaign launch raises the same question: should this landing page be kept out of search results? It rarely gets a real answer, because it mixes two separate issues. The first is strategic: does this page have any value outside the campaign paying for it? The second is technical: which mechanism expresses that decision? A noindex directive, a Disallow rule in robots.txt, a canonical tag, and a clean deletion with a redirect all produce different outcomes — and the most widespread mistake in this area is reaching for one of them while expecting the effect of another.

noindex and robots.txt solve different problems

The noindex directive — a <meta name="robots" content="noindex"> tag in the <head>, or an X-Robots-Tag HTTP header — targets the indexing step. The crawler still fetches the page, downloads the HTML, reads the directive, and concludes that this URL shouldn't appear in results. The key point: crawling still happens. The page is visited, its content is read, its outgoing links are seen. That's what makes noindex usable without breaking anything else.

A Disallow rule in robots.txt targets the previous step, crawling: it asks the crawler not to fetch the URL at all. Most teams discover the consequence too late — if the page is never downloaded, the noindex directive inside it is never read. Blocking a page in robots.txt in order to deindex it is therefore counterproductive: it's the best way to keep it in the index, possibly in a degraded form, since the search engine knows the URL exists thanks to links pointing at it but isn't allowed to fetch the instruction that would tell it to drop the page. The rule fits in one sentence: to deindex, allow crawling and serve a noindex. Disallow is for saving crawl effort, not for pulling a page out of results.

When deindexing is the right call

Thank-you pages. A thank-you page only makes sense after a conversion. Indexed, it pulls in visitors who land past the step they were supposed to complete, and it skews tracking if the conversion is measured on its pageview.
Test variants. During a static A/B test, secondary variants are temporary near-duplicates. A canonical usually covers it; noindex makes sense when the variant is genuinely disposable and receives no links.
Near-identical pages built per ad group. Spinning up one landing page per ad group can help ad relevance, but it produces pages that sometimes differ only by a headline. The ones targeting no query of their own have no business in the index.
Offers reserved for a paid audience. A discount for campaign traffic only, a partner offer, a negotiated rate: publishing these through search is a commercial problem. One caveat — noindex is not access control. Anyone with the URL can still open the page. If confidentiality matters, you need authentication.
Past event pages. A registration page for last quarter's webinar helps nobody. There, noindex is often just a stopgap before the real decision: redirect to the next edition or to the evergreen offer page.

When it's a mistake

The reflex that "paid landing page equals page to deindex" dates back to when those pages really were thin dead ends. Applied by default today, it destroys value the campaign budget already paid to build.

A page that answers a real query. If it covers a specific problem with substantial content, it can keep earning organic traffic long after the media budget stops — the goal described in our guide on getting a landing page to rank on Google. Deindexing it cuts off the only free part of the setup.
City pages. A multi-city build designed for local SEO targets distinct search intents. Setting them to noindex because they "look similar" cancels the only reason they exist.
Pages that earn inbound links. Partner newsletters, directories, press coverage, forums: those links are earned signals. A noindex makes them unusable; a redirect to an evergreen page keeps them.
Pages deindexed "just until we finish." The noindex added in staging and never removed is a classic. A page left that way for weeks loses its place, and reindexing after the directive comes off is not instant.

What this changes on the crawling side

Every crawlable URL consumes a resource: a crawler works with a finite budget and constantly arbitrates between discovering new pages and revisiting ones it already knows. A study by Junghoo Cho and Hector Garcia-Molina published in 2003 in ACM Transactions on Database Systems formalizes that tension around the "freshness" of a local copy: the authors model page changes as a Poisson process and compare several refresh policies (Cho & Garcia-Molina, 2003). They show that with limited resources, how visits are distributed across pages has a major effect on the overall freshness of the index, and that the naive policy of visiting the fastest-changing pages most often is not optimal. The transferable lesson is simple: crawling is a constrained, arbitrated resource. Multiplying crawlable variants of the same landing page has a cost, which is why it's worth explicitly choosing what you let get crawled. Without over-reading it, though: on a site of a few dozen pages, crawl budget isn't the limiting factor.

Finer alternatives

Canonical to the reference version. When two pages are near-duplicates but both need to stay reachable, the canonical tag designates which one to index and consolidates signals onto it, without blocking anything. For campaign variants, that's almost always the right move.
Merging. Two landing pages targeting the same intent with 90% identical content don't need a technical arbitration — they need to become one stronger page, with a redirect from the abandoned URL.
noindex, follow rather than noindex, nofollow. The follow form lets the crawler keep following links on the deindexed page, preserving flow to the rest of the site — useful if the page takes part in internal linking between landing pages.
A clean removal at end of campaign. For a page with no reason left to exist, the question isn't "noindex or not" but redirect or delete. A 301 to the closest equivalent keeps inbound links; a 410 on a page with no equivalent is an honest signal.

Which decision fits which situation
Situation	Decision	Why
Thank-you page	`noindex, follow`	Useless as an entry point, but it still passes links along
Secondary A/B test variant	Canonical to the reference variant	Temporary duplicate — consolidate rather than hide
Two near-identical pages for two ad groups	Canonical to the reference version	Neither targets a query of its own
City page with genuinely local content	Index	Distinct search intent
Substantive landing page answering a query	Index	Organic traffic outlives the media budget
Offer restricted to a paid audience	`noindex` (plus auth if confidential)	Commercial concern, not an SEO one
Past event page with no equivalent	Delete (410)	No value left, clear signal
Campaign over but inbound links earned	301 redirect	Keeps the signals instead of discarding them

How to verify what's actually happening

URL inspection in Search Console. The one tool that tells you at once whether the URL is indexed, whether crawling is allowed, and which directive was detected. A robots.txt block and a noindex show up as two distinct states — which is how you catch the case where one prevents the other from being read.
The page indexing report. It groups URLs by exclusion reason. A growing "excluded by noindex tag" bucket right after a deployment is worth checking immediately, especially after a push from a staging environment.
Reading the HTTP response directly. A curl -I against the URL shows the X-Robots-Tag header if one is set. That header is the only way to apply an indexing directive to a non-HTML file — PDF, image, downloadable asset — since you can't insert a <meta> tag into one.
A consistency check. A page carrying both noindex and a self-referencing canonical, or blocked in robots.txt while still listed in the sitemap, sends contradictory instructions. Before concluding that a search engine "ignores" your directive, check that you aren't handing it two at once.

One last note of caution: how search engines handle these directives evolves, and some formulations that used to be recommended no longer are. The principles above — crawling and indexing are separate steps, a directive that's never read is never applied, a contradictory signal is a wasted signal — are stable and documented. For implementation details and timing, the current official documentation is the only up-to-date reference.

So the real question is never "should we add a noindex?" but "will this page have a reason to exist six months from now?" If yes, index it and treat it as an asset; if not, choose between canonical, noindex, follow, and deletion based on what you want to keep. The 10 LanderKit templates (€89 each, €229 for the full pack) are standard Next.js projects: indexing directives are declared route by route in the metadata, so you can set a thank-you page to noindex while leaving the main landing page indexable.

FAQ

Frequently asked questions

Is blocking a page in robots.txt enough to remove it from Google?

No, and it often does the opposite. A Disallow rule stops the crawler from downloading the page, so it never reads the noindex tag inside. If links point to the URL, the search engine can still know about it and surface it in a degraded form. To deindex, leave crawling open and serve a noindex directive.

How long does it take for a noindexed page to drop out of results?

It depends on how often the crawler revisits that URL: the directive only applies once the page has been recrawled. That can range from a few days to several weeks depending on the popularity of the page and the site. Requesting a URL inspection in Search Console usually speeds things up, but doesn't guarantee it.

Should I use noindex, follow or noindex, nofollow?

In the vast majority of cases, noindex, follow. The nofollow variant additionally asks the crawler to ignore the page's links, which cuts off flow to the rest of the site with no real benefit. Reserve nofollow for pages that point heavily to destinations you don't control.

How do you deindex a PDF or a downloadable file?

A non-HTML file can't carry a meta tag, so you need the X-Robots-Tag HTTP header, configured at the server or host level for that path. You verify it by reading the response headers, for example with curl. And if the file must stay private, an indexing directive isn't enough — you have to restrict access.