Filter URLs on a Large CatalogBlock, Consolidate, or Keep Crawlable

Decide first whether filtered pages need to be searchable

If your catalog has filters for things like size, rating or material, the first question is not which technique to use. It is whether any filtered page needs to be found in search. Google’s documentation on managing crawling of faceted navigation URLs splits its advice on exactly that point. If you don’t need the filtered URLs potentially indexed, prevent crawling them. If you do, follow its best practices for keeping them crawlable.

Separate the filtered pages you want available in search from the variants you don’t need indexed. The first group stays crawlable and follows the practices near the end of this guide. The second you block or steer Google away from. Google’s documentation places canonical and nofollow with that second job, not among the practices for pages you want indexed, so consolidating unwanted variants is a different decision from keeping valuable pages findable.

The fork itself is a business call, and the documentation leaves it to you. I’d start by listing the filter combinations a buyer would plausibly search for, such as one material in one size, and treat everything else as a candidate for blocking.

Keep one limit in view as you read: this is Google describing its own system. It does not show what any change would do for your site, and it is not written for manufacturing catalogs, so applying it is your judgment.

Why filter combinations waste crawling

Faceted navigation is the set of filters that lets a visitor narrow a list of products by properties such as size, rating or material. The most common way to build it puts each filter in the URL as a parameter, so every combination of filter values becomes its own URL, for example /items?products=a&color=b&size=c. Google says this can generate infinite URL spaces.

A crawler cannot tell whether one of those URLs is useful until it has crawled it, so it typically fetches a very large number of them before concluding they are useless. That leaves less time for new, useful URLs. This is the cost the rest of the guide is trying to manage, and it falls on the pages you do want crawled.

Blocking crawling: robots.txt and URL fragments

If filtered URLs don’t need to show up in search, Google’s first suggestion is a robots.txt file that disallows them. There is often no good reason to allow crawling of filtered items, it says, because it uses server resources for no or negligible benefit. The pattern it describes leaves crawlable the individual item pages and one listing page that shows all products with no filters applied. Its example, for filters named products, color and size, looks like this:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

Those are Google’s sample parameter names. Write the patterns from your own URLs and test them before relying on them.

The second route is to move the filters out of the query string and into a URL fragment, the part after the #. A filtering mechanism based on fragments has no impact on crawling, positive or negative. That is a change to how your filters are built, not a setting, so the people who build them need to be part of the decision early.

What can go wrong with blocking

A disallow rule manages crawling. It does not keep a URL out of search, and Google’s robots.txt introduction is direct about it: a robots.txt file tells crawlers which URLs they can access, is used mainly to avoid overloading a site with requests, and is not a mechanism for keeping a web page out of Google. A blocked page can still appear in results, without a description. A disallowed URL can also be indexed if other sites link to it, and the URL and publicly available information such as anchor text can then appear in results.

The usual fix, a noindex tag, does not work on a page you have also blocked. Google’s guidance on blocking indexing with noindex says the rule only takes effect if the page is not blocked by robots.txt and is otherwise accessible to the crawler. If it is blocked, Google never sees the noindex, and the page can still appear in results, for example if other pages link to it. If you use noindex to remove a URL from results, Google must be allowed to crawl it to read that instruction. Blocking crawling and preventing indexing are different jobs; don’t assume one rule accomplishes both. The other methods the robots.txt introduction lists for keeping a page out of results are password protection and removing the page entirely.

There are two further limits. Not every search engine supports every robots.txt rule, and the rules cannot enforce crawler behaviour; it is up to the crawler to obey them. Different crawlers also interpret the syntax differently, so a rule that works as intended for one may not for another.

A filtered page buyers need to find should not be blocked at all.

Consolidating unwanted variants with canonical and nofollow

Google lists two other ways to signal which filtered URLs it should crawl or not, and says both are generally less effective in the long term than blocking or fragments.

A rel="canonical" link names the URL you treat as the main version of a filtered URL, and over time it may decrease the crawl volume of the non-canonical versions. Google’s example points a URL filtered by products, color and size at the same page filtered by products alone. A rel="nofollow" attribute on links to filtered results pages may also be beneficial, but only if every anchor pointing to a given URL carries it. On a large catalog that is a real burden, since every link you control that points to a filtered URL has to be checked, and one link without the attribute means it is not effective for that URL.

Both are ways to steer crawling away from variants, not a substitute for blocking when blocking is what you need. The pages you want found follow the practices in the next section.

Pages you want found: keep them crawlable

When filtered URLs do need to be crawled and potentially indexed, Google lists practices to limit the cost of so many possible URLs. It also warns that crawling faceted URLs tends to cost sites large amounts of computing resources, because of the sheer number of URLs and the work of rendering the pages.

• Use & as the parameter separator. Commas, semicolons and brackets are hard for crawlers to detect as separators.
• If filters are encoded in the URL path, keep the logical order of the filters fixed and make duplicate filters impossible.
• When a filter combination returns no results, return a real 404 status code under the URL where it was encountered. The same applies to duplicate filters, nonsensical combinations and nonexistent pagination URLs. Don’t redirect to a common not-found page.

All three are engineering choices, so give the list to whoever maintains the catalog pages to review against your real URLs.

Putting the decisions in order

If I were working through this, I’d go in this order, because the answer to the first step decides which of the other options apply:

1. List the filter combinations buyers should be able to find.
2. For those, keep the URLs crawlable and apply the three practices above.
3. For the rest, block crawling, or move the filters to fragments if you can change how they work. Check any robots.txt patterns against your real URLs before they go live.
4. If you add canonical or nofollow for some variants, treat them as weaker signals and not as a replacement for blocking. Don’t add noindex to URLs you have blocked.
5. After any change, look at how your own site is actually crawled rather than assuming the documentation’s description holds for it.

Your next step

Start with a list of filtered pages worth making searchable. Then review the remaining URL patterns with whoever maintains the catalog, so useful pages stay accessible and unwanted combinations don’t consume unnecessary crawl effort.