Is your store blocking the AI crawlers that sell for you?

Which crawlers matter, and what each one actually does

AI companies do not run a single crawler. They run several with different jobs, and blocking them is not an all-or-nothing decision. The distinction that matters most is between crawlers that feed shopping and search results, and crawlers that gather training data. Blocking the second has no effect on your visibility. Blocking the first makes you invisible.

That last distinction is worth sitting with. Blocking GPTBot is a defensible position on model training and costs you no shopping visibility. Blocking OAI-SearchBot removes you from ChatGPT results entirely. They are frequently confused, and a blanket AI block toggle usually catches both.

The robots.txt rules that trip people up

Specific rules override the wildcard completely. If a crawler matches its own named group, the User-agent: * group is ignored entirely for that crawler, not merged with it. A named group containing one narrow disallow gives that bot broader access than the wildcard rules you assumed applied.

An empty disallow means allow everything. Disallow: with nothing after it is the opposite of Disallow: /. A stray character between them inverts your intent.

Blank lines separate groups. A blank line between a user-agent line and its rules can orphan those rules, applying them somewhere you did not intend.

Shopify requires a template. You cannot edit robots.txt directly. Create templates/robots.txt.liquid under Edit code, which lets you extend the defaults Shopify generates.

When robots.txt says allowed but you are still invisible

This is the most common situation, and robots.txt is genuinely not the problem. Requests are being rejected before robots.txt is ever consulted.

The usual cause is a CDN or WAF rule. Cloudflare ships an AI bot blocking feature that returns 403 to these crawlers at the edge, and it can be enabled by default on some plans, meaning nobody on your team turned it on. Your robots.txt can be perfect and every request still fails. Check Security, then Bots, and confirm AI bot blocking is off for your domain. Then verify from a terminal:

curl -I -A "OAI-SearchBot" https://yourstore.com/
curl -I -A "PerplexityBot" https://yourstore.com/products/your-bestseller

A 200 means the request succeeded. A 403 means something upstream is still refusing it. Test a product URL as well as the homepage, since some rules only apply to certain paths.

Other causes worth ruling out: aggressive rate limiting that rejects crawlers as suspicious traffic, password protection left on a staging environment that became production, geographic restrictions blocking the regions crawlers request from, and JavaScript-rendered product data that returns an empty shell in the raw HTML.

Being crawlable is step one of three

Access gets you considered, not recommended. Once crawlers can reach your pages, the next questions are whether your product data is structured enough to parse, using the Product Schema Generator, and whether your store describes itself the way you want, using the llms.txt Generator. The full sequence is in the free AI Shopping Checklist.