Robots (SEO & GEO)
The platform includes configurable Search Engine Optimization (SEO) and Generative Engine Optimization (GEO) controls. SEO manages how traditional search engines like Google and Bing index your site. GEO controls how AI systems like ChatGPT, Claude, and Gemini discover and use your content.
Robot controls are split into two independent sections: Read Access governs who may see your content (crawling, indexing, AI training), while Write Access governs who may acton your site (placing orders, submitting forms, uploading files). The two are never coupled — changing AI read access does not open or close checkout to AI assistants, and vice versa.
Read Access
Read Access controls who may crawl, index, and train on your content. Search engine settings use standard directives from the Robots Exclusion Protocol, honored by all major search engines; they control the meta robots tag and the /robots.txt file. AI settings additionally cover training and image use.
| Setting | Standard | Default | Effect |
|---|---|---|---|
| Allow Indexing | Robots Exclusion Protocol | On | When off, search engines will not add your pages to their results (noindex) |
| Allow Link Following | Robots Exclusion Protocol | On | When off, search engines will not follow links on your pages (nofollow) |
| Allow AI Crawlers | Robots Exclusion Protocol | On | Master toggle for AI bots reading your content. When off, all known AI bots are blocked via robots.txt and the page opts out with noai + noimageai — your business stops appearing in answers from ChatGPT, Claude, Perplexity, and Google AI Overviews |
| Allow AI Indexing | Experimental | On | When off, adds noai meta tag to opt out of AI training and retrieval. Not all providers honor this. |
| Allow AI Image Use | Experimental | On | When off, adds noimageai meta tag to opt out of AI image training. Not all providers honor this. |
Write Access
Write Access controls who may take actions on your site — placing orders, creating accounts, submitting contact forms, uploading files. Bot detection on these endpoints is provided by our bot-protection partner, Vercel. Unknown bots and scripted traffic are always blocked from write actions, regardless of any setting here.
| Setting | Default | Effect |
|---|---|---|
| Allow AI Assistants | On | Some customers let an AI assistant (like ChatGPT) act for them. When on, assistants that Vercel has verified are working for a real person may place orders, sign up, and submit forms. When off, every action requires a person in a browser — safer against automation, but genuine assistant-driven orders and signups are turned away too. |
Blocked AI Bots
Block specific AI bots by user-agent name, even when the Read and Write toggles above would otherwise allow them. Each entry has two flags, so you can block one side of a bot's behavior while allowing the other:
| Field | Default | Effect |
|---|---|---|
| Bot User-Agent | — | The bot to block, chosen from the known AI bots list below |
| Block Reading | On | Blocks the bot from crawling and indexing your content (per-agent robots.txt rule) |
| Block Writing | On | Blocks the bot from taking actions — orders, accounts, forms, uploads — even when it is a verified assistant and Write Access would otherwise allow it |
A newly added bot is blocked from bothreading and writing. Uncheck one flag to carve out that side — for example, blocked from AI training but still allowed to check out on behalf of a customer. An enabled block here beats every other setting in the Robots section.
Enforcement Layers
Robot controls are enforced at multiple levels for maximum coverage:
- robots.txt — Dynamic, database-driven rules. Primary read-side enforcement for well-behaved crawlers.
- Meta robots tags —
index/followplus thenoaiandnoimageaidirectives in the HTML<meta>tag. Database-driven, and the layer that reaches an AI bot which ignores robots.txt. - Bot verification on write endpoints — Vercel bot detection challenges every order, signup, form submission, and upload. Enforces the Write Access toggle and per-bot Block Writing flags; unverified automation is always rejected.
- llms.txt — Structured site description endpoint that AI systems can consume.
Known AI Bots
The platform recognizes these AI bot user-agents for per-agent robots.txt blocking:
| Bot | Organization | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data & ChatGPT web browsing |
| ChatGPT-User | OpenAI | User-initiated real-time web search |
| anthropic-ai | Anthropic | Claude training data |
| ClaudeBot | Anthropic | Claude web search/retrieval |
| CCBot | Common Crawl | Open dataset used by many AI companies |
| PerplexityBot | Perplexity AI | Search engine crawling |
| Google-Extended | Gemini training (NOT Google Search) | |
| Bytespider | ByteDance | TikTok parent company crawler |
| Amazonbot | Amazon | Alexa and Amazon AI services |
| FacebookBot | Meta | Content preview and AI training |
| Meta-ExternalAgent | Meta | Meta AI assistant web browsing |
Google-Extended only controls Gemini AI training. Blocking it does not affect your site's Google Search rankings or indexing.llms.txt
The /llms.txt endpoint provides a structured text description of your site that AI systems can read. This is an emerging standard (see llmstxt.org) that helps AI chatbots accurately answer questions about your business. Publishing is controlled by the Publish llms.txt toggle (off by default); when enabled, the llms.txt Content field seeds the endpoint.
If you leave the llms.txt content field empty, the platform auto-generates content from your site name, description, location, and contact information. For best results, write custom content that includes:
- A clear description of what your business does
- Your location and service area
- Products and services offered
- Contact information
- FAQ section answering common customer questions
