# Google crawls at full speed — Googlebot ignores Crawl-delay anyway, and we # want fast, complete indexing from Google. User-agent: Googlebot Allow: / Disallow: /deleteUser Disallow: /en/deleteUser # Google-Extended controls whether our content may be used to train and ground # Gemini models. AI Overviews in Google Search follow Googlebot, not this token. # Without this group it would inherit the throttled `*` rules below. User-agent: Google-Extended Allow: / Disallow: /deleteUser Disallow: /en/deleteUser # Bing powers ChatGPT Search and Copilot, so a slow Bing crawl directly costs us # AI-answer visibility. Unlike Googlebot, Bingbot DOES honour Crawl-delay — it # must have its own group here or it falls back to the throttled `*` rules. User-agent: Bingbot Allow: / Disallow: /deleteUser Disallow: /en/deleteUser # AI answer engines. These fetch on demand and at low volume, so the origin # rate-limiter is not a concern; throttling them only delays our content # reaching ChatGPT / Claude / Perplexity. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: DuckAssistBot Allow: / Disallow: /deleteUser Disallow: /en/deleteUser # Everything else (third-party SEO/audit crawlers): throttle to one request every # few seconds so request bursts don't trip the host's rate limiter. This is what # caused the intermittent 429 / 502 responses during audits. User-agent: * Allow: / Disallow: /deleteUser Disallow: /en/deleteUser Crawl-delay: 5 Sitemap: https://velyx.ai/sitemap.xml