# ============================================================ # robots.txt — Bot access policy # ============================================================ # Four groups: blocked, search + AI answer engines, link # unfurlers, archives. Anything unlisted is denied at the bottom. # # A group's rules apply to every `User-agent` line directly above # them (RFC 9309), so each rule set is written once per group # rather than once per bot. The per-bot copies this replaced had # already drifted out of sync with the router. # # Paths are listed twice — bare and as `/*/` — because every # page is also served under a locale prefix: `/fr/chat` is the same # page as `/chat`, and `Disallow: /chat` does not match it. # ============================================================ # ----------------------------------------------------------- # AI training crawlers, scrapers and aggressive bots — BLOCK # ----------------------------------------------------------- # Training-only siblings of the answer engines allowed further # down (GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-SearchBot). # Real crawlers that would walk every page, for training # alone: the answer engines already reach the site through their # search bots. # OpenAI, Anthropic, Meta — training User-agent: GPTBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: ClaudeBot User-agent: FacebookBot User-agent: Meta-ExternalAgent # Dataset builders and scraping services User-agent: CCBot User-agent: cohere-ai User-agent: Bytespider User-agent: ByteDance User-agent: Amazonbot User-agent: Diffbot User-agent: omgili User-agent: omgilibot User-agent: Timpibot User-agent: YouBot User-agent: Webzio-Extended User-agent: AI2Bot User-agent: iaskspider # SEO crawlers — all cost, no traffic User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: PetalBot # Search engines outside this game's markets User-agent: Sogou User-agent: Baiduspider Disallow: / # ----------------------------------------------------------- # Search engines and AI answer engines — ALLOW (with limits) # ----------------------------------------------------------- # Google. The subtype crawlers are named explicitly rather than # left to inherit the Googlebot group, so the policy is readable # from this file alone. User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: Google-InspectionTool # GoogleOther is not the Search index — it is the fetcher behind # one-off and Search-adjacent product crawls. Unlisted it fell # through to the `User-agent: *` deny at the bottom, the same way # Slurp did. Note it is NOT Google-Extended: training stays blocked # in the group above. User-agent: GoogleOther # Other search indexes. Yandex holds most of Russian search, and ru, # uk, kk and tr are taught here (unblocked 2026-10-08: Russia was the # best-converting large market in Search Console while Yandex could # not crawl at all). User-agent: Bingbot User-agent: SeznamBot User-agent: Yandex User-agent: YandexBot User-agent: DuckDuckBot User-agent: Applebot # Yahoo. Its results come from Bing, but Slurp still crawls in its own right — # unlisted it fell through to the `User-agent: *` deny at the bottom, which is # the one engine this file blocked by omission rather than on purpose. User-agent: Slurp User-agent: Yeti User-agent: MojeekBot User-agent: Qwantbot User-agent: Kagibot # AI answer engines — they index to cite in results, not to train. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot # Meta AI's fetch of a link a user gave it, like ChatGPT-User. Meta # says it may ignore robots.txt anyway, so blocking it did nothing. User-agent: Meta-ExternalFetcher # Not crawlers: tokens that say what Googlebot's and Applebot's # fetches may be used for, so allowing them adds no traffic. # Google-Extended also gates grounding: blocked, Gemini could not # answer from these pages (moved here 2026-10-08). Applebot-Extended # covers Apple Intelligence. User-agent: Google-Extended User-agent: Applebot-Extended Allow: / # Never public. Disallow: /admin/ Disallow: /*/admin/ Disallow: /api/ Disallow: /metrics Disallow: /content/ # The LiveView socket. A renderer that cannot hold a websocket falls back # to /live/longpoll, which mounts a server process per render for nothing: # the first HTML already holds the page. It was 15% of Googlebot's # fetches (all JSON in Crawl stats, 2026-10). Never locale-prefixed. Disallow: /live/ # Account surfaces that are NOT the login and register forms: # settings, log_out, update_password, confirm. Those two are linked # from the nav of every page, so Google finds them ~800 times over # — and a Disallow is the wrong tool for them. `PageMeta` marks # both `noindex`, and a blocked page is never fetched, so the # noindex is never read: they sat in "Blocked by robots.txt" # permanently and could still be indexed URL-only, which is the # exact outcome the noindex exists to prevent. Let them be crawled # once, read the tag, and drop out properly. Disallow: /users/settings Disallow: /*/users/settings Disallow: /users/log_out Disallow: /*/users/log_out Disallow: /users/update_password Disallow: /*/users/update_password Disallow: /users/confirm Disallow: /*/users/confirm # Signed-in surfaces. They carry a locale prefix like every other # page, but they redirect to the login form for anyone not signed # in — so a crawler can never see them. Blocking beats noindex # here: Google has to fetch a page to discover a noindex, which # spends crawl budget either way. # # `/store` is NOT one of these. It renders its catalogue to anyone # — what the game sells is a page a stranger has a reason to read, # and it is in the sitemap. Disallow: /chat Disallow: /*/chat Disallow: /notifications Disallow: /*/notifications # Generated PDFs, not pages. Each language pair is a separate URL # of the same endpoint, so crawling them produced a pile of # "Duplicate without user-selected canonical" — 10 of 13 on this # site — and burned crawl budget rendering binaries. The # vocabulary pages carry the same words as HTML. Disallow: /courses/units/*/download Disallow: /*/courses/units/*/download Disallow: /courses/dictionary/download Disallow: /courses/paradigms/download Disallow: /*/courses/paradigms/download Disallow: /*/courses/dictionary/download Disallow: /tests/download Disallow: /*/tests/download # The word index behind the live vocabulary filter — data, not a page. Disallow: /vocabulary/*/search.json Disallow: /*/vocabulary/*/search.json # ----------------------------------------------------------- # Link preview unfurlers — ALLOW # ----------------------------------------------------------- # These fetch one URL a player shared to build its preview card. # Blocked, a link posted in Discord or on Bluesky renders as a # bare URL with no title, image or description. User-agent: Twitterbot User-agent: facebookexternalhit User-agent: Discordbot User-agent: Slackbot User-agent: Slackbot-LinkExpanding User-agent: TelegramBot User-agent: WhatsApp User-agent: LinkedInBot User-agent: redditbot User-agent: Pinterestbot User-agent: Mastodon User-agent: Bluesky Allow: / Disallow: /admin/ Disallow: /*/admin/ Disallow: /api/ Disallow: /metrics Disallow: /users/ Disallow: /*/users/ # ----------------------------------------------------------- # Web archives — ALLOW # ----------------------------------------------------------- # The Wayback Machine. No search value either way; this is about # the site having a history that outlives the server. User-agent: archive.org_bot User-agent: ia_archiver Allow: / Disallow: /admin/ Disallow: /*/admin/ Disallow: /api/ Disallow: /metrics Disallow: /content/ Disallow: /users/ Disallow: /*/users/ Disallow: /chat Disallow: /*/chat Disallow: /notifications Disallow: /*/notifications Disallow: /live/ # Snapshotting these means running the Typst renderer per fetch. Disallow: /courses/units/*/download Disallow: /*/courses/units/*/download Disallow: /courses/dictionary/download Disallow: /*/courses/dictionary/download Disallow: /tests/download Disallow: /*/tests/download # ----------------------------------------------------------- # Default rule — block everything else # ----------------------------------------------------------- User-agent: * Disallow: / # ----------------------------------------------------------- # Sitemap # ----------------------------------------------------------- Sitemap: https://polyglotpirates.com/sitemap.xml