# BuyMix crawler policy - reviewed 2026-10-30. # # Terms of use: https://buymixx.com/terms.html - section 9.4 says which pages may be read, indexed and quoted # (including by search engines and AI assistants, to the extent this file allows them); sections 3 to 5 say what may not be copied. # # PUBLIC: the description on the home page, the "about" pages (about*.html), the terms (terms*.html), llms.txt and sitemap.xml. # NOT PUBLIC: the app itself, the API (/api/) and the data behind it. BuyMix's value is aggregated, normalized price # data across dozens of chains, and that is the one thing no crawler below is invited to take. # # This file asks, it does not enforce. What refuses is the API itself: a program that announces itself as an AI agent (or as a # plain HTTP library) gets a 403 there, there is a per-IP rate limit in front of the site, and whatever needs an account asks for a sign-in. # # 2026-10-30, CHANGED. Until today every AI crawler was disallowed outright ("no AI training", 2026-10-01). That also shut out the # crawlers that find a site in order to ANSWER a question about it - ChatGPT-User, PerplexityBot and Google-Extended (Google's own # page says that token covers grounding Gemini's answers in the Search index, not only training) - while the site itself gave a # crawler nothing to read. The answer engines are allowed below, on the public pages only. Training-only crawlers stay out. # # A group that names a crawler REPLACES the "*" group for that crawler, it does not add to it. So the group that allows anything # repeats the rules that keep the API and the share links closed. frontend/src/discoverability.test.js reads this file the way a # crawler does and fails when a group forgets. Sitemap: https://buymixx.com/sitemap.xml # 1. AI search and answer engines (they find a page in order to show it, summarize it and link to it). # ChatGPT search, ChatGPT when a person asks about a page, Perplexity, Claude's search and Gemini's grounding. # Google-Extended is one switch for two things - grounding in Gemini and training future Gemini models - so allowing it # allows both, on the public pages only. Disallow comes first on purpose: a parser that stops at the first matching line # (Python's, for one) must meet "/api/" before it meets "Allow: /". User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Google-Extended Disallow: /api/ Disallow: /*?join= Allow: / # 2. Crawlers that collect pages only to train models: no. User-agent: GPTBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: CCBot User-agent: Bytespider User-agent: Applebot-Extended User-agent: Amazonbot User-agent: cohere-ai User-agent: Diffbot User-agent: Omgilibot User-agent: YouBot User-agent: Meta-ExternalAgent Disallow: / # 3. Everyone else (Googlebot, Bingbot, a shopper's own browser): the public pages are open. # /api/ is JSON, not a page. A share link (?join=...) carries a code that lets whoever holds it join someone's basket, # so a search engine must not list it. User-agent: * Disallow: /api/ Disallow: /*?join=