# robots.txt for Expert Tablă User-agent: * Allow: / # Sitemaps Sitemap: https://experttabla.ro/sitemap.xml Sitemap: https://experttabla.ro/calculator/ro/sitemap.xml # AI/LLM information files (https://llmstxt.org/) # llms.txt: https://experttabla.ro/llms.txt # llms-full.txt: https://experttabla.ro/llms-full.txt # Disallow crawling of admin and API endpoints # (the public read-only product catalogue is explicitly re-allowed because # llms-full.txt instructs AI agents to fetch live prices from /api/products) Allow: /api/products Disallow: /api/ Disallow: /admin/ # NOTE: /preferinte-cookies is intentionally NOT disallowed here. It is kept out of the # index by a "noindex, follow" robots meta on its route (app.routes.ts) instead. The two are # mutually exclusive: a path blocked here is never fetched, so the crawler would never # read the noindex tag — and a robots-blocked URL can still be indexed from inbound links, # which is exactly how it ended up in GSC's "Blocked by robots.txt" bucket. # ─── AI / LLM crawlers ───────────────────────────────────────────────── # We explicitly grant ChatGPT, Claude, Perplexity, Google Gemini, Apple # Intelligence and Meta AI full access. They are the primary source of # generative-search citations and we want our content indexed. User-agent: GPTBot Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: OAI-SearchBot Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: ChatGPT-User Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: ClaudeBot Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Claude-Web Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Claude-SearchBot Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Claude-User Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: anthropic-ai Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: PerplexityBot Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Perplexity-User Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Google-Extended Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: Applebot-Extended Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: meta-externalagent Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/ User-agent: cohere-ai Allow: / Allow: /api/products Disallow: /api/ Disallow: /admin/