User-agent: * Allow: / # Search crawlers: skip the Markdown (.md) exports. They're noindex duplicates of # the HTML pages, so crawling them only adds noise to Search Console's coverage # report. AI crawlers keep full access via their own User-agent groups below. Disallow: /*.md$ Sitemap: https://nexpay.com.au/sitemap-index.xml # LLM-friendly index LLMs: https://nexpay.com.au/llms.txt LLMs-index: https://nexpay.com.au/llms-index.txt LLMs-full: https://nexpay.com.au/llms-full.txt # AI crawlers — explicitly welcome for discoverability User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: anthropic-ai Allow: / User-agent: cohere-ai Allow: /