Doc 14578 high priority geo fixes

[osfameron]

requires https://github.com/couchbase/docs-infra/commit/f77d7a880c8dd220d091482348d35f955e2e9258 (already pushed) to work.

The major categories to review are:

llms.txt is now output by category

See preview

There's no need to review the llms-txt.js (though comments are welcome, and I'm happy to walk through the code or answer any comments)

llms-full.txt now contains a distinct workaround

See preview

This now refers to our repo https://github.com/couchbaselabs/docs-markdown-snapshot/

robots.txt improvements

No preview, but should be easy enough to make out in the antora-playbook.yml configuration

YAML frontmatter fix to namespace editURL and xref

See for example

Again, there's no need to worry about reviewing the markdown-for-llm.js (though this one is probably fairy straight-forward JavaScript review if you want to tackle it)

[Copilot]

Pull request overview

Updates Couchbase docs-site generation to improve LLM-oriented indexing and crawling behavior, including splitting llms.txt into per-category indexes and tightening robots.txt rules via the Antora playbook.

Changes:

• Namespace Antora metadata (editUrl, xref) into YAML frontmatter for markdown-for-LLM output.
• Generate per-category llms-<category>.txt files (based on keys.nav_groups) plus an llms-full.txt pointer file, and include them in a dedicated sitemap.
• Update robots directives in antora-playbook.yml to disallow specific SEO scrapers and explicitly permit selected AI crawlers.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lib/markdown-for-llm.js	Moves editUrl and Antora link info into a namespaced antora frontmatter object.
lib/llms-txt.js	Splits llms.txt into per-nav-group category indexes, adds llms-full.txt, and expands sitemap generation accordingly.
home/modules/ROOT/partials/llms-txt.md	Adds an LLM usage note and clarifies local vs web link resolution guidance.
home/modules/ROOT/partials/llms-full-txt.md	Introduces a short “full corpus” pointer file directing users to per-category index or git clone.
antora-playbook.yml	Adjusts robots rules for SEO scrapers/AI crawlers and defines keys.nav_groups used for categorization.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jamesoquendo]

Markdown and robots.txt look good.