fix(seo): resolve APISIX crawl, indexing, and sitemap issues

[Yilialinn]

Summary

• generate canonical and hreflang signals from verified English/Chinese content pairs
• keep historical and next docs crawlable while marking them noindex,follow
• exclude non-indexable, redirected, missing, and non-self-canonical URLs from sitemaps
• repair confirmed 403/404 routes and restore the English and Chinese RSS/Atom feeds
• add deployment assertions and browser coverage for canonical, hreflang, sitemap, and feed behavior

Details

Canonical and hreflang

• Real document translations emit reciprocal en, zh, and x-default hreflang links.
• Untranslated Chinese fallback pages remain accessible, canonicalize to English, and are excluded from the Chinese sitemap.
• Blog translations are paired only by verified source identity, including explicit keys for translations published on different dates.
• Current plugin docs retain their existing API7 documentation canonicals:
  • English: docs.api7.ai
  • Chinese: docs.apiseven.com

Historical docs and crawl handling

• Versioned and next docs remain accessible and receive noindex,follow.
• Historical paths are no longer blocked in robots.txt, allowing crawlers to process canonical and robots directives.
• Bare historical getting-started paths redirect within the same version and locale.
• Deployment checks cover English and Chinese historical/next pages and verify that canonical targets exist in the final output.

Sitemap, redirects, and feeds

• Sitemaps include only valid, indexable, self-canonical pages.
• Historical versions, next, cross-site canonicals, untranslated fallback pages, redirects, 404s, and low-value aggregate pages are excluded.
• Confirmed event, empty documentation root, and wrong-language URLs use exact redirects; unknown URLs continue to return 404.
• /blog/rss.xml, /blog/atom.xml, /zh/blog/rss.xml, and /zh/blog/atom.xml are generated again.

Validation

• npm run build in next: passed, 1,331 pages generated
• npm run test:sitemap: passed
• Playwright SEO regression suite: 12 passed on desktop and mobile
• Existing Playwright suite before the final focused rerun: 23 passed, 15 expected skips
• npm run lint: 0 errors; 10 existing content warnings
• yarn lint:frontmatter: passed
• repository pre-commit checks (eslint, Remark, lint-staged): passed
• RSS and Atom output validated as XML
• Apache redirect checks confirmed exact redirects and same-version historical targets
• git diff --check: passed

The standalone production Docusaurus build was not reproduced locally because it depends on freshly synchronized external documentation repositories. The deployment workflow now validates the affected historical pages against that fresh CI data.

Scope safeguards

• No Core Web Vitals code was changed because GSC currently reports all monitored URLs as Good.
• No API7 commercial content was introduced.
• Apache APISIX remains positioned as the open-source project.
• No MCP Gateway product capability was added or claimed.

[kayx23]

The historical/next canonical rewrite still produces a local canonical whose target does not exist. docs/apisix/next/plugins/mcp-bridge/ is present in the published asf-site tree and canonicalizes to https://apisix.apache.org/docs/apisix/plugins/mcp-bridge/, but the release/3.17 docs used to generate the versionless Astro pages contain no plugins/mcp-bridge.md, so that final canonical target is a 404. This contradicts the stated canonical-target contract and leaves a crawlable noindex,follow page pointing at a missing canonical. The deployment assertion currently checks only the installation-guide/CORS fixtures, so it does not catch this case. Please make the rewrite conditional on a real latest target (otherwise self-canonicalize or omit the override), and validate every APISIX-local canonical target in the final output rather than only the samples.

[Yilialinn]

The historical/next canonical rewrite still produces a local canonical whose target does not exist. docs/apisix/next/plugins/mcp-bridge/ is present in the published asf-site tree and canonicalizes to https://apisix.apache.org/docs/apisix/plugins/mcp-bridge/, but the release/3.17 docs used to generate the versionless Astro pages contain no plugins/mcp-bridge.md, so that final canonical target is a 404. This contradicts the stated canonical-target contract and leaves a crawlable noindex,follow page pointing at a missing canonical. The deployment assertion currently checks only the installation-guide/CORS fixtures, so it does not catch this case. Please make the rewrite conditional on a real latest target (otherwise self-canonicalize or omit the override), and validate every APISIX-local canonical target in the final output rather than only the samples.

Thanks for catching this. Fixed in 917a9b3.

• LayoutHead now resolves the active document through Docusaurus version data and emits a versionless local canonical only when the same document ID exists in the newest released version.
• When no released counterpart exists, as with next/plugins/mcp-bridge/, the override is omitted. The page keeps its original self-canonical while retaining noindex,follow.
• Added an explicit deployment regression assertion for the mcp-bridge case.
• Added a final-output validator that scans every generated HTML file and verifies that every apisix.apache.org canonical resolves to an actual file. It also rejects directory-only targets without an index.html.
• Existing intentional API7 and APISeven plugin-documentation canonicals remain unchanged.

The focused ESLint, workflow YAML parsing, positive/missing/empty-directory canonical checks, and an independent code review have passed. The full GitHub workflows have been triggered and are currently queued.

[kayx23]

The new exhaustive validator is the right direction, but the fallback canonical fix is not taking effect yet. The current build fails in the new docs/apisix/next/plugins/mcp-bridge/ assertion. When latestDoc is absent, latestUrl becomes null, so this wrapper emits only noindex,follow and leaves OriginalLayoutHead’s existing versionless canonical (/docs/apisix/plugins/mcp-bridge/) unchanged—the missing target we were trying to avoid. Please explicitly override the canonical (and og:url, if kept aligned) to ${siteUrl}${pathname} for a matched historical/next page with no latest-release counterpart, rather than omitting the override. Then rerun the build so the per-page assertion and exhaustive local-target validator both pass.

[Yilialinn]

Please explicitly override the canonical (and og:url, if kept aligned) to ${siteUrl}${pathname} for a matched historical/next page with no latest-release counterpart, rather than omitting the override. Then rerun the build so the per-page assertion and exhaustive local-target validator both pass.

Addressed in 6a7c688 and 5514b4d:

• Historical/next pages now point to the versionless latest URL only when that document exists in the latest release.
• Pages without a latest-release counterpart, including docs/apisix/next/plugins/mcp-bridge/, explicitly self-canonicalize and keep og:url aligned.
• Versionless latest canonicals are normalized with a trailing slash.
• The exhaustive validator now decodes canonical href values using browser-compatible HTML attribute semantics, so encoded paths such as q&a are validated correctly without hiding real missing targets.
• Added regression coverage for numeric/named entities, /, mixed-case invalid entities, attribute-context entities, external origins, and real missing targets.

The full Test and Deploy Website build now passes, including the canonical contract, Astro overlay, local canonical target validation, sitemap generation, and desktop/mobile tests.