app/robots.ts keeps crawlers out of the admin area, the API and the edit-token pages:
disallow: ["/admin", "/api/", "/edit/"],
/edit/[token] is listed because the URL contains a secret: the raw edit token emailed to the submitter, whose SHA-256 hash is what is stored in posts.edit_token_hash. But two more routes are built the same way and are not listed:
app/(public)/status/[token]/page.tsx, reachable at /status/[token]app/(public)/unsubscribe/[token]/page.tsx, reachable at /unsubscribe/[token]
Both put a private token in the path, both are linked from outbound email, and both are therefore reachable by anything that follows links out of an inbox or a referrer header. Neither should end up in a search index.
Fix: add /status/ and /unsubscribe/ to the disallow array.
Two things worth doing in the same change:
"/admin" has no trailing slash while the others do. It still matches /admin and everything under it as a prefix, but the inconsistency is confusing; consider making it uniform.- Check whether these pages also want
robots: { index: false } in their route metadata. robots.txt is a request, not an enforcement mechanism, and a page that leaks a token should not rely on crawlers being polite.
Where to start: app/robots.ts. Verify by running npm run dev and fetching http://localhost:3000/robots.txt. Then run npx prettier --check ., npm run lint, npx tsc --noEmit and npm run build.
If you would like to take this on, comment here to claim it. Contributors can hold two open claims at a time.
app/robots.tskeeps crawlers out of the admin area, the API and the edit-token pages:/edit/[token]is listed because the URL contains a secret: the raw edit token emailed to the submitter, whose SHA-256 hash is what is stored inposts.edit_token_hash. But two more routes are built the same way and are not listed:app/(public)/status/[token]/page.tsx, reachable at/status/[token]app/(public)/unsubscribe/[token]/page.tsx, reachable at/unsubscribe/[token]Both put a private token in the path, both are linked from outbound email, and both are therefore reachable by anything that follows links out of an inbox or a referrer header. Neither should end up in a search index.
Fix: add
/status/and/unsubscribe/to thedisallowarray.Two things worth doing in the same change:
"/admin"has no trailing slash while the others do. It still matches/adminand everything under it as a prefix, but the inconsistency is confusing; consider making it uniform.robots: { index: false }in their route metadata.robots.txtis a request, not an enforcement mechanism, and a page that leaks a token should not rely on crawlers being polite.Where to start:
app/robots.ts. Verify by runningnpm run devand fetchinghttp://localhost:3000/robots.txt. Then runnpx prettier --check .,npm run lint,npx tsc --noEmitandnpm run build.If you would like to take this on, comment here to claim it. Contributors can hold two open claims at a time.