Add robots.txt and Referrer-Policy meta tag#91
Open
santichausis wants to merge 1 commit into
Open
Conversation
robots.txt disallows crawling of internal-only directories that are never meant to be visited as standalone pages: /scripts/ (internal admin/migration tools), /api/ (raw JSON endpoints), /includes/, /meta/, and /translations/ (PHP/JS partials loaded by other pages). Points to both existing sitemaps. Also adds <meta name="referrer" content="strict-origin-when-cross-origin"> to header-2025.php, loaded on every page. Note on the other commonly-recommended security headers (X-Content-Type-Options, X-Frame-Options, HSTS, CSP): this repo's own .gitignore explicitly excludes .htaccess* from version control, and every page here echoes HTML (the opening <!DOCTYPE html>) before header-2025.php is ever required -- so a PHP header() call placed there would fail with "headers already sent" on every real page. Meta tags don't have that restriction (hence going that route for Referrer-Policy), but X-Content-Type-Options/X-Frame-Options have no meta-tag equivalent that browsers honor, and CSP isn't safe to guess at without a way to test it against every external resource the site loads (fonts.googleapis.com, unpkg.com/leaflet, SVGator, Facebook widgets, etc.). Recommend adding those directly to the server's own .htaccess/vhost config outside this repo, per its existing gitignore convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
robots.txtdisallowing/scripts/,/api/,/includes/,/meta/,/translations/— none of these are meant to be visited as standalone pages (they're internal admin tools, raw JSON endpoints, or PHP/JS partialsrequire'd by real pages), so there's no reason for them to be crawled or indexed. Points to bothsitemap.xmlandsitemap-multilingual.xml.<meta name="referrer" content="strict-origin-when-cross-origin">toheader-2025.php, which is loaded on every page.Why not the other usual security headers (X-Content-Type-Options, X-Frame-Options, HSTS, CSP)?
Wanted to flag this rather than silently skip it or half-implement something that looks like it works but doesn't:
.gitignorehas.htaccess*under "Core server / hosting clutter" —.htaccessis explicitly excluded from version control here, presumably because it's managed directly on the live server. Adding one in this PR would go against that existing convention (and could conflict with whatever the server already has).header()doesn't work as an alternative: every page (en/how.php, etc.) starts with literal<!DOCTYPE html>output beforeincludes/{page}-inc2.phprequiresheader-2025.php— so aheader()call placed inheader-2025.phpwould hit "headers already sent" on every real page and silently no-op.<meta>tags don't have that "already sent" restriction, which is whyReferrer-Policycould be done this way — butX-Content-Type-OptionsandX-Frame-Optionshave no meta-tag equivalent that any browser honors; they require a real HTTP header.<meta http-equiv="Content-Security-Policy">tag, but I didn't want to guess at a policy without being able to test it against every external resource the site actually loads (fonts.googleapis.com, unpkg.com/leaflet, SVGator, Facebook widgets, etc.) — a wrong CSP silently breaks functionality rather than failing loudly.Recommendation: add
X-Content-Type-Options: nosniff,X-Frame-Options: SAMEORIGIN, and HSTS directly in the live server's.htaccess/vhost config (outside this repo, consistent with how.htaccessis already handled). Happy to help draft a CSP in a follow-up if someone can test it against staging/beta.ecobricks.orgfirst.Testing
robots.txtisn't caught by any existing.gitignorerule (git check-ignorereturns nothing).php -lonheader-2025.php— no syntax errors.