Build the search index from Jekyll instead of a Rake task

Indexing hung off `rake build`, so `rake serve` served a site with no search at
all and needed a second task to preview one. Moving it to a `post_write` hook
means any path that writes the site ends with a matching index: `rake build`,
`rake serve` and its regenerations, and the bare `jekyll build` the deploy
workflow runs.

`rake serve` only ever handed out whatever `_site` already held, since it called
`Serve.process` without the `Build.process` the `jekyll serve` command pairs it
with. It now runs both, the way the README has always described it.

Pagefind indexes silently because this fires on every regeneration, and the hook
reports one line instead. Without Node it says so and lets the build through, so
editing content still only needs Ruby.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: hsbt

.github/workflows/jekyll.yml

@@ -43,8 +43,6 @@ jobs:
         run: bundle exec jekyll build --baseurl "${{ steps.pages.outputs.base_path }}"
         env:
           JEKYLL_ENV: production
-      - name: Build search index
-        run: bundle exec rake search-index
       - name: Upload artifact
         uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
 

CLAUDE.md

@@ -14,12 +14,9 @@ This is the Jekyll-based source for the official Ruby programming language websi
 # Build the site and its search index (takes several minutes)
 bundle exec rake build
 
-# Serve locally at http://localhost:4000/ (rebuilds, so no search index)
+# Build, watch and serve at http://localhost:4000/, search index included
 bundle exec rake serve
 
-# Serve an already-built site, search included, without live reload
-bundle exec rake serve-built
-
 # Alternative: Jekyll direct serve with incremental builds
 bundle exec jekyll serve --watch --future --incremental
 ```
@@ -116,9 +113,12 @@ The news system is powered by a custom Jekyll plugin (`_plugins/news.rb`):
 
 ### Search (Pagefind)
 
-Search is client-side and needs no backend. `rake build` runs `npm run
-build-search` over `_site` once Jekyll is done, so the index only exists after a
-full build.
+Search is client-side and needs no backend. `_plugins/search_index.rb` hooks
+Jekyll's `post_write` and calls `lib/search_index.rb`, so every build and every
+regeneration under `rake serve` ends with a matching index. It has to run after
+the write because Pagefind reads the generated HTML, and it has to run every
+time because Jekyll deletes destination files with no source counterpart, the
+previous bundle included. Without Node installed the hook warns and skips.
 
 - **Indexed region**: `_includes/search_body.html` emits `data-pagefind-body` on
   the `<article>` of `page.html` and `news_post.html`. Everything outside it, and

README.md

@@ -140,18 +140,15 @@ npm run watch-css   # watch and rebuild CSS automatically
 
 ## Search
 
-Search is [Pagefind](https://pagefind.app/), which indexes the built site as a
-post-processing step, so it needs no server of its own. `bundle exec rake build`
-runs it after Jekyll and writes the index to `_site/pagefind`.
-
-`rake serve` rebuilds `_site` on every change and wipes that index, so the search
-button opens on an empty index there. To try search locally, build once and serve
-the result:
-
-``` sh
-bundle exec rake build
-bundle exec rake serve-built   # http://localhost:4000/, no live reload
-```
+Search is [Pagefind](https://pagefind.app/), which indexes the HTML Jekyll
+produced rather than the Markdown behind it, so it needs no server of its own.
+`_plugins/search_index.rb` runs it whenever Jekyll finishes writing the site,
+which means `rake build` and `rake serve` both leave you with a search index
+matching the site they just produced. Nothing extra to run.
+
+Indexing adds a few seconds to a build. It needs Node, so run `npm install`
+first; without it the build says the index was skipped and carries on, which is
+enough for editing content.
 
 Pagefind builds one index per language and picks one by the `lang` attribute of
 the page the visitor is on, so each translation searches its own pages and reads

Rakefile

@@ -26,32 +26,19 @@ task build: :"build-css" do
   require "jekyll"
 
   Jekyll::Commands::Build.process({})
-
-  Rake::Task[:"search-index"].invoke
-end
-
-desc "Build the Pagefind search index over the built site"
-task :"search-index" do
-  require_relative "lib/search_index"
-
-  sh "npm run build-search"
-  SearchIndex.new.apply_fallbacks
 end
 
 desc "Serve the Jekyll site locally"
 task serve: :"build-css" do
   require "jekyll"
 
-  Jekyll::Commands::Serve.process({})
-end
-
-desc "Serve the built site without rebuilding, so its search index survives"
-task :"serve-built" do
-  require "webrick"
+  # Same pair `jekyll serve` runs. Serving on its own only hands out whatever
+  # _site already holds, which leaves a fresh clone with nothing to serve and
+  # never picks up an edit.
+  options = { "serving" => true, "watch" => true }
 
-  server = WEBrick::HTTPServer.new(Port: 4000, DocumentRoot: "_site")
-  trap("INT") { server.shutdown }
-  server.start
+  Jekyll::Commands::Build.process(options)
+  Jekyll::Commands::Serve.process(options)
 end
 
 namespace :new_post do

_plugins/search_index.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require_relative "../lib/search_index"
+
+# Pagefind indexes the HTML Jekyll wrote, so it hangs off post_write rather than
+# off a Rake task. That way `rake build`, `rake serve` and a bare `jekyll build`
+# all end with a search index that matches the site they just produced.
+Jekyll::Hooks.register :site, :post_write do |site|
+  entry = SearchIndex.new(
+    bundle_dir: File.join(site.dest, "pagefind"),
+    search_config: site.config["search"],
+  ).build
+
+  next unless entry
+
+  # The languages that fall back share another language's index, so count the
+  # indexes rather than the keys pointing at them.
+  indexes = entry["languages"].values.uniq
+  pages = indexes.sum { |index| index["page_count"] }
+  Jekyll.logger.info "Search index:", "#{indexes.size} languages, #{pages} pages"
+end

lib/search_index.rb

@@ -1,33 +1,58 @@
 # frozen_string_literal: true
 
 require "json"
-require "yaml"
 
-# Post-processes the Pagefind bundle that `npm run build-search` writes into
-# `_site/pagefind`.
+# Builds the Pagefind search index over a written site and points the languages
+# Pagefind cannot index at the one they fall back to.
 #
-# Pagefind builds one index per `lang` attribute it finds in the built site, and
-# the browser picks an index by the `lang` of the page the visitor is on. The
-# languages under `search.unsupported` in _config.yml are left out of the index
-# by the layouts, which leaves them with no index of their own. Pagefind then
-# falls back to whichever index holds the most pages, a language we do not pick
-# and that shifts as translations are added. This rewrites the bundle entry so
-# those languages resolve to `search.fallback` instead.
+# Pagefind reads the HTML Jekyll produced, so it can only run once the site has
+# been written. Jekyll deletes anything in the destination that has no source
+# counterpart, which includes the bundle from the previous build, so this runs
+# after every write rather than once. _plugins/search_index.rb does the calling.
+#
+# Pagefind builds one index per `lang` attribute it finds and the browser picks
+# an index by the `lang` of the page the visitor is on. The languages under
+# `search.unsupported` in _config.yml are kept out of the index by the layouts,
+# which leaves them with no index of their own. Pagefind then falls back to
+# whichever index holds the most pages, a language we do not pick and that
+# shifts as translations are added, so the bundle entry is rewritten to resolve
+# them to `search.fallback` instead.
 class SearchIndex
   class Error < StandardError; end
 
-  def initialize(bundle_dir: "_site/pagefind", config_path: "_config.yml")
+  PAGEFIND = File.join(File.expand_path("..", __dir__), "node_modules", ".bin", "pagefind")
+
+  def initialize(bundle_dir:, search_config:, pagefind: PAGEFIND)
+    @bundle_dir = bundle_dir
     @entry_path = File.join(bundle_dir, "pagefind-entry.json")
-    @config_path = config_path
+    @search_config = search_config || {}
+    @pagefind = pagefind
+  end
+
+  # Returns the bundle entry, or false without indexing when Pagefind is not
+  # installed, so that editing content only needs Ruby. Everywhere the site is
+  # published runs `npm ci` first.
+  #
+  # Pagefind runs silently because this fires on every rebuild under
+  # `rake serve`; it still exits non-zero when it fails.
+  def build
+    unless File.executable?(@pagefind)
+      warn "Search index skipped: #{@pagefind} is missing. Run `npm install` to search locally."
+      return false
+    end
+
+    site_dir = File.dirname(@bundle_dir)
+    raise Error, "Pagefind failed to index #{site_dir}" unless system(@pagefind, "--site", site_dir, "--silent")
+
+    apply_fallbacks
   end
 
   # Maps each unsupported language to the language it searches instead.
   def fallbacks
-    config = read_config
-    fallback = config["fallback"]
-    raise Error, "search.fallback is not set in #{@config_path}" unless fallback.is_a?(String)
+    fallback = @search_config["fallback"]
+    raise Error, "search.fallback is not configured" unless fallback.is_a?(String)
 
-    Array(config["unsupported"]).to_h { |lang| [lang, fallback] }
+    Array(@search_config["unsupported"]).to_h { |lang| [lang, fallback] }
   end
 
   def apply_fallbacks
@@ -52,19 +77,10 @@ def apply_fallbacks
 
   private
 
-  def read_config
-    config = YAML.load_file(@config_path)["search"]
-    raise Error, "No search section in #{@config_path}" unless config.is_a?(Hash)
-
-    config
-  rescue Errno::ENOENT
-    raise Error, "#{@config_path} is missing"
-  end
-
   def read_entry
     JSON.parse(File.read(@entry_path))
   rescue Errno::ENOENT
-    raise Error, "#{@entry_path} is missing, run `npm run build-search` first"
+    raise Error, "#{@entry_path} is missing, Pagefind wrote no index"
   rescue JSON::ParserError => e
     raise Error, "#{@entry_path} is not valid JSON: #{e.message}"
   end

package.json

@@ -4,8 +4,7 @@
   "description": "Ruby Language Website with Tailwind CSS",
   "scripts": {
     "build-css": "tailwindcss -i ./stylesheets/tailwind.css -o ./stylesheets/compiled.css",
-    "watch-css": "tailwindcss -i ./stylesheets/tailwind.css -o ./stylesheets/compiled.css --watch",
-    "build-search": "pagefind --site _site"
+    "watch-css": "tailwindcss -i ./stylesheets/tailwind.css -o ./stylesheets/compiled.css --watch"
   },
   "devDependencies": {
     "@tailwindcss/typography": "^0.5.19",

test/test_search_index.rb

@@ -2,27 +2,21 @@
 
 require "helper"
 require "json"
-require "yaml"
 require_relative "../lib/search_index"
 
 describe SearchIndex do
   before do
     setup_tempdir
-    @bundle_dir = File.join(TEMP_DIR, "pagefind")
+    @bundle_dir = File.join(TEMP_DIR, "_site", "pagefind")
     FileUtils.mkdir_p(@bundle_dir)
     @entry_path = File.join(@bundle_dir, "pagefind-entry.json")
-    @config_path = File.join(TEMP_DIR, "_config.yml")
-    write_config(fallback: "en", unsupported: ["bg"])
+    @search_config = { "fallback" => "en", "unsupported" => ["bg"] }
   end
 
   after do
     teardown_tempdir
   end
 
-  def write_config(search)
-    File.write(@config_path, YAML.dump({ "search" => search.transform_keys(&:to_s) }))
-  end
-
   def write_entry(languages)
     File.write(@entry_path, JSON.generate({ "version" => "1.5.2", "languages" => languages }))
   end
@@ -31,29 +25,48 @@ def read_entry
     JSON.parse(File.read(@entry_path))
   end
 
-  def search_index
-    SearchIndex.new(bundle_dir: @bundle_dir, config_path: @config_path)
+  def search_index(pagefind: SearchIndex::PAGEFIND)
+    SearchIndex.new(bundle_dir: @bundle_dir, search_config: @search_config, pagefind: pagefind)
+  end
+
+  describe "#build" do
+    it "skips indexing when Pagefind is not installed" do
+      index = search_index(pagefind: File.join(TEMP_DIR, "no-such-pagefind"))
+
+      result = nil
+      _, stderr = capture_io { result = index.build }
+
+      _(result).must_equal false
+      _(stderr).must_match(/npm install/)
+    end
   end
 
   describe "#fallbacks" do
     it "maps every unsupported language to the fallback language" do
-      write_config(fallback: "en", unsupported: %w[bg xx])
+      @search_config = { "fallback" => "en", "unsupported" => %w[bg xx] }
 
       _(search_index.fallbacks).must_equal({ "bg" => "en", "xx" => "en" })
     end
 
     it "is empty when no language is unsupported" do
-      write_config(fallback: "en", unsupported: [])
+      @search_config = { "fallback" => "en", "unsupported" => [] }
 
       _(search_index.fallbacks).must_be_empty
     end
+
+    it "raises when no fallback language is configured" do
+      @search_config = { "unsupported" => ["bg"] }
+
+      error = _(-> { search_index.fallbacks }).must_raise SearchIndex::Error
+      _(error.message).must_match(/search.fallback is not configured/)
+    end
   end
 
   describe "#apply_fallbacks" do
     it "points an unsupported language at the index it falls back to" do
       write_entry({
-        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 563 },
-        "ja" => { "hash" => "ja_def", "wasm" => nil, "page_count" => 541 },
+        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 560 },
+        "ja" => { "hash" => "ja_def", "wasm" => nil, "page_count" => 536 },
       })
 
       search_index.apply_fallbacks
@@ -64,8 +77,8 @@ def search_index
 
     it "does not leave the fallback to the largest index" do
       write_entry({
-        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 563 },
-        "uk" => { "hash" => "uk_def", "wasm" => nil, "page_count" => 564 },
+        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 560 },
+        "uk" => { "hash" => "uk_def", "wasm" => nil, "page_count" => 900 },
       })
 
       search_index.apply_fallbacks
@@ -75,8 +88,8 @@ def search_index
 
     it "leaves the other languages untouched" do
       write_entry({
-        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 563 },
-        "zh-cn" => { "hash" => "zh-cn_ghi", "wasm" => nil, "page_count" => 281 },
+        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 560 },
+        "zh-cn" => { "hash" => "zh-cn_ghi", "wasm" => nil, "page_count" => 276 },
       })
 
       search_index.apply_fallbacks
@@ -88,7 +101,7 @@ def search_index
 
     it "warns when a language meant to be skipped was indexed anyway" do
       write_entry({
-        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 563 },
+        "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 560 },
         "bg" => { "hash" => "bg_xyz", "wasm" => nil, "page_count" => 60 },
       })
 
@@ -99,7 +112,7 @@ def search_index
     end
 
     it "raises when the index it falls back to is missing" do
-      write_entry({ "ja" => { "hash" => "ja_def", "wasm" => nil, "page_count" => 541 } })
+      write_entry({ "ja" => { "hash" => "ja_def", "wasm" => nil, "page_count" => 536 } })
 
       error = _(-> { search_index.apply_fallbacks }).must_raise SearchIndex::Error
       _(error.message).must_match(/missing index "en"/)
@@ -112,17 +125,9 @@ def search_index
       _(error.message).must_match(/did the site build/)
     end
 
-    it "raises when the bundle entry does not exist" do
-      error = _(-> { search_index.apply_fallbacks }).must_raise SearchIndex::Error
-      _(error.message).must_match(/npm run build-search/)
-    end
-
-    it "raises when the config has no search section" do
-      write_entry({ "en" => { "hash" => "en_abc", "wasm" => "en", "page_count" => 563 } })
-      File.write(@config_path, YAML.dump({ "url" => "https://www.ruby-lang.org" }))
-
+    it "raises when Pagefind wrote no index" do
       error = _(-> { search_index.apply_fallbacks }).must_raise SearchIndex::Error
-      _(error.message).must_match(/No search section/)
+      _(error.message).must_match(/wrote no index/)
     end
   end
 end