Updateable indexes?

[justsml]

Is it possible to make the index modifiable? Or partially cachable?
I was thinking of possibly using Sqlite or duckdb to maintain a "local" updateable index staging file, which then generates the file output?

[bglw]

So far it hasn't been a goal, since a lot of builds happen in clean CI environments anyway. Some large sites can take a while to index, though in my experience Pagefind is still not using the bulk of the total build time compared to the site generation.

Firstly, the pattern of the Pagefind index means nearly any change to the input data will invalidate the cache for all index chunks. One of the more common changes is adding a page to a site, and the words in that site are very likely to be distributed over all index chunks, which means they all need to cache bust. So any incremental feature would mainly be built to improve the speed of the CLI indexing a site.

In the past I have looked at having Pagefind re-ingest its own index to then incrementally update it, but you don't gain any meaningful performance there since it's parsing about same amount of data.

Another approach is to provide Pagefind a list of new files and files that have changed, which could be used to build an extra index patch. The original index would remain untouched, and the small index patch would also be loaded at search time. It adds bandwidth though, and would only work for some number of rebuilds before you need to rebuild the full thing. There are some ideas that fall out of this, like partitioning the index by month/year and only updating parts as needed, but that needs config and documentation.

Finally there is the option of having Pagefind write some kind of database somewhere, though I don't love the concept.

• It's something that you need to avoid putting into the output site.
• If it's in the repository path it's something you need to tell people to gitignore.
• If it's tucked away in a system path it's something you need to tell people to cache in CI.

All options are predicated on knowing what files have changed since the prior build, which is also difficult to guarantee in CI. System modification dates are often meaningless, and parsing git histories can be fraught. Plus you do pay some speed (and memory) penalty on building/writing/reading/parsing any extra database while indexing the site.

My gut feeling, from the couple of times I have looked at the surface of this, is that Pagefind's parsing is generally quite fast and most incremental approaches don't save enough time for the extra code maintenance / complexity to be worth it.

[dscho]

I also would love to see something like incremental updates, even if I totally agree that it is often not so easy to determine which files have changed.

As a point of reference: I added Pagefind to https://git-scm.com/ a year ago (at least that's when the change landed), and one quite common update of that site is the Git version, which changes one line in hugo.yml. That just changes a property, though, which could be used anywhere in the site. And of course it is quite difficult to figure out exactly which of the generated files in public/ change as a consequence!

Some large sites can take a while to index, though in my experience Pagefind is still not using the bulk of the total build time compared to the site generation.

About that... I don't know whether git-scm.com qualifies as large. The compressed pages, including Pagefind are just shy of 400 megabytes. It's definitely larger than many blogs. I tried to find larger sites that are also deployed via GitHub Actions, but failed to find any.

In any case, here is the most recent deployment of that site. It ran for 5m49s, i.e. 349 seconds, total. 44 seconds of that were spent by Hugo to generate that site, a (probably inefficient) find invocation then spent 15 seconds on ensuring HTTPS, installing Pagefind took 10 seconds, and Pagefind reported to have spent 141 seconds after that. For comparison, the deployment took 1m15 seconds (start to finish, you can toggle the timestamps via the sprocket icon on the upper right), link-checking via lychee 14 seconds, installing Playwright and running the tests took a combined 21 seconds.

Here is a diagram:

pie showData
  title "git-scm.com"
  "setup" : 1
  "checkout" : 18
  "setup" : 5
  "hugo" : 44
  "https" : 15
  "install-pagefind" : 10
  "pagefind" : 141
  "compat" : 2
  "deploy" : 75
  "lychee" : 14
  "playwright" : 21

Loading

tl;dr in git-scm.com's instance, I would contend that Pagefind does use the bulk of the total deployment time.

I am not trying to complain, but I want to add my data point to the discussion. I am incredibly grateful for Pagefind!

[bglw]

ah fantastic, thanks for the data points!

A good first step will be to just do a performance pass on Pagefind in general. I've never looked hard at the indexing speed, since it's always been "fast enough" (for me). Which is good news, because it means there will be a lot of easy wins in there to improve performance on a cold index 😄

(After that, we can look at whether it still makes sense to do something incremental, and we'll have better numbers to compare & improve)

[bglw]

In saying that, it isn't at the top of my current list, so I don't know exactly when I'd be likely to get to it.

[dscho]

If you could write up some instructions and/or outlines how to do such a performance pass (can be free-form as far as I am concerned, I often do such things on my phone by dictating via FUTO voice input), I could try to find some time. Sounds like fun!

[dscho]

@bglw unless it is super-complicated, I would love to hear how you would pinpoint performance bottlenecks in Pagefind, so that I could potentially play around with this...