Update bibliographies to include additional metadata.

[stumbo]

Annotate Bibliography entries with metadata based on schema.org creative works schema and its specializations for different types of scholarly works. The end result will be pages crawled and indexed by Google and other search engines will be able to better understand and represent the material on the pages. This should allow better search results.

Additionally, in our internal search assisted AI work, this will provide additional information on the contents of these pages and provide fodder for building a more detailed model of the Interlisp space that can be leveraged in responding to queries.

What all is in this PR:

• Updated generation of bibliography pages. layouts/_partials/bibliography-json-ld.html - this piece of code writes the json-ld containing the metadata for each bibliographic item. Specific metadata and overall format is driven by the Zotero type assigned to each bibliographic entry.
• Update layouts/_partials/hooks/head-end.html to call bibliographic-json-ld.html
• Moved shared publication date functionality into layouts/_partials/bibliography-date.html
• Replaced publication date functionality in layouts/bibliography/single.html with call to bibliography-date.html
• Minor cleanup of scripts/bibSplit.pl. Also, added placeholder for concepts - goal is to start annotating Zotero entries with information on different computer science concepts addressed by different papers. This information would then flow into the bibliographic metadata and become part of the searchable knowledge base
• Updated config/_default/params.yaml - fixes a long standing issue with warning messages
• Updated content/en/history/bibliography/_index.md - the other half of the warning message fix
• Created set of test cases and added test harness in tests/ to help validate generation of metadata and correct formatting
  • Test Bibliography metadata for each type of bibliography entry
  • Validates front matter on each bibliography page is formatted correctly
  • Validate that all html links on the website are correctly formatted
  • Validate Hugo build process
• Created params file used when running test cases, config/testing/hugo.yaml
• Updated .github/workflows/gh-pages.yml to run the test suite as part when a push is made to a branch and when merged into main.

To do: Further revise test cases - augment hardcoded markdown files to a combination of hardcoded and actual files generated from Zotero output. Want to flag errors that occur if Zotero output or processing change.

---

The resulting page and it's metadata can be viewed using Google's Rich Results test: https://search.google.com/test/rich-results/result/r%2Farticles?id=1glR_Xd8sI4btiGYnczSdg

Note: Pages on the staging site are set to not index - you can safely ignore indexing errors.

[pamoroso]

I have no idea how to test the new metadata features but browsing and searching the bibliography seems to be working fine on the staging server, except for one test case. I entered "knowledge programming in loops" in the search box expecting this bibliography entry to show up in the results:

Stefik, M., Bobrow, D. G., Mittal, S., & Conway, L. (1983). KNOWLEDGE PROGRAMMING IN LOOPS. 11.

But the entry doesn't show up at all in the search results.

[stumbo]

There isn't much you can do to test it via searching at present. This PR only adds additional metadata to each bibliographic entry. Once this PR is merged and built into the production website and recrawled the metadata will become available and usable by search engines. Ideally the result will be better, more informative search results.

Specifically, this PR adds a script containing metadata to each bibliographic entry. Looking at the Knowledge Programming in Loops article: The following is injected into the page:

<script type="application/ld+json">{
  "@context": "https://schema.org",
  "@type": "ScholarlyArticle",
  "author": [
    {
      "@type": "Person",
      "familyName": "Stefik",
      "givenName": "Mark",
      "name": "Mark Stefik"
    },
    {
      "@type": "Person",
      "familyName": "Bobrow",
      "givenName": "Daniel G.",
      "name": "Daniel G. Bobrow"
    },
    {
      "@type": "Person",
      "familyName": "Mittal",
      "givenName": "Sanjay",
      "name": "Sanjay Mittal"
    },
    {
      "@type": "Person",
      "familyName": "Conway",
      "givenName": "Lynn",
      "name": "Lynn Conway"
    }
  ],
  "dateModified": "2025-10-24T21:18:05Z",
  "datePublished": "1983-01-01",
  "description": "Early this year fifty people took an experimental course at Xerox PARC on knowledge programming in Loops. During the course, they extended and debugged small knowledge systems in a simulated economics domain called Truckin'. Everyone learned how to use the Loops environment, formulated the knowledge for their own program, and represented it in Loops. At the end of the course a knowledge competition was run so that the strategies used in the different systems could be compared. The punchline to this story is that almost everyone learned enough about Loops to complete a small knowledge system in only three days. Although one must exercise caution in extrapolating from small experiments, the results suggest that there is substantial power in integrating multiple programming paradigms.\n",
  "inLanguage": "en-US",
  "name": "KNOWLEDGE PROGRAMMING IN LOOPS\n",
  "pagination": "11",
  "sameAs": [
    "https://www.markstefik.com/wp-content/uploads/2011/04/1983-AI-Mag-Stefik-LOOPS.pdf",
    "https://www.zotero.org/groups/2914042/items/A7ZZFYKB"
  ],
  "url": "https://stumbo.github.io/InterlispDraft.github.io/history/bibliography/a7zzfykb/"
}</script>

All of this is separate from the AI search activities. A secondary task will be to incorporate the metadata into those data used by the AI Search Engine work I've started. That will require additional work to expand the data collected from the webpages, the data we currently pass into the LLM has minimal metadata with it.

Prior to moving this PR out of the draft state, I need to spend additional time reviewing the generated metadata and ensuring it is accurate, identifying gaps and documenting the schema. We may decide there is additional upstream work we need to do either with Zotero or extracting information from Zotero,

Additional questions that need to be addressed are whether the format chosen for representing metadata is the best for our needs (JSON-LD). It seems to be the preferred method for SEO related activities. But, it may be less ideal for integration with AI.

[pamoroso]

I confirm I see the metadata in the page source.

[pamoroso]

At commit 3940d92 the bibliography still seems to have no issues on the staging site.

[pamoroso]

Still nothing unusual on the staging site at commit 9222f83.

[pamoroso]

The staging site is still working with no issues at commit 90cae4c.

[pamoroso]

I can find no issues on the staging site at commit 4c2749c but I see there are failed checks.

[pamoroso]

Still nothing unusual at commit a8e01c4.

[pamoroso]

Looks good to me.

[stumbo]

@Eleanor-Young Adding you to this PR. Herb asked that you weigh in. The general idea is to push more metadata into each bibliography entry. Information that can be used by search engines to provide better results. Please give it a quick look and let me know if you have any thoughts, questions or concerns.

[Eleanor-Young]

Thanks for the tag, I don't check GitHub as often as I should.

Better searchability is probably good, that all sounds swell.

Just to be sure I understand ground level of what's happening, though, my main questions are: 1) what metadata is being pushed, 2) where's it coming from, and 3) how does it change the resultant page?

Is it about grabbing more metadata from what's in Zotero and putting it on the website? Or am I missing the mark here?

[stumbo]

Thanks for the tag, I don't check GitHub as often as I should.

Better searchability is probably good, that all sounds swell.

Just to be sure I understand ground level of what's happening, though, my main questions are: 1) what metadata is being pushed, 2) where's it coming from, and 3) how does it change the resultant page?

Is it about grabbing more metadata from what's in Zotero and putting it on the website? Or am I missing the mark here?

Yes, it is all about grabbing metadata from Zotero and putting it on the website. What I am doing is taking what we currently retrieve from Zotero and formatting it and adding it into the a metadata section on each bibliographic entry. The changes are not visible to a human reading a page, but are visible and formatted for a web crawler and search indexer to find them and understand them.

Here's an example, if you were to view the html for the bibliographical entry, "A graphical inference mechanism", you'd find this block of code in it:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "ScholarlyArticle",
  "author": [
    {
      "@type": "Person",
      "familyName": "Mott",
      "givenName": "Peter",
      "name": "Peter Mott"
    },
    {
      "@type": "Person",
      "familyName": "Brooke",
      "givenName": "Simon",
      "name": "Simon Brooke"
    }
  ],
  "dateModified": "2025-10-23T07:05:17Z",
  "datePublished": "1987-01-01",
  "description": "This paper describes an inference system which lends itself to graphical representation. An implementation of the system is described, and its application in a legislation based domain is illustrated. The methodology for knowledge elicitation which the system is intended to support is briefly indicated. The algorithm is described, and semantics for the system are given.\n",
  "inLanguage": "en-US",
  "isPartOf": {
    "@type": "Periodical",
    "name": "Expert Systems"
  },
  "issueNumber": "2",
  "name": "A graphical inference mechanism",
  "pagination": "106-117",
  "sameAs": [
    "https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-0394.1987.tb00133.x",
    "https://www.zotero.org/groups/2914042/items/EA3EKFV7"
  ],
  "url": "https://stumbo.github.io/InterlispDraft.github.io/history/bibliography/ea3ekfv7/",
  "volumeNumber": "4"
}
</script>

Schema.org provides a structured format for defining objects - in this case a Scholarly Article, the authors are all listed in a standardized format, publication date, description, information on the periodical it was published in, name of the article, page range, other well-known locations that house information on the article and lastly the url (This points to my staging site -- a location that is set to not be crawled and not be discoverable by search engines -- we want them finding our main site, not my site) and lastly the volume number of Expert Systems that this article appeared in.

Google, Bing, or other crawlers that index our site will be able to use this information to better understand the contents of each bibliographical entry. Reading the web page the information is there and easily understandable by a human reader. This just makes it easily understandable by a computing system.

Longer term as we explore improving our own internal search system, I plan to leverage this information.

[Eleanor-Young]

Gotcha, understood. That all sounds good to me! I think pulling more metadata from the Zotero to put on the website sounds good all-around.