Skip to content

Local dev + full d3 v7 / hedotools v7 / d3-shifterator v7 migration - #53

Open
andyreagan wants to merge 16 commits into
masterfrom
local-dev-setup
Open

Local dev + full d3 v7 / hedotools v7 / d3-shifterator v7 migration#53
andyreagan wants to merge 16 commits into
masterfrom
local-dev-setup

Conversation

@andyreagan

@andyreagan andyreagan commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Makes the site runnable locally from a clean checkout and migrates the entire front end to d3 v7, wiring in the published @andyreagan/d3-shifterator@7 and @andyreagan/hedotools@7.

Local dev

  • Add psycopg2-binary (was missing → Postgres backend had no driver).
  • Fix migration 0002 (defunct twython_django FK → 'self') so migrate replays from scratch.
  • local_config.dev.sh helper; db.sqlite3 (stray sqlite fallback) gitignored.
  • Fix render(..., Context(...)) → dict on all four embed views (Context is rejected by modern Django → 500).

d3 v7 migration (every page)

Templates now load d3@7 + d3-shifterator@7 + @andyreagan/hedotools@7 modules from jsDelivr instead of the local d3.js/hedotools.*. No hedonometer template loads a local d3.js anymore. Per-page glue migrated d3 v4→v7 (and v3→v7 for the movie/books chain): promise loaders, event.selection/d3.pointer, (event,d) handlers with index recovery, d3.brushX, d3.bin, d3.forceSimulation, chained .attr.

Pages migrated & browser-verified (shot-scraper, zero JS errors): timeseries (main), maps, statesankey, citysankey, cities/cities2, nyt/cbs/outside lists, movielist-ranked, movie, embed, books v3/gutenberg/harrypotter, words, shifttest.

hedotools@7 issues worked around (please fix upstream)

  • Factory bug: lens/map/barchart/sankey ship un-invoked (function(){…} with no trailing ()), unlike shifter. Worked around with a defensive instantiate-if-function shim in each glue.
  • barchart dropped its mousedown handlers (re-wired the click-to-open on the outside page).
  • lens needs hedotools.urllib.js loaded; computeHapps/nonsankey aren't in v7 (kept local, migrated).

Pre-existing data outages flagged (not migration; data gone on live server)

  • Embed daily word-vectors/vacc/<date>.csv — 404 for all dates (embed shift can't render).
  • v3 book per-id CSVs (bookdata/gutenberg-007/*) — 404 (graceful catalog fallback).
  • statesankey combined-word-vectors/ — 404 (repointed to yearly wordCounts).
  • movies "all" baseline row — gone (synthesized in glue).

🤖 Generated with Claude Code

andyreagan and others added 11 commits June 22, 2026 22:01
@andyreagan @claude
Make a fresh install/migrate work locally
2e029a3 
- Add psycopg2-binary (was missing from pyproject/lock; the postgres
  backend configured in settings had no driver)
- Repoint the Embeddable.author FK in migration 0002 from the defunct
  twython_django.TwitterProfile to 'self'. The field is removed in 0010
  and Embeddable deleted in 0011, so the target is irrelevant to the final
  schema; this lets `migrate` replay from scratch (new dev env / CI) without
  the uninstalled twython-django app.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Add local dev config helper
f24d87e 
local_config.dev.sh sources the dummy local_config.sh, overrides
DJ_DB_NAME to a dedicated 'hedonometer' database, and puts Postgres.app
on PATH. Source it instead of local_config.sh for local development:

    createdb -O andyreagan hedonometer
    . ./local_config.dev.sh
    uv run python manage.py migrate
    uv run python manage.py runserver

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Wire in d3-shifterator v7 on the timeseries page
1cc80b1 
Bump the wordshift library from @andyreagan/d3-shifterator@4 to @7. Use the
self-contained bundle (shifterator-bundle.js, its own internal d3 v7) rather
than the non-bundled build, because timeseries.js still drives the rest of
the page with global d3 v4 (callback-style d3.text/d3.csv, d3.event, etc.).
The bundle isolates the new shifterator's d3 v7 from the page's d3 v4; the
shifter is invoked purely through its stable public API (strings + arrays),
so no other changes are needed.

Verified: the Dec 25 2022 wordshift renders correctly (200 bars, real prod
data fetched cross-origin). When the page is migrated to d3 v7, switch back
to the non-bundled dist/shifterator.js sharing the global d3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate the timeseries page to d3 v7
4f991b1 
timeseries.js drove the page on d3 v4; ported it to d3 v7:
- async loaders d3.text/d3.csv/d3.json(url, cb) -> promise form
  d3.X(url).then(cb) (7 call sites); drop the v4 error-first arg on the
  events d3.json
- event listeners: the brush handler uses the passed event (event.selection)
  instead of the removed global d3.event; the 4 mousedown handlers that read
  the datum switch from (d, i) to (event, d)
- fix a latent bug: `.style(opacity, ...)` (undefined var) -> "opacity"

Template (indexlang.html): load d3 v7 (jsDelivr d3@7) instead of d3.v4, and
switch the shifterator from the self-contained bundle back to the lean
non-bundled build, now that it shares the page's global d3 v7.

Verified on d3 7.9.0: timeseries chart, event annotations, brush date-range,
the big modal wordshift (200 bars), and the small hover shift (10 bars) all
render with no JS errors. Data fetched cross-origin from prod /data (CORS *).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate maps page to hedotools v7 + d3 v7
46e2286 
Template: local d3.js/hedotools.*/topojson -> CDN d3@7 + topojson-client@3 +
d3-shifterator@7 + @andyreagan/hedotools@7 modules (init.v4, urllib, shifter,
lens, map, barchart). Kept local urllib.js (glue uses d3.urllib) and the pure
computeHapps helper (not shipped in hedotools@7).

maps-on-load.js: d3.text/json(url,cb) -> .then(); click/mousedown handlers
(d,i) -> (event,d) with index recovered from the datum / DOM position.

Two hedotools@7 gotchas worked around here (see report):
- lens/map/barchart are shipped as factories (missing the IIFE call that
  shifter has), so instantiate defensively at glue top.
- lens uses hedotools.urllib, so load hedotools.urllib.js alongside.

Verified on d3 7.9.0: choropleth (51 states), lens, wordshift (200 bars),
and state-ranking barchart all render, no JS errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate sankey pages (state + city) to hedotools v7 + d3 v7
7830164 
statesankey.html + citysankey.html: local d3.js/topojson/hedotools.* -> CDN
d3@7 + topojson-client@3 + d3-shifterator@7 + hedotools@7 (init.v4, urllib,
shifter, lens, sankey). Dropped local sankey.js (hedotools.sankey@7 is
self-contained; glue never calls d3.sankey). Kept local jquery/bootstrap/
urllib.js + computeHapps.

Glue (sankey-on-load.js, city-sankey-on-load.js): factory shim for lens/sankey;
loaders -> .then(); (d,i) handlers -> (event,d) with index recovery.

Data fix: statesankey's combined-word-vectors/*.csv are 404 (dir removed) —
under d3 v7 .then() that rejects and blocks render, so repointed to the
existing yearly wordCounts2013/2012.csv (same 51-state format). NOTE: this
changes statesankey from last-7-days-vs-prior to 2013-vs-2012; heading text
unchanged.

Verified: statesankey (51 sankey paths + shift + lens) and citysankey
(101 paths) render, no JS errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate list pages (nyt/cbs/outside) to hedotools v7 + d3 v7
6aea158 
nytlist/cbslist/outsidelist/outside templates: local d3.js/hedotools.* -> CDN
d3@7 + d3-shifterator@7 + hedotools@7 (init.v4, urllib, shifter, lens,
barchart). Kept local jquery/bootstrap/urllib.js/computeHapps.

Glue (nyt/cbs/outside-on-load.js): factory shim for lens/barchart; loaders ->
.then(); dropdown/rotate/selbutton handlers (d,i) -> (event,d) (the <a> are
data-bound so d is the datum).

hedotools@7 regression handled: the v7 barchart module dropped the mousedown
handlers that invoked barchartonclick, so outside's click-article-to-open was
dead — re-wired a click handler on the bars in outside-on-load.js. (Flag for
the hedotools repo.)

Routing note: /showcase/outside/ renders outside.html (outsidelist.html is an
orphan template, no route). Live routes verified: /showcase/{nyt,cbs,outside}/
render (25/3/50 barchart bars + 212 shift rects), zero JS errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate cities pages (cities + cities2) to hedotools v7 + d3 v7
f0a540e 
cities2.html (the live page for both /cities.html and /cities2.html routes) +
cities.html (orphan nonsankey template): local d3.js/topojson/hedotools.* ->
CDN d3@7 + topojson-client@3 + d3-shifterator@7 + hedotools@7 (init.v4, urllib,
shifter, lens, barchart). Dropped local sankey.js (unused here). Kept local
jquery/bootstrap/urllib.js/computeHapps and hedotools.nonsankey.js (not in @7).

Glue: factory shim for lens/barchart; loaders -> .then(); dropdown/selbutton/
rotate handlers (d,i) -> (event,d) with index recovery.

hedotools.nonsankey.js (kept local) also migrated to d3 v7: d3.scale.linear ->
d3.scaleLinear, object-form .attr({})/.style({}) -> chained, d3.mouse ->
d3.pointer, mouseover/mouseout (d,i) -> (event,d) with i from datum.

Verified: cities2 (304 barchart bars + 212 shift bars) and cities/nonsankey
(608 city texts) render, no JS errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate movie + embed pages to hedotools v7 + d3 v7
22ba4ba 
movielist-ranked, embed, and movie (detail) templates -> CDN d3@7 +
d3-shifterator@7 + hedotools@7 (init.v4, urllib, shifter, lens, barchart);
kept local jquery/bootstrap/typeahead/urllib.js/computeHapps and the custom
draw*/select*/spin glue.

Glue migrated d3 v4->v7 (rankedlist, embed) AND d3 v3->v7 for the movie
detail chain (movies-on-load, drawMovieTimeseries, selectChapter*,
drawLens-chapters): scale/axis/line/area API, d3.layout.histogram->d3.bin,
d3.svg.brush->d3.brushX (pixel-space, event.selection), .attr({})->chained,
d3.mouse->d3.pointer, loaders->.then, (d,i)->(event,d). Factory shim for
barchart on rankedlist. shifter stop-window setter is `_stoprange`.

embedviews.py: fix render(..., Context(filenames)) -> dict on all four embed
views (Context is rejected by modern Django -> 500). Reverted the local-only
data-fallback hack (see note). db.sqlite3 (stray sqlite fallback) gitignored.

Verified: rankedlist (barchart + shift), movie detail (timeseries + lens
brush + script; interactive shift 429 nodes) render with no JS errors.

NOTE: embed daily data (data/word-vectors/vacc/<date>.csv) and the movies
"all" baseline row are 404/gone on the live server — pre-existing data issues,
not migration; embed's shift can't render until that data is restored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate books pages (v3 / gutenberg / harrypotter) to hedotools v7 + …
f83d5b2 
…d3 v7

Templates: js/d3.js -> d3@7; hedotools.init.js + hedotools-1.0.1 init ->
@7 init.v4.js; d3-shifterator@7 before shifter + @7 hedotools.urllib.js;
shifter -> @7. Kept local jquery/bootstrap(-tour)/typeahead/urllib/spin.

Glue migrated d3 v3 -> v7 (11 files): scale/axis/line/area API,
d3.layout.histogram -> d3.bin, d3.svg.brush -> d3.brushX (pixel-space,
event.selection), d3.layout.force -> d3.forceSimulation (harry annotations),
.attr({}) -> chained, d3.mouse -> d3.pointer, loaders -> .then/.catch,
(d,i) -> (event,d). Reused the already-migrated js/{drawLens-chapters,
selectChapter-harry,selectChapterTop-harry}.js untouched. computeHappsChapters*
left as-is (pure d3.min/max/sum/mean helpers).

Gotcha: hedotools-1.0.1 shifter was a factory; v7 shifter is a singleton, so
`hedotools.shifter()` -> `hedotools.shifter` in books-on-load-v3.js.

Verified: harrypotter (/books/v2/...) renders the emotional-arc timeseries
(6 line paths + area + both brushes), zero JS errors. v3 book per-id CSVs are
404 on live (pre-existing outage; the .catch fallback shows the catalog card);
gutenberg.html has no route (migrated, not runtime-verifiable).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate words + shifttest pages to d3 v7
fd9b3cb 
words_api.html (/words/<list>/): d3.js -> d3@7; inline
d3.json(url, cb) -> d3.json(url).then(cb) (the DataTable loader).
words_table.html: d3.js -> d3@7 (d3 was loaded but unused).
shifttest.html + shifttest-on-load.js: d3.js/hedotools.* -> CDN d3@7 +
d3-shifterator@7 + hedotools@7 (init.v4, shifter); 4 d3.text loaders -> .then().

No hedonometer template loads the local js/d3.js anymore.
Verified: /words/labMT-en-v1/ loads d3 7.9.0 and populates its table, 0 errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan andyreagan changed the title Local dev setup + d3 v7 migration for the timeseries page Local dev + full d3 v7 / hedotools v7 / d3-shifterator v7 migration Jul 1, 2026
andyreagan and others added 5 commits June 30, 2026 22:23
@andyreagan @claude
Fix embedMain data URLs (real bug, not an outage)
f7cfc96 
The daily word-vector files exist on the server as <date>-sum.csv /
<date>-prev7.csv (14980 of them under word-vectors/vacc/), but embedMain
requested <date>.csv -> 404, so the embed wordshift never rendered (on prod
too). Point ref/comp at <date>-sum.csv (comparing the two days' sum vectors),
matching embedMainSimple which already used -sum.csv/-prev7.csv.

Verified /embed/main/2022-12-25/2022-12-24/ renders a real wordshift (200
bars, ref 6.27 vs comp 6.18), zero errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Make embedNYT data URL absolute (consistent with embedCBS/embedMain)
3a9e0a3 
embedNYT used a relative /data/NYT/... URL (works on prod, 404s off-site);
the NYT_labVecs/*.stripped.indexed files exist on the server, so point at
the absolute https://hedonometer.org/data URL like the other embed views.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Adopt hedotools 7.2.0: pin version, drop factory shims + barchart re-…
3dbc88b 
…wire

hedotools 7.2.0 fixes the two issues the glue worked around: lens/map/
barchart/sankey now ship as invoked IIFE instances (not factories), and
barchart re-wires its own mousedown click handler.

- Pin @andyreagan/hedotools@7 -> @7.2.0 in the dashboard templates.
- Remove the now-no-op factory-instantiation shims from 8 glue files.
- Remove the manual barchart click re-wire in outside-on-load.js (7.2.0's
  barchart handles mousedown; the manual re-wire double-fired -> article
  opened twice). Kept the hedotools.barchartonclick.test definition.

(maps.html also carries the og: meta tags added in the working tree.)

Verified zero errors on maps/statesankey/outside/nyt/cities2; the outside
bar click now opens exactly once.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Migrate v1 books page off the bundle to modular d3 v7 + hedotools 7.2.0
7635b44 
books.html (the live /books/v1/ page) loaded the monolithic books-all.js
bundle (its own old d3+jquery). Switch it to modular CDN like every other
page: d3@7 + d3-shifterator@7 + hedotools@7.2.0 (init.v4 + shifter) + the
book glue as separate scripts (d3-before-urllib order, matching movie.html).

Root glue migrated d3 v3->v7: drawBookTimeseries.js, selectChapter.js,
selectChapterTop.js (copied from the already-migrated js/books/ twins),
and books-on-load.js (7 loaders -> .then). drawLens-chapters.js already v7;
computeHappsChapters.js is pure.

Verified: /books/v1/?book=Into%20Thin%20Air and Pride and Prejudice render
the timeseries + lens + wordshift + catalog, zero JS errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@andyreagan @claude
Pin books v3/gutenberg/harrypotter templates to hedotools@7.2.0
c7813b8 
Consistency: the remaining hedotools@7 refs -> @7.2.0 so every template
pins the fixed release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andyreagan