feat(subtitles): add streaming subtitle adapter framework - #1757
feat(subtitles): add streaming subtitle adapter framework#1757semantic-craft wants to merge 3 commits into
Conversation
Introduce a systematic adapter framework for official bilingual subtitles on streaming sites, with Netflix as the first adapter: - StreamingSiteAdapter + StreamingSubtitlesFetcher: shared source/target track selection and official-to-official bilingual alignment, in one place. - MAIN-world interceptor captures Netflix's own subtitle download URLs from its manifest JSON (ttDownloadables), ignoring non-subtitle media; replays missed manifests after a late extension inject. - One streaming-site registry; new platforms plug in by adding an adapter. Directly addresses the maintainer request on mengxi-ream#1751 for a systematic, non one-off design. Kept under the new-contributor changed-line threshold.
🦋 Changeset detectedLatest commit: 99712c7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
Bot
added
feat
contrib-trust:new
Contributor trust score16/100 — New contributor This score estimates contributor familiarity with Outcome
Score breakdown
Signals used
Policy
Updated automatically when the PR changes or when a maintainer reruns the workflow. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0ab6962fde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (++attempts >= WATCH_WAIT_MAX_ATTEMPTS) { | ||
| stop() |
There was a problem hiding this comment.
When the content script starts on a Netflix browse page, this timeout removes the only listener that can call bootstrapRuntime() after about 20 seconds. Netflix navigation to /watch/... is SPA-only, so the content script will not run again; users who spend more than 20 seconds choosing a title get no subtitle UI until a full page reload.
Useful? React with 👍 / 👎.
| measureHeight: () => 0, | ||
| checkVisibility: () => true, | ||
| }, | ||
| getVideoId: () => window.location.pathname, |
There was a problem hiding this comment.
Returning every pathname as a video id makes non-player routes participate in the generic navigation reset. After a user leaves /watch/... for a browse/details page, resetForNavigation() clears the session id; the next SPA navigation back to /watch/... is then ignored by videoIdChanged because there is no previous session id, leaving the Netflix adapter without a fresh scheduler for the new title.
Useful? React with 👍 / 👎.
- index.tsx: keep listening for SPA navigation to /watch/ instead of removing the listener after ~20s, so users who browse a while before picking a title still get the subtitle UI without a reload. - netflix/config.ts: scope getVideoId to /watch/<id> (null on browse pages) so leaving a title and returning isn't dropped by the navigation-reset logic.
|
Thanks @codex — both points addressed in
|
|
To use Codex here, create an environment for this repo. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6609c86d41
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (event.data?.type === STREAMING_SUBTITLE_TRACKS_TYPE && Array.isArray(event.data.tracks)) { | ||
| event.data.tracks.forEach((track: StreamingTrack) => { | ||
| const key = getTrackKey(track) | ||
| if (key) | ||
| tracksByKey.set(key, { ...track, pagePath: track.pagePath ?? getCurrentPagePath() }) |
There was a problem hiding this comment.
This accepts any window.postMessage from the Netflix page as trusted subtitle tracks. In a Netflix page context (or an XSS/third-party script running there), a script can post READ_FROG_STREAMING_SUBTITLE_TRACKS with an arbitrary url; when auto-start is enabled or the user turns subtitles on, resolveTrackText later sends that URL through backgroundFetch(..., { credentials: "include" }) using the extension's host permissions, so untrusted page data can drive privileged credentialed cross-origin requests and render subtitle-like responses back into the page UI. Please authenticate the capture path or at least restrict accepted track URLs to known Netflix subtitle hosts before storing them.
Useful? React with 👍 / 👎.
| window.fetch = async function (input: RequestInfo | URL, init?: RequestInit) { | ||
| const response = await originalFetch.call(this, input, init) | ||
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | ||
| void response.clone().text().then(text => inspectJson(url, text, response.headers.get("content-type"))).catch(() => {}) |
There was a problem hiding this comment.
When Netflix uses fetch for non-JSON or large resources, this clone is fully consumed as text before inspectJson checks the content type or URL. That means media chunks or other large responses fetched through this patched API can be duplicated and decoded just to be ignored, which can add avoidable memory/CPU pressure during playback; gate on response.headers.get("content-type")/the request URL before calling clone().text().
Useful? React with 👍 / 👎.
| "@read-frog/extension": patch | ||
| --- | ||
|
|
||
| Add a streaming subtitle adapter framework with Netflix as the first adapter. On Netflix it pairs the platform's own official source-language captions with its official target-language subtitles into bilingual cues — an official-to-official comparison rather than machine translation. Other streaming sites plug into the same shared capture + alignment pipeline through one adapter registry. |
There was a problem hiding this comment.
The root AGENTS.md PR Notes say to “use conventional commit format for the changeset content,” but this release note is plain prose. This user-facing feature changes @read-frog/extension, so the changeset should start with conventional-commit wording such as feat(subtitles): ... to satisfy the repository release-note convention.
Useful? React with 👍 / 👎.
semantic-craft
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99712c71c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else { | ||
| await this.processTranslatedSubtitles() | ||
| } | ||
| this.startLiveCapture() |
There was a problem hiding this comment.
When official Netflix source/target tracks are available, the branch above renders pre-aligned bilingual cues and does not create a TranslationCoordinator; this unconditional call still starts DOM live capture afterward. Those live cues are source-only and use current playback time as their start, so titles whose manifest exposes tracks can get untranslated duplicate/extended cues merged into the scheduler. Start live capture only when the static/downloadable subtitle fetch actually fell back to no fragments.
Useful? React with 👍 / 👎.
| if (previous && !previous.isNoneTrack) | ||
| player.setTimedTextTrack(previous) |
There was a problem hiding this comment.
When the user had Netflix captions off, previous is the none track, but this stop callback skips restoring it because previous.isNoneTrack is true. Live capture switches the player to the source track before reading DOM captions, so disabling Read Frog then removes the hiding CSS while Netflix native captions remain enabled. Restore the saved none track, or explicitly switch back to none, as part of stopping capture.
Useful? React with 👍 / 👎.
| }, | ||
| // Only /watch/<id> pages carry a video id; browse/details pages return null so | ||
| // they don't trigger a navigation reset that would drop the next title's session. | ||
| getVideoId: () => window.location.pathname.match(/^\/watch\/([^/?#]+)/)?.[1] ?? null, |
There was a problem hiding this comment.
Returning null on browse/details URLs means a /watch/... -> non-watch SPA navigation is ignored by UniversalVideoAdapter.videoIdChanged, which requires a truthy current video id before it resets. If subtitles are active when the user exits playback, the scheduler/live capture and current subtitle state can remain mounted on body, leaving stale overlay/capture state over Netflix browsing until another watch navigation occurs. Add a separate leave-watch cleanup instead of suppressing all handling for null ids.
Useful? React with 👍 / 👎.
1 participant
Summary
This supersedes #1751 with the systematic adapter shape requested there: streaming subtitle support now goes through one shared pipeline plus a site adapter/registry entry, with Netflix as the first adapter.
Netflix has two paths:
The MAIN-world bridge matters because Netflix's player API is not visible from the extension isolated world. The content script now coordinates through
postMessage; the interceptor does the player track switch and live cue capture wherewindow.netflixactually exists.Size / Review
Final diff: 991 insertions / 27 deletions, kept under the 1000-line threshold. HBO stays out of this PR.
Real Browser Evidence
Verified in the user's Chrome profile on
https://www.netflix.com/watch/81613638after reloading the unpacked extension:zh-Hantto English CC (bcp47: en,rawTrackType: CLOSEDCAPTIONS);Probably around the time Emily and I set our boat out on the lake.plus the Chinese translation.Tests
pnpm type-checkpnpm exec eslint ...on touched subtitle/interceptor filesWXT_SKIP_ENV_VALIDATION=true pnpm buildSKIP_FREE_API=true pnpm exec vitest runtargeted subtitle tests: 5 files / 85 testsCloses #1751.