Nebularr

AI-assisted development — The code, docs, and tooling in this repository were written primarily with AI coding assistants, with human direction and review.

Nebularr is a Docker-first service that ingests Sonarr/Radarr API metadata into PostgreSQL for analytics and the built-in Web UI.

Security: built-in login (session cookie + optional bearer API token), secrets encrypted at rest with an auto-provisioned key, egress policy for outbound URLs, hardened non-root container. Details, threat model, and vulnerability reporting: SECURITY.md.

Supply chain: releases are cut by the tag-driven release.yml workflow (multi-arch buildx push with SBOM + provenance (mode=max), so Docker Scout attestation policies pass); ./scripts/docker-release-build.sh --push is the manual fallback. Image base python:3.14-slim is digest-pinned; distro CVE rows without patches must wait on Debian/Python image updates. Changes per release: CHANGELOG.md.

What it does

• Runs first full sync and incremental sync (history polling).
• Accepts Sonarr/Radarr webhooks and processes a durable queue.
• Stores data in PostgreSQL with two schemas:
  • app: config/state/schedules/queue/ops tables.
  • warehouse: analytics entities + views (v_episode_files, v_movie_files, etc.).
• Exposes:
  • /healthz with app version and git SHA.
  • /metrics (Prometheus text format).
  • Web UI at / for operational visibility.

Architecture and HTTP API

Low-level architecture (single process)

• FastAPI serves the SPA, JSON APIs, Prometheus metrics, and webhook ingress on one port.
• SQLAlchemy sessions run short request-scoped transactions; sync jobs use the same pool with longer statement timeouts where configured.
• SyncService pulls from Sonarr/Radarr via ArrClient, upserts into warehouse, and updates app state (watermarks, locks, run summaries).
• SyncScheduler triggers incremental/reconcile work from cron rows stored in the database (after setup).
• Webhook path validates the shared secret, writes to app.webhook_queue, and workers process jobs with retries and dead-letter states.
• Background tasks (asyncio): Arr capability probe after startup, periodic health evaluation for outbound alert webhooks.

flowchart TB
  subgraph clients [Clients]
    br[Browser]
    arr[Sonarr / Radarr]
  end
  subgraph neb [Nebularr process]
    mw[HTTP middleware]
    rt[Router handlers in api.py]
    ss[SyncService]
    sc[SyncScheduler]
    ac[ArrClient REST]
    dbw[SQLAlchemy + repository]
    bg[Background tasks: capabilities + health alerts]
    mw --> rt
    rt --> dbw
    rt --> ss
    sc --> ss
    ss --> ac
    ss --> dbw
    bg --> dbw
  end
  pg[(Postgres app + warehouse)]
  dbw --> pg
  br --> mw
  arr --> mw
  arr --> ac

Loading

API surface (quick reference)

Area	Examples	Role
Observability	GET /healthz, GET /metrics, GET /api/status	Liveness, Prometheus text, derived health
Setup	GET/POST /api/setup/*	First-run wizard, skip, initial sync
Configuration	GET/PUT /api/config/integrations/{source}, POST .../test, /api/config/webhook, /api/config/alert-webhooks (+ POST .../test), /api/config/schedules (+ POST .../validate), /api/config/retention, /api/config/queue, /api/config/saved-views	Integrations (+ connection test), webhook verifier, alert targets (+ per-channel test), cron (+ validation/next-run preview), retention & queue policy, saved views
Sync control	POST /api/sync/<source>/<mode> — sources: sonarr, radarr; modes: full, incremental, reconcile	Manual or scripted runs
Webhook ingress	POST /hooks/sonarr, POST /hooks/radarr, POST /hooks/<source>/<instance>	Signed payloads from Arr into the queue (per-instance form for multi-instance setups)
Queue admin	POST /api/webhooks/replay-dead-letter/<source>, POST /api/webhooks/requeue/<job_id>, POST /api/webhooks/requeue-bulk	Recover failed webhook jobs (single, per-source, or bulk by status)
Library UI	GET /api/ui/* (shows, episodes, movies, runs, queue, …)	Paged JSON + streamed CSV exports for the SPA
Reporting	GET /api/reporting/dashboards, .../{key}, CSV under panels	Whitelist-only SQL; no ad-hoc SQL from browser
Danger zone	POST /api/admin/reset-data	Wipe library/sync/MAL data (auth, webhook secret, and alert/policy settings survive)

Static assets: GET /, /setup, /assets/..., and the SPA catch-all route for client-side navigation.

Typical request flows

Browser → Web UI and JSON APIs

sequenceDiagram
  participant B as Browser
  participant F as FastAPI
  participant P as PostgreSQL

  B->>F: GET / SPA shell
  B->>F: GET /api/ui/shows or reporting
  F->>P: Parameterized queries
  P-->>F: Rows
  F-->>B: JSON or CSV

Loading

Arr → webhook → warehouse

sequenceDiagram
  participant A as Sonarr or Radarr
  participant H as POST /hooks/source
  participant Q as app.webhook_queue
  participant W as Sync worker
  participant R as Arr REST
  participant WH as warehouse tables

  A->>H: Webhook JSON + x-arr-shared-secret
  H->>Q: Enqueue
  H-->>A: 2xx ACK
  W->>Q: Claim job
  W->>R: Targeted GETs
  R-->>W: Entity JSON
  W->>WH: Idempotent upsert
  W->>Q: Done or retry backoff

Loading

Operator → manual full sync

sequenceDiagram
  participant O as Operator curl or UI
  participant S as POST /api/sync/source/full
  participant L as app.job_lock
  participant Y as SyncService
  participant R as Arr API
  participant W as warehouse

  O->>S: Trigger full sync
  S->>L: Acquire lease
  S->>Y: Run full pipeline
  Y->>R: List series movies etc
  R-->>Y: Batches
  Y->>W: Upserts
  Y->>L: Release on completion

Loading

More diagrams: docs/ARCHITECTURE.md.

Quickstart

One-click stack (Postgres + App)

./scripts/one-click-all-in-one.sh

Requires an existing .env (copy from .env.example first; the script exits with an error if .env is missing—see docs/SECRETS.md). Then it starts the core stack (app URL and Postgres lines are printed when the script finishes). The script reads NEBULARR_BUNDLED_POSTGRES (default true) and syncs COMPOSE_PROFILES so bundled Postgres starts unless you opt out.

Local Docker testing (auto .env)

For quick local runs without preparing .env by hand:

./scripts/docker-local-up.sh

Creates .env from .env.example if missing, then docker compose up -d --build using only your .env (no DATABASE_URL= / COMPOSE_PROFILES= shell overrides). Same bundled-postgres profile merge as the one-click script.

Bundled Postgres vs external

• Default (local bundled Postgres): .env from .env.example sets NEBULARR_BUNDLED_POSTGRES=true and COMPOSE_PROFILES=nebularr-bundled-postgres, which starts the postgres service in docker-compose.yml. The Web UI setup can still use host postgres on port 5432 (the Docker service name on the compose network).
• External / existing Postgres: set NEBULARR_BUNDLED_POSTGRES=false in .env and remove nebularr-bundled-postgres from COMPOSE_PROFILES (or rely on ./scripts/one-click-all-in-one.sh, which strips the profile when bundled is false). Then docker compose up runs app only; in the first setup step, use your real DB hostname, port, database name, and credentials. POSTGRES_* in .env only apply to the bundled image when that service is enabled.

Core stack (optional bundled Postgres + App)

1. Copy env defaults:

cp .env.example .env

2. Start stack:

docker compose up --build

(docker compose loads .env from the project directory, including COMPOSE_PROFILES, so bundled Postgres starts with the defaults above.)

3. Validate app is up:

./scripts/smoke.sh

4. Open the Web UI at http://localhost:8080. If you did not set DATABASE_URL, the first step collects Postgres host, database name, and credentials (for bundled Postgres, match your POSTGRES_*; for external DB, use that server’s host and credentials), waits until Postgres accepts connections, runs Alembic migrations, then continues with integrations, webhook secret, admin password (recommended — protects the whole API), and schedules. Then trigger initial full sync if you have not already (add -H "Authorization: Bearer <token>" when authentication is enabled; generate a token under Integrations → Authentication):

curl -X POST "http://localhost:8080/api/sync/sonarr/full"
curl -X POST "http://localhost:8080/api/sync/radarr/full"

5. Use Reporting and Library views in the Web UI; external BI tools can still connect read-only to Postgres and query warehouse.v_* views if you grant a suitable role.

Webhook setup

Configure Sonarr/Radarr webhook target:

• URL: http://<your-host>:8080/hooks/sonarr (or /hooks/radarr)
• Header: x-arr-shared-secret: <WEBHOOK_SHARED_SECRET>

Webhooks are refused (403) while WEBHOOK_SHARED_SECRET is still the shipped default — set a real secret first.

With multiple instances of the same app, give each its own URL so events land on the right instance:

• URL: http://<your-host>:8080/hooks/sonarr/<instance-name> (must match the integration name under Settings → Integrations; the bare /hooks/sonarr form targets the default instance)

Scheduler and timezone

• Incremental and reconcile schedules are stored in the database after Web UI setup (see docs/SCHEDULER_TIMEZONE.md).
• SCHEDULER_TIMEZONE and APP_TIMEZONE env vars default to UTC and act as fallbacks when a schedule row has no timezone.

Project layout

• src/arrsync/: app code — API routers (routers/), services, clients, sync logic, scheduler. (Naming note: the distribution/package is nebularr; the Python module stays arrsync so deployed uvicorn arrsync.main:app entrypoints keep working. A rename is planned for a future major release.)
• alembic/: DB migrations.
• docker/: Postgres bootstrap.
• docs/: architecture, operations, backup/restore (canonical; docs/wiki/ is a synced snapshot for the GitHub wiki).
  • ARCHITECTURE.md, REPORTING_ARCHITECTURE.md, BRANDING.md, LOCKING_AND_DLQ.md, MIGRATIONS.md, POOLING_AND_TIMEOUTS.md, ALERTS_AND_SLOS.md, DB_BOOTSTRAP.md, BACKUP_RESTORE.md, OPERATIONS_RUNBOOK.md, COMPOSE_RESOURCE_HINTS.md, PERF_INDEXING_PLAN.md, SCHEDULER_TIMEZONE.md, SECRETS.md, WEBUI_FRAMEWORK.md, WEBUI_AGENT_WORKFLOW.md

Development

python3 -m venv .venv
source .venv/bin/activate
pip install -e .[dev]
ruff check src tests
mypy src
pytest -q

WebUI development (React + TypeScript)

cd frontend
npm install
npm run lint
npm run test
npm run build

The frontend build is emitted to src/arrsync/web/dist and served by FastAPI. The root Dockerfile copies that tree into the image (it does not run npm in Docker), so run npm run build in frontend/ after WebUI changes before docker build / docker compose build, or use CI that does the same.

For Docker Hub releases, prefer ./scripts/docker-release-build.sh --push (Buildx, linux/amd64 and linux/arm64 by default, SBOM + SLSA provenance (mode=max) attached by default; set DOCKER_ATTESTATIONS=0 to opt out). Plain docker build / --load still works for local smoke tests and skips attestations; see the script header for trade-offs.

WebUI docs

• docs/WEBUI_FRAMEWORK.md: frontend architecture, contracts, and local workflows.
• docs/WEBUI_AGENT_WORKFLOW.md: task tracker + quality guardian gates and completion criteria.
• docs/WEBUI_REFERENCE_ASSETS.md: generated 1:1 page screenshots + sample API/demo data.

WebUI page references (top-down)

Screenshots are generated from the current React app render and stored in docs/reference/webui-pages/.

1) Setup

Initial first-run flow for database/bootstrap, integration credentials, webhook secret, and initial scheduler defaults.

2) Home

Landing page that orients operators and links quickly to the main operational views.

3) Dashboard

Mission-control summary with health state, sync telemetry, queue pressure, and fast action buttons for sync operations.

4) Reporting

Analytics-focused view for dashboard panels, distribution/table outputs, and export-oriented review.

5) Library

Search, browse, and inspect synced show/movie/episode rows with paging, sorting, and export workflows.

6) Sync & Queue (Overview)

Live progress snapshot for active sync runs plus high-level queue status and quick controls.

7) Sync & Queue (Runs)

Historical run table showing source/mode/status timing and row/write outcomes for recent jobs.

8) Sync & Queue (Webhooks)

Webhook queue/dead-letter visibility with per-job status, attempts, and error detail context.

9) Sync & Queue (Manual)

Operator actions for incremental/full sync triggers, replay/reset controls, and MAL pipeline execution.

10) Integrations

Configuration surface for Sonarr/Radarr/MAL/logging/webhook settings and related connectivity state.

11) Schedules

Cron/timezone controls for incremental and reconcile cadence management.

12) Logs

In-app log stream for quick troubleshooting across sync, webhook, and runtime behaviors.

13) Not Found route

Fallback route shown when navigating to an unknown path.

GitHub wiki (optional)

A companion wiki can track how the code and in-repo docs fit together. Source files (including a navigation sidebar) live in docs/wiki/; publish them to the GitHub wiki (separate *.wiki git repository) or browse them on the default branch. Replace OWNER/REPO in the wiki with your org/repo when forking.

CI runs lint/type/tests on push and pull request. An optional Docker smoke job is available via manual workflow dispatch.