Rootline

Rootline finds what went wrong, proves why, shows what is affected, fixes it, verifies the fix, and remembers what it learned in DataHub.

An autonomous AI detective for broken data pipelines, built for the **Build with DataHub: The Agent Hackathon**.

---

The Problem

When a data pipeline breaks — a revenue dashboard shows $0, a SLA metric suddenly drops — data engineers spend hours manually clicking across Snowflake, dbt, Looker, and Slack to find out:

• What changed?
• Which dataset is the root cause?
• What else is broken downstream?
• Has this happened before?

Rootline answers all of these automatically, in seconds, using DataHub's metadata graph as its brain.

---

How DataHub & MCP Are Used

Rootline connects to a real DataHub instance via the official DataHub MCP (Model Context Protocol) Server and invokes 9 real MCP tools:

MCP Tool	How Rootline Uses It
search	Locates affected dataset URNs in the catalog
get_entities	Fetches ownership, platform, and tags for each entity
list_schema_fields	Detects silent field type drift (e.g. INTEGER → STRING)
get_lineage	Traverses upstream/downstream table and dashboard dependencies
get_lineage_paths_between	Extracts exact causal multi-hop lineage paths
get_dataset_queries	Surfaces real SQL query patterns to ground the fix
save_document	Writes the verified resolution report into DataHub document store
add_structured_properties	Tags affected fields with resolution metadata
search_documents	Searches prior resolutions to power Memory Replay

The Live MCP Activity panel in the UI displays the actual JSON request/response for each call — not fabricated logs.

---

The 4-Step Flow

SELECT      →  Pick an active incident from the console
INVESTIGATE →  Rootline runs an 8-step MCP trace across DataHub (schema, lineage, queries)
FIX & VERIFY → Generates a grounded SQL fix; validates metric recovery ($0 → $142,500)
REMEMBER    →  Writes resolution back to DataHub; future incidents resolve via Memory Replay (0.1s)

Memory Replay: After INC-101 resolves and writes back to DataHub, running INC-102 (same pattern on a different table) completes in 2 MCP steps (~0.076s) instead of 8 — institutional memory in action.

---

Architecture

Data Pipeline Anomaly
        │
        ▼
Rootline FastAPI Backend
        │
        ├─ MCP Client ──────────────────► DataHub GMS (MCP Server)
        │    └─ 9 MCP tools (read & write)      │
        │                               schema, lineage, queries,
        ├─ Memory Replay Engine         ownership, document store
        ├─ 6-Factor Root Cause Analyzer
        ├─ Blast Radius Analyzer
        ├─ Grounded Remediation Generator
        └─ Fix Validator
        │
        ▼
Next.js Console UI
  ├─ Active incident list
  ├─ Live 8-step MCP investigation trace
  ├─ Root cause card + 6-factor confidence gauge
  ├─ Before/After metric recovery cards
  ├─ Live MCP Activity technical inspector
  └─ Step-locked action flow (Validate → Write Back)

---

Quick Start

Prerequisites

• Python 3.10+
• Node.js 18+
• A running DataHub instance (local Docker or remote)

1. Clone & configure

git clone https://github.com/dhruvil-codes/rootline.git
cd rootline
cp .env.example .env
# Edit .env: set DATAHUB_GMS_URL and optionally DATAHUB_GMS_TOKEN

2. Install backend & seed demo incidents

pip install -r backend/requirements.txt
python -m demo.seed_datahub

3. Start the backend API

python -m uvicorn backend.app.main:app --reload --port 8000

4. Start the frontend (in a separate terminal)

cd frontend
npm install
npm run dev

Open http://localhost:3000 in your browser.

5. DataHub Setup (if using the official hackathon datapack)

# Install the DataHub CLI
pip install acryl-datahub

# Load the official showcase-ecommerce datapack
datahub datapack restore --path showcase-ecommerce

---

Official DataHub Dataset

Rootline uses the showcase-ecommerce official hackathon datapack as its primary demo dataset.
It provides realistic e-commerce entities: orders, payment_events, subscription_events, with ownership, schema, lineage, and query history already ingested into DataHub.

The 5 controlled demo incidents (INC-101 through INC-105) are seeded by demo/seed_datahub.py on top of this catalog data — they are Rootline-created investigation scenarios, not planted in the official datapack.

---

Demo Flow

1. Open http://localhost:3000 → Landing page
2. Click Launch Rootline → → Incident console
3. Click Investigate with Rootline on INC-101 (Revenue Dashboard Failure)
4. Watch the 8-step MCP trace execute in real time
5. View the root cause card (98% confidence: payment_events.amount INTEGER→STRING drift)
6. Click Execute Fix Validator → Before ($0) → After ($142,500) recovery
7. Click Write Back to DataHub → resolution saved via save_document + add_structured_properties
8. Run INC-102 → observe Memory Replay: resolves in 2 steps / 0.076s from DataHub document store

See examples/ for captured outputs from each stage.

---

Technology Stack

Layer	Technology
Backend API	FastAPI + Python 3.11
DataHub Integration	DataHub MCP Server (JSON-RPC) + acryl-datahub SDK
Frontend	Next.js 14 (App Router) + TypeScript + Tailwind CSS
Database	SQLite (auto-created; incident/investigation state)
Animation	Framer Motion + shadcn Particles

---

Running Tests

# Backend unit tests
python -m pytest backend/tests

# Frontend TypeScript check
cd frontend && npx tsc --noEmit

# Frontend production build
cd frontend && npm run build

---

License

Apache 2.0 — see LICENSE.