An agentic RAG assistant that reads your technical documents, then either drafts citation-backed test cases or answers questions about them, with a human approval step and a guardrail that refuses to answer from the wrong document.
In regulated and safety-critical fields (medical devices, avionics, automotive, industrial control), every test case has to be traceable back to the documentation it verifies. Writing them by hand is slow, and asking a plain chatbot is risky because it makes things up.
TestGenRAG solves this with retrieval-augmented generation and a small agent. You upload your documents, and the system:
- Drafts formal test cases where every expected result cites the exact source passage it came from, or
- Analyzes the document to answer a question (for example, reading a blood test to assess a patient's results),
and in both cases a second AI pass verifies the output is grounded in the source, a human reviews and approves it, and you can download the result as a ready-to-use file.
It is model-agnostic (Llama, Mistral, Claude, GPT, or fully local Ollama) and runs free and offline by default.
💡 No installation needed to try it: open the live demo, paste a free NVIDIA API key, upload a PDF, and go. Your key is used only for your request and is never stored. Uploaded documents are indexed in a private per-browser-session store — other visitors cannot see or query them — and session indexes are deleted automatically after 24 hours. Still, treat the public demo as a demo: don't upload documents you wouldn't paste into any web service.
One app, three complete personalities. TestGenRAG ships as a single self-contained page (HTML + vanilla JS, no build step), yet it swaps between three fully-realised themes instantly — pure CSS over the same DOM, no reload, no rebuild, and your choice is remembered between visits. The same agentic pipeline, the same API, three very different rooms to run it in. It's live now at afnansadiq-testgenrag.hf.space/app — switch themes from the panel in the bottom-left corner.
Each section of the app becomes a genuine late-90s desktop window — purple gradient title bars, the classic – □ ✕ controls, chiselled 3D buttons, the teal void of an old CRT, even a working Start menu, taskbar, and clock. The title bars aren't a background image; they're injected live onto the real sections, so a serious RAG pipeline ends up wearing a perfectly nostalgic disguise. It's a memorable, conversation-starting front door that still does real work underneath.
A near-black canvas, oversized ghosted "01 / 02" section numerals, and bold condensed display type (Syne) give it a confident, magazine-like rhythm. The whole palette is monochrome except for one disciplined pop of acid-lime (#d4ff00) reserved for the primary action — so your eye always knows where to go next.
The same strong typographic layout, reimagined for daylight. A clean white canvas with soft pastel section tints (lavender for ingest, warm cream for the requirement step), rounded cards, and generous whitespace make it calm and presentation-friendly — ideal for screen-sharing, demos, or anyone who simply prefers light mode.
What's special: same code, zero rebuild — a retro-OS novelty, a high-contrast dark workspace, and a clean light reading mode all live in one file, switchable in a click and remembered for next time. A live "tweaks" panel (accent colour, section tints) is included too.
| Feature | What it does | |
|---|---|---|
| 🧪 | Test-case generation | Drafts 2-5 structured test cases (title, priority, preconditions, steps, expected results) for a requirement, each tied to a cited source page. |
| 🔬 | Document analysis | Answers a question about the document with a structured report: findings, assessment, recommendation, and caveats, every point cited. |
| 🛡️ | Relevance gate | Detects when the uploaded document does not match the question (e.g. a restaurant menu uploaded for a health report) and refuses rather than hallucinate. |
| ⚖️ | LLM-as-judge | A second model pass checks that every claim traces to the retrieved source; if not, the agent retrieves more and retries. |
| ✍️ | Human in the loop | Nothing counts as output until a reviewer approves it; each approval is stored with a server-side SHA-256 hash (tamper-evident record — not a full 21 CFR Part 11 audit trail). |
| 📤 | Export | Download approved test cases as a Markdown test plan or a CSV for import into Jira / TestRail, and analysis as a Markdown report. |
| 🔑 | Bring your own key | Visitors enter their own model API key in the UI, nothing to configure server-side, no shared quota. |
| 🔒 | Private sessions | Every browser session gets its own document index and registry; visitors can never see or query each other's documents, and idle session indexes expire after 24 h. |
| 🎛️ | Model picker | Choose any supported model (Llama 3.1 / 3.3, Mistral, Nemotron, or a custom ID) right in the interface. |
| 📡 | Live pipeline view | Watch the agent's steps as they run, then click any step to see exactly what it did. |
Every heavy or external integration (Redis, PostgreSQL, S3, auth, hosted LLMs) is optional and gated. The app runs end to end with zero external services using Ollama + local embeddings + SQLite.
Two flows share one FastAPI backend and one FAISS index. The browser UI (a self-contained HTML / vanilla-JS app) talks to the same-origin API.
flowchart LR
U["🧑 User in browser"]
subgraph API["⚡ FastAPI backend"]
ING["Ingest<br/>extract · chunk · embed"]
AG["🤖 LangGraph agent<br/>(test cases)"]
AN["🔬 Analysis<br/>+ relevance gate"]
end
DB[("🔎 FAISS<br/>vector index")]
LLM{{"LLM provider<br/>NVIDIA · Ollama · Claude"}}
U -->|"1 · upload PDF"| ING
ING --> DB
U -->|"2a · requirement"| AG
U -->|"2b · question"| AN
AG <--> DB
AN <--> DB
AG <--> LLM
AN <--> LLM
AG -->|"cited test cases"| U
AN -->|"grounded report"| U
This is a true cyclic agent graph, not a linear chain: if the grounding check fails, it loops back and retrieves more context before trying again.
flowchart TD
S(["Requirement"]) --> H["generate_hyde<br/>write a hypothetical spec"]
H --> R["retrieve<br/>query selection · FAISS · re-rank"]
R --> D["draft<br/>structured, cited test cases"]
D --> J{"ground_check<br/>every claim cited?"}
J -->|"✅ yes"| E(["Test cases"])
J -->|"🔁 no · widen the search"| R
- HyDE generates a hypothetical answer document from the requirement and embeds that for retrieval, matching the target document's wording far better than the raw question, so retrieval is sharper.
- Retrieve runs query selection, FAISS similarity search, and re-ranking.
- Draft produces structured cases using only the retrieved context.
- Ground check is the judge; on failure the loop widens
kand retries (capped).
The gate is the part that makes this trustworthy. It checks whether the document holds the kind of data the question needs (not whether the answer is already written out), so a blood test passes a health question, but a menu does not.
flowchart TD
Q(["Your question"]) --> R["retrieve relevant passages"]
R --> G{"🛡️ Relevance gate<br/>does the document hold<br/>the right kind of data?"}
G -->|"❌ no"| X["⚠️ Mismatch warning<br/>e.g. menu asked for a health report<br/>· no analysis produced"]
G -->|"✅ yes"| A["Analyze<br/>findings + assessment<br/>each cited to a source"]
A --> JC{"grounding check"}
JC --> O(["Report → human approval"])
- Get a free key. Sign up at build.nvidia.com (no credit card) and copy your
nvapi-...key. - Paste the key into the field at the top of the app and pick a model (Mistral is a good, fast default).
- Upload a PDF (a requirements spec, a manual, a lab report, anything). It is extracted, chunked, embedded, and indexed.
- Type your requirement or question and choose an action:
- ▶ Generate test cases for a requirement you want verified.
- 🔬 Analyze & answer for a question about the document.
- Watch the pipeline run live, then click any step to inspect the hypothetical spec, the retrieved passages, or the verdict.
- Review and approve. Read the drafted cases or the analysis, then approve & e-sign the ones you accept.
- Download your test plan (
.md/.csv) or analysis report (.md).
| Layer | Technologies |
|---|---|
| Frontend | Self-contained HTML + vanilla JS (no build step) · responsive single-page UI · three themes (Windows 98 / dark / light) · served directly by the API at /app |
| Backend / AI | Python · FastAPI · LangChain · LangGraph · RAG (HyDE, query selection, re-ranking, LLM-as-judge) · FAISS |
| PDF & data | PyPDF · PDFPlumber · Docling · dynamic page classification |
| Models | NVIDIA NIM (Llama, Mistral, Nemotron...) · Anthropic Claude · OpenAI · AWS Bedrock · Ollama (local) · sentence-transformers embeddings |
| Data & infra | PostgreSQL (SQLAlchemy, SQLite fallback) · Redis cache (in-memory fallback) · AWS S3 (optional) · Docker |
| Auth & security | AWS Cognito / Okta-OIDC JWT verification (disabled by default) |
testgenrag/
├── Dockerfile # single-stage: serves the API + HTML UI on one port (7860)
├── docker-compose.yml # app (+ optional redis / postgres)
├── render.yaml # one-click Render deploy
├── TestGenRag Design/ # the live UI: self-contained HTML (+ screenshots)
├── docs/screenshots/ # interface screenshots used in this README
├── backend/
│ ├── requirements.txt
│ ├── .env.example
│ └── app/
│ ├── main.py # FastAPI: /ingest /generate /analyze /approve /approved /health
│ ├── session.py # per-visitor session isolation: cookie, index dir, limits
│ ├── textproc.py # shared LLM-text helpers + prompt-injection guard rails
│ ├── llm.py # model-agnostic LLM + embeddings factory (bring-your-own-key)
│ ├── agent.py # LangGraph: HyDE → retrieve → draft → judge (+ retry loop)
│ ├── analysis.py # relevance gate → grounded analysis → judge
│ ├── retrieval.py # query selection + MMR / page-type re-ranking
│ ├── ingestion.py # PDF → chunks → FAISS
│ ├── extractors.py # PyPDF / PDFPlumber / Docling + page classification
│ ├── schemas.py # Pydantic contracts (test cases, analysis report, relevance)
│ ├── cache.py # Redis cache (in-memory fallback)
│ ├── database.py # PostgreSQL persistence (SQLite fallback)
│ ├── auth.py # Cognito / Okta-OIDC JWT (disabled by default)
│ └── aws.py # S3 raw-file storage (no-op if unset)
└── frontend/ # legacy SvelteKit SPA (kept for reference; not deployed)
Prerequisites: Python 3.11+ and Ollama. No Node / build step — the UI is a single HTML file the API serves directly.
# 0. One-time: pull a local model
ollama pull mistral
# 1. Backend
cd backend
python -m venv .venv && source .venv/bin/activate # macOS/Linux
# Windows: python -m venv .venv; .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env # defaults are fine, no keys needed
# 2. Serve the UI from the same origin as the API (mirrors the Docker build)
mkdir -p ../frontend/build
cp "../TestGenRag Design/TestGen RAG Redesign.html" ../frontend/build/index.html
# 3. Run it
uvicorn app.main:app --reload --port 8000 # UI: http://localhost:8000/app · API docs: /docsOpen http://localhost:8000/app, upload a PDF, type a requirement, and click Generate test cases — or ask a question and click Analyze & answer.
First run downloads the embedding model (~90 MB) once. With Ollama, the LLM runs entirely on your machine and nothing leaves it.
The hosted demo lets each visitor pick a model and supply their own key in the UI. For local development, set it in backend/.env:
| Goal | Setting |
|---|---|
| Local & free (default) | LLM_PROVIDER=ollama |
| Hosted & free | LLM_PROVIDER=nvidia_deepseek + NVIDIA_API_KEY=nvapi-... |
| Best quality | LLM_PROVIDER=anthropic + ANTHROPIC_API_KEY=... |
The same idea applies to PDF_EXTRACTOR (pypdf / pdfplumber / docling) and RERANK_METHOD (mmr / page_type / none).
Ollama cannot run on free hosting, so cloud deploys use a free hosted model via NVIDIA NIM. Full walkthrough in DEPLOY.md. Short version:
Hugging Face Spaces (recommended, 16 GB RAM free, one public URL):
- Create a new Space → SDK: Docker.
- Push this repo to it (
git push). - In Settings → Variables and secrets, set
LLM_PROVIDER=nvidia_deepseekandEMBEDDINGS_PROVIDER=huggingface. LeaveNVIDIA_API_KEYblank so every visitor brings their own key. - The Space builds the Dockerfile and serves the UI at
/app.
Render (alternative): connect the repo as a Blueprint (render.yaml); note the free tier is RAM-limited, so use OpenAI embeddings there (see DEPLOY.md).
cd backend
source .venv/bin/activate
pip install -r requirements-test.txt
pytest -qThe suite stubs the model layer (deterministic fake embeddings + a fake LLM) so the full pipeline (extraction → FAISS → agent graph → JSON parsing → /ingest + /generate + /analyze + /approve) runs with no network calls — including the agent's judge-fails→retry loop, the relevance gate (both verdicts), per-session isolation, cache invalidation on model/document change, upload size + rate limits, and the JWT algorithm allowlist. ruff check and the same tests run on every push via GitHub Actions.
The analysis feature reports what a document says and how values compare to reference ranges present in that document. It is an automated reading to assist a qualified reviewer, not a medical diagnosis, legal opinion, or financial advice, and it never prescribes treatment. The relevance gate and the mandatory human approval step exist precisely so the tool stays an assistant, not an unattended decision-maker. Always have a qualified professional confirm any consequential conclusion.
MIT. See LICENSE for details.