VidEngram encodes a long video once into a reusable, inspectable memory of segments, episodes, and entities, then answers later questions with a text-only planner, improving long-video QA without re-encoding the video per query.
Long-video question answering is wasteful and lossy when a multimodal model must re-encode the same video for every query: context limits force sparse sampling or truncation, and no reusable representation remains for later questions. We present VidEngram, an agentic video QA system that builds a persistent hierarchical memory once at ingest and answers later questions from that memory. VidEngram captions video and audio jointly with Qwen2.5-Omni, consolidates segment memories into episode and entity layers, and uses a text-only agentic planner with specialized retrieval tools to produce grounded answers. The web interface makes this process inspectable: users can watch memory formation, inspect retrieval traces, verify cited evidence in the video player, and explore entity relationships. On the full OmniVideoBench, VidEngram with a GPT-5-mini planner outperforms a strong single-pass Qwen2.5-Omni baseline (37.0% vs. 32.3%), with gains concentrated on longer videos where single-pass coverage becomes sparse. Because perception is paid once, subsequent queries use low-cost text-only planning.
| Feature | Description |
|---|---|
| Native multimodal understanding | Qwen2.5-Omni processes video frames and audio in a single pass — no separate vision/audio pipelines |
| Memory Manager three-layer memory | Structured hierarchy: MemCell → Episode summary → Entity profile, with hybrid retrieval (BM25 + vector via RRF) |
| ReAct agentic Q&A | 6 tools, autonomous dispatch, evidence-sufficiency checks before answering |
| Forced timestamp citations | All answers include [Video M:SS - N:SS] anchors grounded in actual video time |
| Streaming ingest architecture | Speech and segment memories written concurrently as captions arrive |
| Web UI + knowledge graph | Real-time subtitle overlay, scene navigation, D3.js entity relationship graph |
flowchart LR
VID([🎬 Video Input])
subgraph CREATE["🧠 Create Memory"]
direction TB
A["① ASR Transcriber
──────────────
Whisper API
speech → transcript + timestamps"]
B["② Temporal Segmenter
──────────────
Whisper ASR sentence
boundaries (~7–8s segments)"]
C["③ Captioner
──────────────
Qwen2.5-Omni-7B · vLLM
11-field structured output"]
A -->|transcript| B
B -->|segments| C
end
subgraph INGEST["💾 Ingest Memory"]
direction TB
D["④ Consolidator
──────────────
Jaccard dedup · episode grouping
cross-episode entity resolution"]
E["⑤ Memory Manager Writer
──────────────
POST /api/v1/memories
MongoDB + ES + Milvus"]
D -->|consolidated| E
end
subgraph RETRIEVE["🔍 Retrieve Memory"]
direction TB
F["⑥ Agentic Query Agent
──────────────
ReAct reasoning loop"]
G["search_episodes · search_profiles
search_deep · look_at_clip
search_speech · get_timeline"]
F --> G
end
VID --> CREATE
CREATE -->|captions stream| INGEST
INGEST -->|retrieval index| RETRIEVE
-
Memory Manager as the Memory Backbone — All consolidated memories are stored in the Memory Manager's native three-layer hierarchy (MemCell → Episode → Entity Profile). Retrieval fuses BM25 keyword search and dense vector search via reciprocal rank fusion (RRF) in a single hybrid call, with an LLM-guided reranking stage for multi-hop agentic retrieval — capabilities provided entirely by the Memory Manager with no custom retrieval code needed.
-
Structured Three-Layer Memory Pipeline — Not naive caption→RAG. Three-stage consolidation (dedup, episodes, profiles) creates a hierarchical memory structure that enables multi-hop reasoning, mapping directly onto the Memory Manager's native storage layers.
-
Memory-First Reasoning — The design goal is that all reasoning over long videos — and across multiple videos — should rely entirely on the Memory Manager's long-term memory, without re-analyzing the original video. The Memory Manager natively supports cross-video retrieval, making multi-video reasoning an architectural property of the system rather than an afterthought.
-
Agentic ReAct Orchestrator — The query agent plans which tools to use, executes searches and video analysis, observes results, and iterates. Six tools cover fast retrieval, profile lookup, multi-hop retrieval, speech search, video grounding, and timeline queries.
-
Context Grounding — When memory alone isn't enough, the agent extracts the specific video clip and re-analyzes it with Qwen2.5-Omni, verifying and enriching its answer with fresh multimodal evidence.
For the complete step-by-step setup guide, see E2E_SETUP.md.
- Python 3.10+,
uv, Docker & Docker Compose, ffmpeg - GPU with ≥30GB VRAM (Qwen2.5-Omni + Embedding + Reranker), or 2 GPUs to split
Download to a local directory on your GPU server:
| Model | Size | Purpose |
|---|---|---|
| Qwen2.5-Omni-7B | ~14GB bf16 | Video captioning + video grounding |
| Qwen3-Embedding-4B | ~8GB bf16 | EverMemOS vector embeddings |
| Qwen3-Reranker-4B | ~8GB bf16 | EverMemOS search reranking |
VidEngram splits compute across two machines:
flowchart LR
subgraph LOCAL["💻 Local Machine"]
L["EverMemOS (port 8001)
MongoDB / ES / Milvus / Redis (Docker)
VidEngram Backend (port 7860)"]
end
subgraph REMOTE["🖥️ Remote GPU Server"]
R["Qwen2.5-Omni-7B (port 8091)
Qwen3-Embedding-4B (port 8000)
Qwen3-Reranker-4B (port 12000)
Videos SCP'd here; ffmpeg/Qwen run via SSH"]
end
LOCAL -->|SSH / API| REMOTE
Use SSH port forwarding to connect local services to the remote GPU server:
ssh -L 8091:localhost:8091 -L 8000:localhost:8000 -L 12000:localhost:12000 user@remote-gpu-server| # | Service | Command | Port |
|---|---|---|---|
| 1 | Docker infra | cd EverMemOS && docker compose up -d |
27017, 19200, 19530, 6379 |
| 2 | vLLM servers (remote) | bash serve_all.sh |
8091, 8000, 12000 |
| 3 | EverMemOS | cd EverMemOS && uv run python src/run.py --port 8001 |
8001 |
| 4 | VidEngram backend | uvicorn backend.server:app --host 0.0.0.0 --port 7860 |
7860 |
Then open http://localhost:7860 in your browser.
For CLI usage:
python -m demo.cli ingest path/to/video.mp4
python -m demo.cli batch path/to/videos/ # batch-ingest a directory
python -m demo.cli query path/to/video.mp4 "What was the main argument?"
python -m demo.cli chat path/to/video.mp4For full setup details including .env configuration, Memory Manager setup, GPU layout tips, troubleshooting, and batch ingestion of many videos, see E2E_SETUP.md.
videngram/
├── videngram/
│ ├── __init__.py
│ ├── config.py # Dataclass configs (Qwen, EverMemOS, Segmenter, Consolidator, Agent, ...)
│ ├── utils.py # Data classes (VideoSegment, Caption, ConsolidatedMemory,
│ │ # MemoryResult, AgentAction, AgentResponse) + timestamp helpers
│ ├── pipeline.py # End-to-end orchestration (ingest + query)
│ ├── segmenter.py # ASR-guided temporal segmentation
│ ├── captioner.py # Qwen2.5-Omni structured captioning (11 fields, local + external API)
│ ├── transcriber.py # Speech transcription (Whisper-compatible ASR)
│ ├── consolidator.py # Memory consolidation (dedup → episodes → entity profiles)
│ ├── memory_writer.py # EverMemOS ingestion adapter (streaming, concurrent writes)
│ ├── memory_reader.py # EverMemOS retrieval — fast (RRF/BM25/embedding) + agentic
│ ├── agent.py # ReAct agent with 6 tools
│ └── visualizer.py # t-SNE memory embedding visualization
├── backend/
│ ├── server.py # FastAPI backend + serves frontend/index.html (port 7860)
│ └── graph_builder.py # Async entity relationship graph extraction (background task)
├── frontend/
│ └── index.html # Single-file Web UI — subtitles, scene nav, D3.js graph (no build step)
├── demo/
│ └── cli.py # CLI: ingest / batch / query / chat
├── config/
│ └── default_config.yaml # Default YAML configuration
├── serve_all.sh # vLLM model server launcher (Omni + Embedding + Reranker)
├── docker-compose.yml # MongoDB, Elasticsearch, Milvus, Redis (local dev)
├── requirements.txt # Python dependencies
├── pyproject.toml # Package metadata (Python 3.10+)
├── .env.template # Environment variable template
└── E2E_SETUP.md # Full deployment guide
| Module | VidEngram Implementation |
|---|---|
| Multimodal Encoding | captioner.py: Qwen2.5-Omni 11-field structured captions (unified AV) |
| Temporal Segmentation | segmenter.py: Whisper ASR sentence-boundary segmentation (~7–8 s segments); dedup filtering in consolidator.py |
| Memory Consolidation | consolidator.py: episode summaries + entity profile construction → Memory Manager MemCell/Episode/Profile |
| Agentic Reasoning | agent.py: ReAct agent tool dispatch over Memory Manager memories |
| Fast Retrieval | memory_reader.py: Memory Manager RRF / BM25 / embedding hybrid search |
| Deep Retrieval | memory_reader.py: LLM-guided multi-hop retrieval + video grounding via look_at_clip |
| Structured Storage | memory_writer.py: Memory Manager MemCell → Episode → Profile hierarchy |
| Component | Version | Purpose |
|---|---|---|
| Qwen2.5-Omni-7B | — | Multimodal video+audio→text understanding and video grounding |
| Qwen3-Embedding-4B | — | EverMemOS vector embeddings |
| Qwen3-Reranker-4B | — | EverMemOS search reranking |
| Planning LLM | — | Text-only agentic planner (swappable, e.g. GPT-5-mini or self-hosted Qwen2.5-7B) |
| vLLM | ≥0.16.0 | Model serving with OpenAI-compatible API |
| EverMemOS | ≥1.2.0 | Structured long-term memory system |
| FastAPI / uvicorn | — | Backend web server |
| MongoDB | 7.0 | EverMemOS document storage |
| Elasticsearch | 8.15 | BM25 keyword search |
| Milvus | 2.4 | Vector similarity search |
| Redis | 7.x | Caching layer |
| ffmpeg | — | Video segmentation and clip extraction |
If you use VidEngram, please cite:
@misc{videngram2026,
author={Zinuo Cheng and Yueqian Lin and Hai "Helen" Li and Yiran Chen},
title={VidEngram: Agentic Long-Term Video Understanding via Structured Hierarchical Memory},
year={2026}
}Also cite the foundational works:
- HippoMM: Lin et al., "HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding", arXiv:2504.10739
- EverMemOS: EverMind AI, https://github.com/EverMind-AI/EverMemOS
- Qwen2.5-Omni: Qwen Team, Alibaba Cloud
MIT