CLIP Search Project

This is an end-to-end Machine Learning system that allows searching through video collections using natural language queries (e.g., "a person playing tennis").

It leverages Contrastive Learning (CLIP) to understand video content and FAISS for vector search. The system is fully containerized, accessible via a REST API, and includes a Streamlit frontend.

---

Features

• Deep Learning Core: Custom PyTorch Lightning module wrapping OpenAI's CLIP (ViT-B/32) fine-tuned on the UCF101 dataset.
• Vector Search: FAISS (Facebook AI Similarity Search) index for efficient embedding retrieval.
• API First: High-performance FastAPI backend using asynccontextmanager for resource lifecycle management (loading heavy models once at startup).
• MLOps & Deployment:
  • Docker: Multi-stage builds optimized for CPU inference (separated API and UI containers).
  • CI/CD: GitHub Actions pipeline with ruff linting, pyrefly type checking, and Docker build verification.
• Robustness: Traffic control using Locust.

Installation

Prerequisites

• Docker & Docker Compose
• Python 3.10+ (for local development)
• uv or the package manager of your choice (uv recommended)

Quick Start with Docker

1. Clone the repository (here is example with ssh):

git clone git@github.com:Dominik-Galus/Clip-Search.git

3. Pull data and Models from: drive:

4. Launch system:

docker-compose up --build

5. Access:
   • Frontend: http://localhost:8501
   • API Docs: http://localhost:8000/docs

Local Development

If you want to run the code without Docker:

1. Install dependencies:

uv sync

2. Run API:

uv run uvicorn scripts.app:app --reload

3. Run UI:

uv run streamlit run scripts/ui.py

UI

Performance & Load Testing

The system was verified using Locust (Grafana + Prometheus would be also good for monitoring)

• The system successfully handled up to 51.7 RPS on a standard CPU environment. This demonstrates that the decoupling of the heavy CLIP model and the lightweight FAISS index is effective choice.
• With an average response time of 638ms, the search experience remains under the 1-second threshold, ensuring a smooth user experience despite heavy matrix operations involved in the text embedding process.
• The primary bottleneck is the Text Encoder inference (CLIP) performed for every query. Future optimization could involve ONNX Runtime export or quantization (int8) to further reduce latency and increase throughput.

CI Pipeline

Project have CI running when pushing on main

Stages

• Lint: Running ruff check and pyrefly check
• Build-check: Checking if the package and docker built correctly