Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
api
api
 
 
libs
libs
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
TODO.md
TODO.md
 
 
main.py
main.py
 
 
parameters.yaml.example
parameters.yaml.example
 
 
pyproject.linux.toml
pyproject.linux.toml
 
 
pyproject.toml
pyproject.toml
 
 

Repository files navigation

SD API Backend

A FastAPI-based backend service for Stable Diffusion image generation, inpainting, multi-model merging, and model management. Supports SD 1.5 and SDXL pipelines with asynchronous job processing, model caching, and optional LoRA adapter integration.

This is a rewrite effort of the SD-WebApp project, and has been brewing for a bit now. The idea is to decouple the UI from the actual inference backend. This repo will host the backend code.

Implemented Features

  • Text-to-Image Generation - Submit async jobs for SD 1.5 and SDXL models, receive results via job status polling
  • Inpainting - Image inpainting using specialized pipelines with base image and mask (base64) inputs
  • Batch Generation - Submit multiple generation jobs in a single request
  • Model Comparison - Generate images from multiple checkpoints side-by-side for comparison
  • Multi-Model Merging - Merge two or more models using configurable methods (add, average, slerp), with weight coefficients and multi-step recipe execution
  • Asynchronous Job Queue - Non-blocking API with worker-based async pipeline execution via asyncio.Queue
  • Model Caching - LRU cache (max 2 pipelines) for expensive diffusion model loading, auto-eviction via access order
  • LoRA Support - Load and fuse PEFT-based LoRA adapters into the base pipeline with configurable strength
  • Custom VAE - Replace the default VAE with custom models loaded from disk
  • Multi-Device Auto-Detection - Automatic fallback: MPS (Apple Silicon) > CUDA (NVIDIA) > CPU with dtype auto-selection (float16 for GPU, float32 for CPU)
  • Configurable Output - Optional disk persistence of generated images and metadata to outputs/images/ and outputs/json/ as PNG and JSON
  • System Endpoints - Health checks, system info (GPU, cache stats, queue stats), and configuration exposure
  • CORS Middleware - Configurable CORS support, enabled by default with wildcard origins

API Endpoints

All endpoints are prefixed with /api/v1.

Generation

Method Route Description
POST /sd15/generate Submit SD 1.5 text-to-image generation job
POST /sd15/inpaint Submit SD 1.5 inpainting job
POST /sd15/generate/batch Submit multiple SD 1.5 generation jobs
POST /sd15/generate/compare Generate from multiple SD 1.5 models for comparison
POST /sdxl/generate Submit SDXL text-to-image generation job
POST /sdxl/inpaint Submit SDXL inpainting job
POST /sdxl/generate/batch Submit multiple SDXL generation jobs
POST /sdxl/generate/compare Generate from multiple SDXL models for comparison

Model Management

Method Route Description
GET /models/{model_type}/{resource_type} List available models (model_type: sd15/sdxl, resource_type: checkpoints/loras/vaes)
GET /models/{model_type}/{resource_type}/{model_name}/metadata Get safetensors metadata for a specific model

Model Merging

Method Route Description
POST /merge/merge Merge two models into one new checkpoint with configurable method (add, average, slerp), alpha weight, and metadata preservation
POST /merge/batch Merge one base model against multiple target models simultaneously, saving each output into a specified subdirectory
POST /merge/recipe Execute a multi-step merge recipe where each step takes the current result and merges it with a new target model using its own method/alpha

Job Management

Method Route Description
GET /jobs List all jobs with current status
GET /jobs/{job_id} Get detailed status and result for a specific job
DELETE /jobs/{job_id} Cancel a pending or running job

System & Info

Method Route Description
GET /system/health Health check endpoint
GET /system/info System info (GPU, data type, cache stats, queue stats)
GET /info/schedulers List available noise schedulers
GET /info/app_settings Return current application configuration

Additional Swagger UI docs at /docs and ReDoc at /redoc (auto-generated by FastAPI).

Class Diagrams

Core Pipeline Architecture

┌─────────────────────────────────┐
│     BasePipelineGenerator       │  (ABC)
│  ─────────────────────────────  │
│ - _model_path: str              │
│ - _pipeline_type: PipelineType  │
│ - _device: str                  │
│ - _dtype: torch.dtype           │
│  ─────────────────────────────  │
│ + loadPipeline()                │
│ + loadInpaintPipeline()         │
│ + pipeToDevice()                │
│ + forward(prompt, params)       │  # text-to-image
│ + forward_inpaint(base, mask…)  │  # inpainting
│ + addLorasToPipeline()          │
│ + loadCustomVAE()               │
│ + getResolutions()              │
└──────────┬──────────────────────┘
           │ inherits
    ┌───────┴───────┐
    │               │
┌────────┐   ┌────────┐
│ SD15   │   │ SDXL   │
│Pipeline│   │Pipeline│
│ Generator│  │ Generator│
└────────┘   └────────┘

Service Layer

┌──────────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  GenerationService   │────>│   ModelCacheManager  │────>│ BasePipelineGen  │
│ ──────────────────── │     │ ───────────────────  │     │ ───────────────  │
│ + generate()         │     │ + get_pipeline()     │     │ - cache: dict    │
│ + inpaint()          │     │ + load_pipeline()    │     │ + forward()      │
│ + batch_generate()   │     │ + clear_cache()      │     │ + addLoras...()  │
│ + compare_models()   │     │ + get_stats()        │     └──────────────────┘
└──────────┬───────────┘     └─────────────────────┘
           │
           │ submits
           v
┌──────────────────────┐
│     JobQueue          │
│ ──────────────────── │
│ + submit_job()       │  # returns job_id
│ + get_job_status()   │
│ + cancel_job()       │
│ + list_jobs()        │
│ + stop_workers()     │
│ - _queue: Queue      │
│ - _jobs: dict        │
│ - _worker()          │  # async background loop
└──────────────────────┘

Request / Response Models

┌─────────────────┐   ┌────────────────┐   ┌──────────────────┐
│ Generation      │   │ InpaintRequest │   │ LoraConfig       │
│ Request         │   │ ────────────── │   │ ──────────────── │
│ ──────────────  │   │ - image_base64 │   │ - name: str      │
│ - prompt: str   │   │ - mask_base64  │   │ - strength: float│
│ - model: str    │   │ - image: str   │   └────────┬─────────┘
│ - params: dict  │   │ - mask: str    │            │
│ - loras: list[] │──>│ - resolution   │            │
│ - vae: str|None │   │ - negative_prompt
│ ──────────────  │   └────────────────┘
│ + to_dict()     │
└────────┬────────┘
         │ validates
         v
┌─────────────────────────────────────────┐
│        JobResponse / JobStatusResponse   │
│  ─────────────────────────────────────  │
│  - job_id: str (UUID4)                  │
│  - status: Job (QUEUED/RUNNING/...)     │
│  - result: JobResult|None               │
│    - image_base64: str                  │
│    - metadata: dict                     │
│    - seed: int                          │
└─────────────────────────────────────────┘

Configuration

┌──────────────────┐     ┌─────────────────┐
│ parameters.yaml  │────>│   AppConfig     │
│ ─────────────── │     │ ─────────────── │
│ checkpoints:     │     │ - _params: dict │
│   sd15: path     │     │ ─────────────── │
│   sdxl: path     │     │ + get_model_... │
│ loras:           │     │ + setup_paths() │
│ vaes:             │     │ + get_output_.. │
│ storage:         │     │ + as_dict()     │
│ api:             │     └─────────────────┘
│   queue:         │          ▲
│   cache:         │          │ lru_cache
│   cors:          │     ┌─────────────────┐
└──────────────────┘     │ get_app_config()│
                         └─────────────────┘

Overall Call Flow

Image Generation (Primary Workflow)

  1. Client sends POST /api/v1/{sd15,sdxl}/generate with prompt, model, and generation parameters
  2. Router resolves model path from AppConfig, converts LoRA configs to dicts, validates request
  3. JobQueue generates a UUID4 job_id, creates a Job object, enqueues it, and returns immediately with JobResponse(status="queued")
  4. Background workers (_worker loop) dequeue jobs and set status to RUNNING
  5. GenerationService calls ModelCacheManager.get_pipeline() to get or load the diffusion pipeline
  6. Pipeline loads in a thread pool executor (blocking diffusers call)
  7. LoRA adapters are fused via peft and custom VAE is swapped if provided
  8. Pipeline's forward() (text-to-image) or forward_inpaint() is invoked with generation params
  9. Result is an image (base64-encoded PNG) plus metadata dict (GenerationMetadata container)
  10. If persist_to_backend is enabled, image is saved to outputs/images/{uuid}.png and metadata to outputs/json/{uuid}.json
  11. Job is marked COMPLETED and result stored for client polling

Model Merging Call Flow

  1. Client sends POST /api/v1/merge/merge (or /merge/batch, /merge/recipe) with model specs, merge method, and weight parameters
  2. Router validates model paths from AppConfig
  3. JobQueue generates a job_id and returns immediately with JobResponse(status="queued")
  4. Background workers dequeue jobs and invoke MergeService.merge_models()
  5. MergeService loads model weights using diffusers and safetensors, applies the specified merge algorithm (add, average, or slerp), and saves the merged checkpoint to the configured output directory
  6. Recipe merges loop through each step sequentially, feeding the result of the previous step as the base for the next
  7. Job is marked COMPLETED with the path to the newly created merged model

Job Status Polling

Client: GET /api/v1/jobs/{job_id}
    → JobQueue.get_job_status()
    → Returns JobStatusResponse { job_id, job_type, status, result, error, timestamps }
    → If status=CALLED: extract result.image_base64 → decode → save PNG
    → If status=FAILED:  display result.error

Model Cache Flow

ModelCacheManager.get_pipeline(model_type, model_path, pipeline_type)
  1. Build cache key: "{model_type}:{model_name}:{pipeline_type}"
  2. Cache HIT → update access order → return cached pipeline
  3. Cache MISS → _load_pipeline() in thread pool
     → Instantiate pipeline generator → loadSDPipeline() → pipeToConfiguredDevice()
  4. If at capacity (>= max_models) → evict least-recently-used
  5. Insert new pipeline into cache

Startup / Shutdown Flow

Uvicorn starts → api.app:app (pre-created via create_app())

Lifespan startup:
  1. Load AppConfig from parameters.yaml
  2. Create FastAPI app with CORS middleware
  3. Mount routers (sd15, sdxl, models, merge, jobs, system, info)
  4. JobQueue singleton created → start_workers() spawns N async worker tasks
  5. Log "SD API Backend started"

Lifespan shutdown:
  1. JobQueue.stop_workers() → sets _running=False, drains queue, cancels workers
  2. Log "SD API Backend shutdown complete"

Usage Examples

Python (Async with httpx)

import asyncio
import httpx
import json

BASE_URL = "http://localhost:8000/api/v1"

async def generate_image():
    async with httpx.AsyncClient() as client:
        # Submit generation job
        response = await client.post(
            f"{BASE_URL}/sd15/generate",
            json={
                "prompt": "a beautiful sunset over mountains",
                "model": "v2-1024px-pruned.safetensors",
                "params": {
                    "width": 512,
                    "height": 512,
                    "num_inference_steps": 30,
                    "guidance_scale": 7.5,
                    "seed": 42
                }
            }
        )
        job = response.json()
        job_id = job["job_id"]
        print(f"Job submitted: {job_id}")

        # Poll for completion
        while True:
            status = await client.get(f"{BASE_URL}/jobs/{job_id}")
            data = status.json()
            if data["status"] == "completed":
                result = data["result"]
                # Decode and save image
                import base64
                with open(f"output_{job_id}.png", "wb") as f:
                    f.write(base64.b64decode(result["image_base64"]))
                print(f"Image saved. Seed: {result['seed']}")
                break
            elif data["status"] == "failed":
                print(f"Job failed: {data['error']}")
                break
            await asyncio.sleep(1)

asyncio.run(generate_image())

cURL

# Submit a generation job
curl -X POST http://localhost:8000/api/v1/sd15/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "a cyberpunk city at night",
    "model": "v2-1024px-pruned.safetensors",
    "params": {
      "width": 512,
      "height": 512,
      "num_inference_steps": 30,
      "guidance_scale": 7.5,
      "seed": 42
    }
  }'

# Check job status
curl http://localhost:8000/api/v1/jobs/{job_id}

# List all jobs
curl http://localhost:8000/api/v1/jobs

# Cancel a job
curl -X DELETE http://localhost:8000/api/v1/jobs/{job_id}

# List available checkpoints
curl http://localhost:8000/api/v1/models/sd15/checkpoints

# Merge two models
curl -X POST http://localhost:8000/api/v1/merge/merge \
  -H "Content-Type: application/json" \
  -d '{
    "model_a": "model_a.safetensors",
    "model_b": "model_b.safetensors",
    "alpha": 0.5,
    "merge_type": "average"
  }'

# Multi-step recipe merge
curl -X POST http://localhost:8000/api/v1/merge/recipe \
  -H "Content-Type: application/json" \
  -d '{
    "base_model": "base.safetensors",
    "steps": [
      {"target": "style1.safetensors", "alpha": 0.4, "merge_type": "add"},
      {"target": "style2.safetensors", "alpha": 0.3, "merge_type": "average"}
    ],
    "output_path": "merged/"
  }'

# Health check
curl http://localhost:8000/api/v1/system/health

With LoRA Adapters

response = await client.post(
    f"{BASE_URL}/sdxl/generate",
    json={
        "prompt": "a portrait in anime style",
        "model": "sd_xl_base_1.0.safetensors",
        "params": {
            "width": 1024,
            "height": 1024,
            "num_inference_steps": 25,
        },
        "loras": [
            {"name": "anime_style.safetensors", "strength": 0.8}
        ]
    }
)

Configuration

Copy parameters.yaml.example to parameters.yaml and adjust paths:

checkpoints:
  sd15:
    path: 'models/checkpoints/SD15'
  sdxl:
    path: 'models/checkpoints/SDXL'
loras:
  sd15:
    path: 'models/loras/SD15'
  sdxl:
    path: 'models/loras/SDXL'
vaes:
  sd15:
    path: 'models/vae/SD15'
  sdxl:
    path: 'models/vae/SDXL'
storage:
  persist_to_backend: true
  output_images: 'outputs/images'
  output_json: 'outputs/json'
api:
  host: "0.0.0.0"
  port: 8000
  workers: 1
  job_queue:
    num_workers: 1
    result_ttl: 3600
  model_cache:
    max_models: 2
  cors:
    enabled: true
    allow_origins: ["*"]

Running

# Install dependencies
uv sync

# Start the server
uv run uvicorn api.app:app --host 0.0.0.0 --port 8000

About

Stable Diffusion API Backend Server

Resources

Readme
GPL-3.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages