Add video input support for VLM inference - #2359

Open

Reithan wants to merge 30 commits into

LostRuins:concedo_experimentalfrom

Reithan:add-video-input-for-vlm-inference

Commits on Jul 23, 2026

refactor media_object is_audio flag into media type enum
Reithan
committed
34be26e
add video fields to generation_inputs and load_model_inputs contract
Reithan
committed
04df7a2
marshal videos genparam into generation_inputs struct
Reithan
committed
60adf8b
populate video media_objects and skip video eval safely in gpttype_adapter
Reithan
committed
3001653
enable MTMD_VIDEO and probe video capability at load
Reithan
committed
72b1b34
accept video content parts in chat completions paths
Reithan
committed
c7deb62
add video input CLI flags and budget warning
Reithan
committed
4a5963d
reword --videomaxframes help as a sampling budget, not a truncation cap
Reithan
committed
da1673f
implement MTMD_VIDEO frame-streaming eval path in PrepareMediaEmbds
Reithan
committed
e987786
downsample video fps to fit frame budget instead of truncating
Reithan
committed
6b15f41
hash media items for cache signature instead of concatenating base64
Reithan
committed
fb9d839
cap attached videos before placeholder numbering and warn on drop
Reithan
committed
757001a
enforce videomintokens by upscaling undersized video frames
Reithan
committed
cb00e58
document why fnv1a64_hex is implemented locally
Reithan
committed
6d9520e
keep most recent videos when capping, drop oldest
Reithan
committed
afc2b34
set mtmd_input_text text_len on all media tokenize calls to fix lost marker
Reithan
committed
12c84fb
note maxrequestsize limit on video attachments at model load
Reithan
committed
a947ccf
decode video via temp file to avoid ffmpeg feeder-thread teardown deadlock
Reithan
committed
296b16d
aim downsampled fps so the last frame lands inside the frame budget
Reithan
committed
d2e19a4
probe container duration to downsample videos lacking stream metadata
Reithan
committed
e2cac91
respect quiet mode in video pipeline logging
Reithan
committed
7c9e5f7
add TODO outlining future video audio-track embedding
Reithan
committed
d31c7ff
reformat video timestamps to MM:SS and append duration note for model readability
Reithan
committed
0435a5d
filter llama core log spam during media generations per quiet/debug tiers
Reithan
committed
141e72f

Navigation Menu

Add video input support for VLM inference - #2359

Add video input support for VLM inference#2359
Reithan wants to merge 30 commits into
LostRuins:concedo_experimentalfrom
Reithan:add-video-input-for-vlm-inference

Commits on Jul 23, 2026

refactor media_object is_audio flag into media type enum

add video fields to generation_inputs and load_model_inputs contract

marshal videos genparam into generation_inputs struct

populate video media_objects and skip video eval safely in gpttype_adapter

enable MTMD_VIDEO and probe video capability at load

accept video content parts in chat completions paths

add video input CLI flags and budget warning

reword --videomaxframes help as a sampling budget, not a truncation cap

implement MTMD_VIDEO frame-streaming eval path in PrepareMediaEmbds

downsample video fps to fit frame budget instead of truncating

hash media items for cache signature instead of concatenating base64

cap attached videos before placeholder numbering and warn on drop

enforce videomintokens by upscaling undersized video frames

document why fnv1a64_hex is implemented locally

keep most recent videos when capping, drop oldest

set mtmd_input_text text_len on all media tokenize calls to fix lost marker

note maxrequestsize limit on video attachments at model load

decode video via temp file to avoid ffmpeg feeder-thread teardown deadlock

aim downsampled fps so the last frame lands inside the frame budget

probe container duration to downsample videos lacking stream metadata

respect quiet mode in video pipeline logging

add TODO outlining future video audio-track embedding

reformat video timestamps to MM:SS and append duration note for model readability

filter llama core log spam during media generations per quiet/debug tiers

Commits on Jul 24, 2026

budget media context by real KV cell counts instead of position advance

cap video frame resolution and frame count by context KV cell budget

calibrate video frame resolution from measured first-frame cell cost

iterate frame resolution calibration via interpolation search on measured cells

make frame resolution calibration bidirectional with bracketed probes

Merge branch 'concedo_experimental' into add-video-input-for-vlm-inference