Tiling and Padding functions for VapourSynth

A collection of spatial and temporal tiling and padding utilities for VapourSynth. The original idea was just a tiling function to make AI filters less VRAM-hungry and to provide additional options that built-in solutions might not. Over time, more related functions were added.

The functions often come in pairs, with one doing a thing and the other inversing it. For example:

import vs_tiletools
clip = vs_tiletools.tile(clip, width=256, height=256) # splits frames into 256x256 tiles
clip = core.someheavyfilter.AIUpscale(clip)           # placeholder resource intensive filter
clip = vs_tiletools.untile(clip)                      # reassembles the tiles into full frames

Table of Contents

• Requirements
• Setup
• Usage Examples
• Spatial Functions
  _Tiling
    ⚬ Tile - Splits each frame into tiles of fixed dimensions
    ⚬ Untile - Auto reassembles tiles from tile(), even if resized
  _{Padding/Cropping}
    ⚬ Pad - Pads a clip with various padding modes
    ⚬ Mod - Pads or crops a clip so width and height are multiples of the given modulus
    ⚬ Crop - Auto crops padded clip from pad() or mod(), even if resized
    ⚬ Croprandom - Crops to given dimensions, but randomly repositions the window each frame
  _{Filling/Inpainting}
    ⚬ Fill - Fills the borders of a clip with various filling modes
    ⚬ Autofill - Auto detects borders and fills them with various fill modes
    ⚬ Inpaint - Inpaints areas based on a mask with various inpainting modes
• Temporal Functions
  _{Duplicate Detection}
    ⚬ Markdups - Marks identical frames as duplicates, which can later be skipped using skipdups()
    ⚬ Skipdups - Skips processing of duplicate frames marked by markdups()
  _Overlapping
    ⚬ Insert Overlaps - Inserts temporal overlaps at fixed intervals
    ⚬ Trim Overlaps - Auto trims or crossfades overlaps added by insert_overlaps()
  _{Extending/Trimming}
    ⚬ Extend - Extends a clip with various temporal padding modes
    ⚬ Trim - Auto trims extended clip from extend()
  _Other
    ⚬ Crossfade - Crossfades between two clips
• Mode Explanations

Requirements

• fillborders (pad mode fixborders needs v3 or newer)
• cv_inpaint
• autocrop (optional, only for autofill)
• akarin (optional, only for markdups/skipdups)
• libvship (optional, only for markdups/skipdups, requires v4.0.0 or newer)

Setup

Put the vs_tiletools.py file into your vapoursynth scripts folder.
Or install via pip: pip install -U git+https://github.com/pifroggi/vs_tiletools.git

Usage Examples

Examples of how the paired functions can be used together.

• Reduce VRAM usage on heavy AI models via tiling.

  import vs_tiletools
  clip = vs_tiletools.tile(clip, width=256, height=256, overlap=16) # splits frames into 256x256 tiles with an overlap of 16
  clip = core.trt.Model(clip, engine_path="2x_heavy_model.engine")  # heavy AI upscale model
  clip = vs_tiletools.untile(clip, fade=True)                       # reassembles the tiles and uses the overlap to feather

• Fix issues around borders with some filters via padding.

  import vs_tiletools
  clip = vs_tiletools.pad(clip, left=8, right=8, top=8, bottom=8) # pad 8 pixels on all sides
  clip = core.trt.Model(clip, engine_path="model.engine")         # AI model with issues near borders
  clip = vs_tiletools.crop(clip)                                  # automatically crop the padding

• Fix filters that require the input to be divisible by a factor via padding.

  import vs_tiletools
  clip = vs_tiletools.mod(clip, modulus=16)                      # pad to make width and height divisible by 16
  clip = core.trt.Model(clip, engine_path="2x_DAT_model.engine") # DAT and HAT based AI models have this constraint
  clip = vs_tiletools.crop(clip)                                 # automatically crop the padding

• Skip heavy filters on duplicate frames. Most useful for anime.

  import vs_tiletools
  clip = vs_tiletools.markdups(clip, thresh=0.3)                   # marks duplicate frames with a low threshhold
  clip = core.trt.Model(clip, engine_path="2x_heavy_model.engine") # heavy AI upscale model
  clip = vs_tiletools.skipdups(clip)                               # skips duplicates and replaces them with a previous frame

• Fix jumps/hitches on chunk/temporal window based filters via crossfading.

  import vs_tiletools
  clip = vs_tiletools.insert_overlaps(clip, length=10, overlap=4) # creates a temporal overlap of 4 frames
  clip = vs_undistort.tensorrt(clip, temp_window=10)              # filter has 10 input frames and 10 output frames
  clip = vs_tiletools.trim_overlaps(clip, fade=True)              # uses the overlap to fade between chunks/windows

• Fix filters that behave badly at the start/end of clips.

  import vs_tiletools
  clip = vs_tiletools.extend(clip, start=10, end=10) # extend clip by 10 frames at the start and end
  clip = some.temporal_filter(clip)                  # temporal filters can have this issue
  clip = vs_tiletools.trim(clip)                     # automatically trims the added frames

Spatial Functions

• Tile

  Splits each frame into tiles of fixed dimensions to reduce resource requirements. Outputs a clip with all tiles in order. All filters applied to the tiled clip should be spatial only.

  import vs_tiletools
  clip = vs_tiletools.tile(clip, width=256, height=256, overlap=16, padding="mirror")

  clip
  Clip to tile. Any format.

  width, height
  Tile size of a single tile in pixel.

  overlap
  Overlap from one tile to the next. When overlap is increased the tile size is not altered, so the amount of tiles per frame increases. Can be a single value or a pair for horizontal and vertical [16, 32].

  padding
  How to handle tiles that are smaller than tile size. These can be padded with modes mirror, wrap, repeat, fillmargins, telea, ns, fsr, black, a custom color in 8-bit scale [128, 128, 128], or just discarded with discard. For a full explanation of each padding mode, click here.

---

• Untile

  Automatically reassembles a clip tiled with tile(), even if tiles were since resized. Example

  import vs_tiletools
  clip = vs_tiletools.untile(clip, fade=False) # automatic
  clip = vs_tiletools.untile(clip, fade=False, full_width=None, full_height=None, overlap=None) # manual

  clip
  Tiled clip. Any format.

  fade
  If fade is True, the overlap will be used to feather/blend between the tiles to remove visible seams.
  If fade is False, the overlap will be cropped.

  full_width, full_height, overlap (optional)
  You can also enter untile parameters manually. Needed is the full assembled frame dimensions and the overlap between tiles. In manual mode you have to account for resized or discarded tiles yourself.
  Tip: If tiles were discarded, the full_width/full_height are now smaller and a multiple of the original tile size.
  Tip: If tiles were resized 2x, simply double all values.

---

• Pad

  Pads a clip with various padding modes.

  import vs_tiletools
  clip = vs_tiletools.pad(clip, left=0, right=0, top=0, bottom=0, mode="mirror")

  clip
  Clip to be padded. Any format.

  left, right, top, bottom
  Padding amount in pixel.

  mode
  Padding mode can be mirror, wrap, repeat, fillmargins, telea, ns, fsr, black, or a custom color in 8-bit scale [128, 128, 128]. For a full explanation of each mode, click here.

---

• Mod

  Pads or crops a clip so width and height are multiples of the given modulus.

  import vs_tiletools
  clip = vs_tiletools.mod(clip, modulus=64, mode="mirror")

  clip
  Source clip. Any format.

  modulus
  Dimensions will be a multiple of this value. Can be a single value, or a pair for width and height [64, 32].

  mode
  Mode to reach the next upper multiple via padding can be mirror, wrap, repeat, fillmargins, telea, ns, fsr, black, a custom color in 8-bit scale [128, 128, 128], or discard to crop to the next lower multiple. For a full explanation of each padding mode, click here.

---

• Crop

  Automatically crops padding added by pad() or mod(), even if the clip was since resized. Example1 Example2

  import vs_tiletools
  clip = vs_tiletools.crop(clip) # automatic
  clip = vs_tiletools.crop(clip, left=0, right=0, top=0, bottom=0) # manual

  clip
  Padded clip. Any format.

  left, right, top, bottom (optional)
  Optionally you can also enter crop values manually.

---

• Croprandom

  Crops to the given dimensions, but randomly repositions the crop window each frame.

  import vs_tiletools
  clip = vs_tiletools.croprandom(clip, width=256, height=256, seed=0)

  clip
  Clip to be cropped. Any format.

  width, height
  Cropped window dimensions in pixels.

  seed
  Seed used for deterministic crop randomization.

---

• Fill

  Fills the borders of a clip with various filling modes. Basically padding, but inwards.

  import vs_tiletools
  clip = vs_tiletools.fill(clip, left=0, right=0, top=0, bottom=0, mode="mirror")

  clip
  Clip to be filled. Any format.

  left, right, top, bottom
  Fill amount in pixel.

  mode
  Filling mode can be mirror, wrap, repeat, fillmargins, telea, ns, fsr, black, or a custom color in 8-bit scale [128, 128, 128]. For a full explanation of each mode, click here.

---

• Autofill

  Detects uniform colored borders (like letterboxes/pillarboxes) and fills them with various filling modes.

  import vs_tiletools
  clip = vs_tiletools.autofill(clip, left=0, right=0, top=0, bottom=0, offset=0, color=[16,128,128], tol=16, fill="mirror")

  clip
  Source clip. Only YUV formats are supported.

  left, right, top, bottom
  Maximum border fill amount in pixels.

  offset
  Offsets the detected fill area by an extra amount in pixels. Useful if the borders are slightly blurry.
  Does not offset sides that have detected 0 pixels.

  color
  Source clip border color in 8-bit scale [16, 128, 128].

  tol
  Tolerance to account for fluctuations in border color. Can be a single value or a list [16, 16, 16].

  fill
  Filling mode can be mirror, repeat, fillmargins, telea, ns, fsr, black, or a custom color in 8-bit scale [128, 128, 128]. For a full explanation of each mode, click here.

---

• Inpaint

  Inpaints areas in a clip based on a mask with various inpainting modes.

  import vs_tiletools
  clip = vs_tiletools.inpaint(clip, mask, mode="telea")

  clip
  Clip to be inpainted. Any format.

  mask
  Black and white mask clip where white means inpainting. Can be a single frame long, or longer and different each frame. If too short, the last frame will be looped. Can be any format and doesn't have to match the base clip.

  mode
  Inpainting mode can be telea, ns, fsr or shiftmap. For a full explanation of each mode, click here.

---

Temporal Functions

• Markdups

  Marks up to 5 consecutive frames as duplicates if they are near identical, which can later be skipped using skipdups(). Example

  import vs_tiletools
  clip = vs_tiletools.markdups(clip, thresh=0.3)

  clip
  Clip were duplicates should be marked. Any format.

  thresh
  Similarity threshold. If the difference between two consecutive frames is lower than this value, the frame is marked as a duplicate. If the value is 0, only 100% identical frames will be marked as duplicate. Keep it a little above 0 due to noise and compression. The default worked nicely for me on anime.

---

• Skipdups

  Skips processing of up to 5 consecutive duplicate frames marked by markdups(). That means the marked frames will copy one of the previous 5 frames instead of submitting the current frame for processing. This speeds up heavy filters sandwiched inbetween markdups() and skipdups(). Example

  Keep in mind that if you use a heavy spatial filter, followed by a temporal filter, both inside of the sandwich, the speedup will be negated, because the temporal filter will request the marked frames anyway. For this reason, it is recommended to use temporal filters outside the sandwich.

  import vs_tiletools
  clip = vs_tiletools.skipdups(clip, debug=False) # automatic
  clip = vs_tiletools.skipdups(clip, prop_src=None, debug=False) # manual

  clip
  Clip with marked duplicates. Any format.

  prop_src (optional)
  Frame properties source clip. This should be detected automatically. But if the frame props of the first clip got lost, you can set it here manually. It should be the clip directly returned by markdups().

  debug
  Overlays the frame number of the selected frame and the difference value to the previous frame onto the output. This is useful to finetune the sensitivity threshold in markdups().

---

• Insert Overlaps

  Inserts temporal overlaps at fixed intervals into the clip. That means a chunk with length=20 and overlap=5 will produce a clip with this frame pattern: 0–19, 15–34, 30–49, and so on. In combination with the trim_overlaps function, the overlap can then be used to crossfade between chunks/windows and eliminate sudden jumps/hitches that can occur on chunk/window based functions like vs_undistort. Example

  import vs_tiletools
  clip = vs_tiletools.insert_overlaps(clip, length=20, overlap=5, padding="mirror")

  clip
  Clip that should get overlaps inserted. Any format.

  length
  Chunk/temporal window length.

  overlap
  Overlap from one window to the next. When overlap is increased, the temporal window length is not altered, so the total amount of windows per clip increases.

  padding
  How to handle the last window of the clip if it is smaller than length. It can be padded with modes mirror, loop, repeat, black, a custom color in 8-bit scale [128, 128, 128], discarded with discard, or left as is with None. For a full explanation of each padding mode, click here.

---

• Trim Overlaps

  Automatically removes the overlaps inserted by insert_overlaps() and optionally uses them to crossfade between chunks/windows. Example

  import vs_tiletools
  clip = vs_tiletools.trim_overlaps(clip, fade=False) # automatic
  clip = vs_tiletools.trim_overlaps(clip, fade=False, full_length=None, window_length=None, overlap=None) # manual

  clip
  Clip with inserted overlaps. Any format.

  fade
  If fade is True, the overlap will be used to crossfade between the chunks/windows.
  If fade is False, the overlap will be trimmed.

  full_length, window_length, overlap (optional)
  You can also enter trim_overlaps parameters manually. Needed is the full clip length, window length and the overlap between windows. In manual mode you have to account for a discarded window yourself.
  Tip: If the last chunk/window was discarded, the full_length is now smaller and a multiple of window_length.
  Tip: If the clip was interpolated to 2x after inserting overlaps, simply double all values.

---

• Extend

  Extends (temporally pads) a clip using various padding modes.

  import vs_tiletools
  clip = vs_tiletools.extend(clip, start=0, end=0, length=None, mode="mirror")

  clip
  Clip to extend. Any format.

  start, end
  Number of frames to add at the start and/or end. Mutually exclusive with length.

  length
  Extends clip to exactly this many frames. Mutually exclusive with start/end.

  mode
  Padding mode can be mirror, loop, repeat, black, or a custom color in 8-bit scale [128, 128, 128]. For a full explanation of each mode, click here.

---

• Trim

  Automatically trims temporal padding added by extend(). Example

  import vs_tiletools
  clip = vs_tiletools.trim(clip) # automatic
  clip = vs_tiletools.trim(clip, start=0, end=0, length=None) # manual

  clip
  Temporally padded clip. Any format.

  start, end (optional)
  Optional manual number of frames to remove from start and/or end. End is mutually exclusive with length.

  length (optional)
  Optional manual trim to exactly this many frames, starting from start. Mutually exclusive with end.

---

• Crossfade

  Crossfades between two clips.

  import vs_tiletools
  clip = vs_tiletools.crossfade(clipa, clipb, length=10)

  clipa, clipb
  Input clips to crossfade. Any format, as long as they match.

  length
  Length of the crossfade. For example length=10 will fade the last 10 frames of clipa into the first 10 frames of clipb.

---

Mode Explanations

Full explanations for all padding/filling/inpainting modes.

• Spatial

  • mirror Reflects the image into the padded/filled region.
  • wrap Wraps the image around to create a periodic tiling.
  • repeat Repeats the outermost pixel row/column.
  • fillmargins Similar to repeat, but the top and bottom pad/fill gets more blurry the further away it is.
  • fixborders A direction aware fillmargins that also works on all four sides, not just top and bottom.
  • telea Gets more blurry the further away it is.
  • ns Navier-Stokes algorithm. Similar to telea, but less blurry.
  • fsr Frequency Selective Reconstruction algorithm. Better at keeping patterns/textures, but is slow.
  • shiftmap Shifts part of the existing image to fill the holes. Only for inpainting.
  • black Solid black.
  • [128, 128, 128] Solid custom color. 8-bit values per plane in the clip’s color family.

• Temporal

  • mirror Reverses the clip at the start/end.
  • loop Loops the clip to start over.
  • repeat Repeats the first/last frame.
  • black Appends solid black frames.
  • [128, 128, 128] Appends frames in a solid custom color. 8-bit values per plane in the clip’s color family.

Note

The padded/filled/inpainted regions may be generated at a lower bit depth due to plugin limitations (16-bit for fillborders, 8-bit for cv_inpaint), then upsampled and merged onto the original high depth frames. This should usually not be an issue.

Modes fillmargins and fixborders, which are partially broken on some formats when using the fillborders plugin directly, are also fixed here.