Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion packages/phase-vocoder-worklet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ Top-level exports:

Extends `AudioWorkletNode`. Same API as `SoundTouchNode` with additional `fftSize` / `overlapFactor` options.

Output starts one render block (128 samples) after the pipeline warms up; include it when compensating latency.

#### Static methods

| Method | Description |
Expand Down Expand Up @@ -168,7 +170,7 @@ node.addEventListener('metrics', (e) => {
| Quality at extreme ratios | Artifacts above 2× | Smooth at all ratios |
| Transient preservation | Better (time-domain) | Worse (frequency smearing) |
| Computation | Lower | Higher (FFT per hop) |
| Startup latency | Lower | `fftSize` samples |
| Startup latency | Lower | `fftSize` + 128 samples |
| Artifacts | Clicks / repeats | "Phasiness" / smearing |

## Architecture
Expand Down
3 changes: 2 additions & 1 deletion packages/phase-vocoder-worklet/src/PhaseVocoderNode.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ export interface PhaseVocoderNodeConstructorOptions extends PhaseVocoderNodeOpti
* `fftSize` and `overlapFactor` constructor options for tuning the FFT stage.
*
* The phase vocoder produces smoother results than WSOLA at extreme ratios (> 2×)
* at the cost of higher per-frame computation and inherent `fftSize`-sample latency.
* at the cost of higher per-frame computation. Latency is `fftSize` samples
* plus one render block (128 samples) of output priming.
*
* @example
* ```ts
Expand Down
64 changes: 57 additions & 7 deletions packages/phase-vocoder-worklet/src/phase-vocoder-processor.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,18 @@ describe('phase-vocoder-processor', () => {
playbackRate: new Float32Array([1]),
};

// two blocks open the startup gate and do not count as underruns
outputFrameCount = 128;
for (let i = 0; i < 2; i++) {
instance.process(
[[new Float32Array(128)]],
[[new Float32Array(128), new Float32Array(128)]],
params,
);
}

outputFrameCount = 64;
for (let i = 0; i < 100; i++) {
for (let i = 0; i < 98; i++) {
instance.process(
[[new Float32Array(128)]],
[[new Float32Array(128), new Float32Array(128)]],
Expand All @@ -390,7 +400,8 @@ describe('phase-vocoder-processor', () => {
expect(instance.port.postMessage).toHaveBeenCalledWith(
expect.objectContaining({
type: 'metrics',
underrunCount: 100,
blockCount: 100,
underrunCount: 98,
}),
);
});
Expand All @@ -416,15 +427,23 @@ describe('phase-vocoder-processor', () => {
const inputRight = new Float32Array([2, 4]);
const outputLeft = new Float32Array(2);
const outputRight = new Float32Array(2);
const params = {
pitch: new Float32Array([1]),
pitchSemitones: new Float32Array([0]),
playbackRate: new Float32Array([1]),
};

// first block is held by the startup gate
instance.process(
[[inputLeft, inputRight]],
[[new Float32Array(2), new Float32Array(2)]],
params,
);

const result = instance.process(
[[inputLeft, inputRight]],
[[outputLeft, outputRight]],
{
pitch: new Float32Array([1]),
pitchSemitones: new Float32Array([0]),
playbackRate: new Float32Array([1]),
},
params,
);

expect(result).toBe(true);
Expand All @@ -437,4 +456,35 @@ describe('phase-vocoder-processor', () => {
expect(outputRight[1]).toBeCloseTo(0.4, 6);
});
});

describe('startup gate', () => {
it('holds output for one render block after the buffer first fills', async () => {
await import('./phase-vocoder-processor.js');
outputFrameCount = 512;

const instance = new registeredCtor!({
processorOptions: { sampleBufferType: 'circular' },
});

const params = {
pitch: new Float32Array([1]),
pitchSemitones: new Float32Array([0]),
playbackRate: new Float32Array([1]),
};

instance.process(
[[new Float32Array(128)]],
[[new Float32Array(128), new Float32Array(128)]],
params,
);
expect(extract).not.toHaveBeenCalled();

instance.process(
[[new Float32Array(128)]],
[[new Float32Array(128), new Float32Array(128)]],
params,
);
expect(extract).toHaveBeenCalledWith(expect.any(Float32Array), 0, 128);
});
});
});
8 changes: 8 additions & 0 deletions packages/phase-vocoder-worklet/src/phase-vocoder-processor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,14 @@ class PhaseVocoderProcessor extends SoundTouchProcessorBase {
});
}

/**
* The vocoder emits output in analysis-hop-sized bursts, so one held block
* gives the output buffer a permanent 128-frame underrun cushion.
*/
protected override get startupHoldBlocks(): number {
return 1;
}

protected onProcessComplete(result: ProcessCoreResult): void {
if (this._blockCount % 100 === 0) {
this.port.postMessage({
Expand Down
1 change: 1 addition & 0 deletions packages/worklet-base/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ registerProcessor('my-processor', MyProcessor);
|---|---|
| `beforePipeProcess(left, right, frameCount, params)` | Pre-pipe analysis (e.g. LPC analysis for formant correction). Default is a no-op. |
| `extractSamples(leftOutput, rightOutput, frameCount)` | Full extraction/write-back override (e.g. formant synthesis). Default writes both channels and returns RMS/peak metrics. |
| `startupHoldBlocks` (getter) | Override to return `N` when the stretch stage produces output in bursts: the processor emits `N` extra render blocks of leading silence once the output buffer first fills a block, adding `N × 128` frames of latency and the same amount of underrun cushion. Default `0`. |

### Runtime messages

Expand Down
85 changes: 85 additions & 0 deletions packages/worklet-base/src/SoundTouchProcessorBase.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,22 @@ function makeConcreteClass() {
return { TestProcessor, onProcessComplete };
}

function makeGatedClass(hold = 1) {
const onProcessComplete = vi.fn();

class GatedProcessor extends (ProcessorBase as unknown as typeof SoundTouchProcessorBase) {
protected override get startupHoldBlocks(): number {
return hold;
}

protected onProcessComplete(result: ProcessCoreResult): void {
onProcessComplete(result);
}
}

return { GatedProcessor, onProcessComplete };
}

function makeInputs(frameCount = 128, channels = 2): Float32Array[][] {
return [Array.from({ length: channels }, () => new Float32Array(frameCount))];
}
Expand Down Expand Up @@ -165,6 +181,75 @@ describe('SoundTouchProcessorBase', () => {
});
});

describe('startup gate', () => {
it('extracts from the first block when startupHoldBlocks is 0', () => {
const { TestProcessor } = makeConcreteClass();
const proc = new TestProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 64;
const result = proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(result!.toExtract).toBe(64);
expect(extract).toHaveBeenCalledWith(proc['_outputSamples'], 0, 64);
});

it('stays silent until the output buffer first fills a block', () => {
const { GatedProcessor } = makeGatedClass();
const proc = new GatedProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 64;
const result = proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(result!.toExtract).toBe(0);
expect(extract).not.toHaveBeenCalled();
expect(proc['_underrunCount']).toBe(0);
});

it('holds one more block after the first full block, then extracts', () => {
const { GatedProcessor } = makeGatedClass();
const proc = new GatedProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 512;
const held = proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(held!.toExtract).toBe(0);
expect(extract).not.toHaveBeenCalled();

const open = proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(open!.toExtract).toBe(128);
expect(extract).toHaveBeenCalledWith(proc['_outputSamples'], 0, 128);
});

it('holds startupHoldBlocks blocks when greater than one', () => {
const { GatedProcessor } = makeGatedClass(2);
const proc = new GatedProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 512;
expect(proc['processCore'](makeInputs(), makeOutputs(), makeParams())!.toExtract).toBe(0);
expect(proc['processCore'](makeInputs(), makeOutputs(), makeParams())!.toExtract).toBe(0);
expect(proc['processCore'](makeInputs(), makeOutputs(), makeParams())!.toExtract).toBe(128);
});

it('counts underruns only after the gate opens', () => {
const { GatedProcessor } = makeGatedClass();
const proc = new GatedProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 64;
proc['processCore'](makeInputs(), makeOutputs(), makeParams()); // waiting
expect(proc['_underrunCount']).toBe(0);

outputFrameCount = 128;
proc['processCore'](makeInputs(), makeOutputs(), makeParams()); // held
proc['processCore'](makeInputs(), makeOutputs(), makeParams()); // open
expect(proc['_underrunCount']).toBe(0);

outputFrameCount = 64;
proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(proc['_underrunCount']).toBe(1);
});

it('keeps counting blocks while gated', () => {
const { GatedProcessor } = makeGatedClass();
const proc = new GatedProcessor('[Test]', { sampleRate: 44100 });
outputFrameCount = 0;
proc['processCore'](makeInputs(), makeOutputs(), makeParams());
proc['processCore'](makeInputs(), makeOutputs(), makeParams());
expect(proc['_blockCount']).toBe(2);
});
});

describe('port message handling', () => {
it('queues interpolation strategy change', () => {
const { TestProcessor } = makeConcreteClass();
Expand Down
52 changes: 50 additions & 2 deletions packages/worklet-base/src/SoundTouchProcessorBase.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,36 @@ export abstract class SoundTouchProcessorBase extends AudioWorkletProcessor {
null;
private _pendingStretchParameters: StretchParameters | null = null;

/**
* Render blocks left in the startup gate: `-1` = waiting for the output
* buffer's first full block, `> 0` = holding, `0` = open.
*/
private _startupHoldRemaining = -1;

/** Label used in console messages (e.g. `'[SoundTouchProcessor]'`). */
protected readonly processorLabel: string;

/**
* Render blocks the processor stays silent after the output buffer first
* holds a full block, before extraction begins.
*
* @remarks
* A bursty stretch stage refills the output buffer one hop at a time while
* `process` drains one block per quantum, so the buffer's trough rides at
* exactly one render block — any fractional-carry shortfall then comes up
* short and is zero-filled, which is an audible click. Each held block is
* emitted before any real output (so the added silence is inaudible) and
* leaves one block's worth of unconsumed frames as a permanent cushion
* under that trough, at the cost of one render block (128 frames) of
* output latency.
*
* Override in subclasses whose stretch stage produces in bursts. The
* default `0` disables the gate and keeps legacy timing.
*/
protected get startupHoldBlocks(): number {
return 0;
}

/**
* Validates and resolves an interpolation strategy id, falling back to
* `'lanczos'` if the id is unrecognised.
Expand Down Expand Up @@ -291,10 +318,31 @@ export abstract class SoundTouchProcessorBase extends AudioWorkletProcessor {

const outputBuffer = this._pipe.outputBuffer;
const available = outputBuffer.frameCount;
const toExtract = Math.min(available, frameCount);

// Startup gate: while active, the block stays silent and nothing is
// consumed — see startupHoldBlocks.
let gated = false;
if (this._startupHoldRemaining !== 0) {
const hold = this.startupHoldBlocks;
if (hold <= 0) {
this._startupHoldRemaining = 0;
} else if (this._startupHoldRemaining < 0) {
// waiting for the pipe to fill its first full block; the block where
// it does is the first held one
gated = true;
if (available >= frameCount) {
this._startupHoldRemaining = hold - 1;
}
} else {
this._startupHoldRemaining--;
gated = true;
}
}

const toExtract = gated ? 0 : Math.min(available, frameCount);

this._blockCount++;
if (available < frameCount) {
if (!gated && available < frameCount) {
this._underrunCount++;
}

Expand Down
Loading