Learned residual suppressor: measured ML benchmark, clean-licensed pipeline, and the pluggable-post chain - #33
Merged
Merged
Conversation
tools/ml: a scenario dump (the test rig's double-talk protocol plus a near-end-only transparency segment, emitted as raw float64 via the rig's own generators and simulator) and a Python harness that runs MuTap's shipped engines (via the C ABI) and DTLN-aec (MIT, Westhausen & Meyer 2021) on identical signals with one shared meter: delay-compensated ERLE, double-talk true-echo suppression, and near-end preservation SDR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
tools/ml/make_dataset.py synthesizes echo scenarios from LibriSpeech (CC BY 4.0): random rooms, bulk delay, speaker saturation, SER sweep, gated double-talk — then runs MuTap's linear Kalman canceller via the C ABI and emits (features, oracle band gains, loss weights) shards for the learned residual suppressor. tools/ml/features.py is the single source of truth for the analysis geometry (128/64 sqrt-Hann STFT, 22 ERB-spaced bands) that the trainer and the C++ inference must share. A --scenarios mode emits protocol-layout speech scenarios so run_benchmark.py measures speech material with the same meter (--scenario-dir). Measured on three speech scenarios (medians): DTLN-aec-512 43.4 dB single-talk ERLE / 14.2 dB double-talk suppression / 29.3 dB near-end SDR; mutap-chain 30.9 / 17.2 / 38.2 dB; on the rig's synthetic materials DTLN collapses to ~0 dB suppression and 0..5 dB near-end SDR (out of its speech-trained domain) while the classical chain holds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
train_suppressor.py: dense64-tanh -> GRU96 -> dense22-sigmoid (~51k parameters), loss-weighted MSE on sqrt-compressed oracle gains. nn.py: dependency-free numpy reference inference (the contract the C++ header must match) plus the mutap-kalman+nn benchmark system, wired into run_benchmark.py as --nn-weights. README.md: the licensing map this effort exists to establish, the measured baseline tables, and the reproduction recipe. Oracle sanity bound on a speech scenario (upper bound for the learned gains): 67.8 dB single-talk ERLE / 16.5 dB double-talk suppression / 59.3 dB near-end SDR, vs 7.0 / 9.6 / 80 for the linear canceller alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
include/mutap/nn_suppressor.h: dense64-tanh -> GRU96 -> dense22-sigmoid band-gain suppressor with the classical residual_suppressor's process_block(e, yhat, out) contract — sqrt-Hann 128/64 analysis over [previous block, current block], 22 ERB-band features of E and Yhat, per-bin gain application on the E spectrum, one block of added latency, noexcept/allocation-free after construction. Yhat feeds only the features, never the signal path, so a wrong prediction can only mis-gain a band, not synthesize signal. tools/ml/nn_infer.cpp + test_parity.py pin the C++ against the numpy reference on identical frames (measured 2.3e-8 relative — FFT-backend depth); features.py's framing is now streaming-consistent (zero prehistory warm-up frame) so the GRU state trajectories match exactly. tests/test_nn_suppressor.cpp pins the plumbing structurally: forced unit gains reconstruct as a transparent one-block delay at float64 depth, forced zero gains silence the output, wrong geometry throws. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
An interrupted run keeps its best-so-far model instead of losing hours of CPU training; also detach the loss before accumulating the epoch statistic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
pretrained/suppressor_v1.munn: 51,670 parameters, trained 12 epochs on 400 Mini-LibriSpeech mixtures via the tools/ml pipeline; every input is MIT / CC BY 4.0 / synthesized here, so the weights are clean to ship. C++ inference verified against the numpy reference with these exact weights (1.0e-7 relative). Measured (speech scenarios): the hybrid beats the classical chain's single-talk ERLE by ~5 dB at equal near-end transparency; double-talk suppression stays at linear-canceller level. Off-domain (the rig's synthetic materials): degrades gracefully — near end survives at 26-39 dB SDR where DTLN-aec annihilates it (0-5 dB) — but its double-talk contribution evaporates, so the classical coherence suppressor remains the right default engine. README carries the full tables and the ranked next steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
notebooks/ml_aec_comparison.ipynb (built by tools/ml/build_ml_notebook.py, same convention as the other notebooks: the builder is the source, the .ipynb a build product with outputs committed). Two scenarios through one meter — the rig's synthetic music double-talk and a LibriSpeech speech scenario — with residual-level timelines per system, the metric tables, and heatmaps of the hybrid's learned band gains in and out of its training domain (where the coherence-gate behavior it learned, and the off-domain flicker that costs it double-talk suppression, are both directly visible). nn.py gains a MUNN loader so the committed pretrained weights drive the notebook without training artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
The industry chart for this experiment — send level relative to the far end over time, with the -30 dB post-double-talk target line — for all four systems on both scenarios. The far-end reference is local (smoothed block power) with an activity gate, so real speech's pauses read as gaps instead of fake infinite loss; the accompanying text explains the meter's deliberate blindness (near-end speech counts against 'loss' during double-talk, so a trace that stays flat there is suppressing the talker). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
tools/ml/webrtc_aec3_infer.cpp: offline driver for the AEC3 canceller from freedesktop's webrtc-audio-processing (BSD-3, >= 1.0 — the distro 0.3.x packages carry only the legacy canceller and are not accepted by the CMake probe). Optional target, appears when pkg-config finds the library; wired into run_benchmark.py (--webrtc-aec3-bin) and the comparison notebook, build recipe in the README. Measured on the speech scenarios (medians): 37.9 dB single-talk ERLE / 2.7 dB double-talk suppression / 39.2 dB recovery ERLE / 4.5 dB near-end SDR — strong echo removal, and the worst near-end transparency of the field: AEC3's suppressor clamps the talker, a telephony-shaped trade and the opposite of what program material wants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Focused three-way comparison (the clear-verdict companion to ml_aec_comparison): coupling-loss timelines in the industry chart idiom with the double-talk window shaded, true-residual timelines from the simulator's ground truth, and a four-panel scorecard over three speech scenarios and the three synthetic materials. Fixed colorblind-validated color per system across every figure. The executed notebook follows in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Full run committed with outputs: coupling-loss and true-residual timelines for a speech and a music scenario, and the scorecard over three speech scenarios / three synthetic materials. Headline medians — speech: AEC3 37.9 dB stERLE but 4.5 dB near-end SDR; chain 30.9 / 38.2; hybrid 36.0 / 37.6. Synthetic: AEC3 degrades everywhere (26 dB ERLE, 8.7 dB near-end SDR); chain 41.9 / 36.1 with 15 dB double-talk suppression; hybrid 43.6 / 33.2 with the known off-domain double-talk gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Fourth categorical color (yellow, validated colorblind-safe in order with the existing three), intro/verdict updated with DTLN's two faces — best-in-field single-talk ERLE on speech, talker deletion off-domain, 10.4M parameters vs the hybrid's 51k, benchmark-only weights. The re-executed notebook follows in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Four systems on every figure and table. Headline medians — speech: DTLN leads single-talk ERLE (43.4 dB) but trails on recovery (26.3) and near-end SDR (29.3); the chain keeps the double-talk crown (17.2 dB) and the hybrid the fidelity/ERLE balance (36.0 / 37.6). Synthetic: both pure-ML systems lose their double-talk contribution, but where the hybrid's talker survives at 33.2 dB SDR, DTLN's is deleted (0.1 dB). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
aec_chain<Sample, Canceller, Post = residual_suppressor<Sample>> — the post-filter engine becomes pluggable with the same capability-detection idiom the canceller side already uses: any type with the suppressor contract slots in, and the receive guard / re-convergence rescue key on echo_explained() when the post provides it (a post without the statistic certifies the guard immediately and disables the rescue, via if constexpr (requires ...)). The classical path is provably untouched: aec_chain<double> still denotes raw FD-Kalman + residual_suppressor, and the branchless-parity fingerprint (tests/branchless_parity_check.cpp, 600 preset blocks) is bit-identical before and after this change for both suppressor forms (d515415bee83b420). ItuChain/postfilter suites pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
The learned suppressor now composes as aec_chain's Post engine on any trained geometry, not just the 16 kHz/hop-64 benchmark model: - nn_geometry (sample_rate, hop, bands, dense, gru) travels WITH the weights: MUNN0002 files carry it in a header; legacy MUNN0001 files imply the original geometry. Band matrix, buffers and the network all size themselves from it; the chain's block size must equal the trained hop (validated). - Config-only construction (weights ride in the config) so aec_chain<Sample, Canceller, nn_suppressor<Sample>> builds it like any other post stage. - echo_explained(): the classical guard/rescue statistic (smoothed Yhat power over mic power, mic reconstructed as E + Yhat per bin) so the receive guard and re-convergence rescue work unchanged. - Comfort noise: two-window minimum-statistics floor tracker on the per-bin E PSD with deterministic-reset xorshift fill, same contract and defaults as the classical engine. Exporter writes MUNN0002 (geometry from the trainer npz when present); the Python reference loads both formats and sizes its GRU from the weights. Tests: transparency/silence/comfort-floor/echo_explained/ geometry-rejection/chain-composition all pass; Python-C++ parity unchanged at 2.3e-8 relative; legacy MUNN0001 verified end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
include/mutap/nn_chain.h: aec_chain_nn<Sample> (raw FD-Kalman + nn_suppressor) and aec_chain_nn_preset — the classical preset's canceller/guard/rescue/shadow calibration verbatim, with the learned post engine's floor-tracker constants rescaled by the same geometry ratio. tools/capi: MutapAec becomes the repo's std::variant idiom; mutap_aec_create_nn(weights_path) constructs the learned-engine chain; every entry point dispatches via std::visit. mutap_ffi.AecChain grows nn_weights so the benchmark measures the real C++ chain. tools/ml geometry parameterization for the 48 kHz-native model: features.Geometry (GEOM16 benchmark / GEOM48 = mutap.aec~'s block-256 default, 26 bands), make_dataset --rate/--taps/--near-materials with numpy ports of the rig's synthetic near-end materials (frequencies pinned in Hz) mixed into training — the measured fix for the hybrid's off-domain double-talk gap — spectral 3x upsampling for LibriSpeech, and FFT convolution; the trainer sizes the model from the shards' geometry and stamps it into the checkpoint. Sanity (16 kHz v1 weights through the C ABI on a speech scenario): chain-nn 58.1 dB stERLE / 37.8 dB neSDR vs classic 50.9 / 48.6, with the known v1 double-talk gap (9.6 vs 17.2 dB) the 48 kHz retrain's mixed materials target. Parity 2.3e-8 unchanged; NnSuppressor suite passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
pretrained/suppressor_v2_48k.munn: MUNN0002, 52,570 parameters, trained 12 epochs on 400 mixtures at the mutap.aec~ geometry (48 kHz, hop 256, 26 bands) with the off-domain fix in the data: near ends drawn from speech AND the rig's synthetic families (AR envelope, voiced, music). Every input remains clean-licensed (LibriSpeech CC BY 4.0 upsampled, synthetic material ours). Measured through the C ABI chain at 48 kHz (3 speech scenarios, medians): nn48 53.4 dB single-talk ERLE / 11.2 dB double-talk suppression / 34.3 dB near-end SDR vs classic 34.5 / 12.6 / 30.3 — the learned engine now leads ERLE and transparency at near-parity double-talk. Off-domain (music double-talk): 4.0 dB suppression at 30.2 dB near-end SDR, vs the 16 kHz v1's ~0 dB collapse — the training mix closed most of the gap; classic still leads there (9.1 dB) and remains the default. nn_suppressor gains parse_nn_suppressor_weights (in-memory MUNN image) so a package can embed the default model as a byte array; export_weights.py --header emits that array. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
book/src/echo-cancellation.md gains 'The learned engine: @PostFilter 2' — what it is (a 52k-parameter band-gain network in the post-filter slot, structurally unable to synthesize signal), the measured trade (~53 vs ~35 dB single-talk ERLE at better transparency; classical keeps the double-talk crown off-domain and stays the default), and the practicalities (@block follows the model, @model loads user-trained weights, all training inputs clean-licensed) — plus the revised knobs list. README gains the library-status bullet for nn_suppressor / aec_chain_nn / tools/ml; HANDOFF records the whole effort as Stage 6 with the filed follow-ups (coherence feature, int8/CMSIS-NN, 44.1 kHz model, ITU characterization run). webrtc_aec3_infer gains an optional sample-rate argument so the AEC3 baseline joins the notebooks' 48 kHz shipping-geometry section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Markdown-only cross-reference (no outputs touched), synchronized between the builder and the committed notebook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
New closing section measured at mutap.aec~'s native geometry through the engines' real deployment paths (the C ABI chain — exactly what @PostFilter 1/2 run): the v2 learned engine posts 53.4 dB single-talk ERLE / 51.9 recovery / 34.3 near-end SDR vs the classical chain's 34.5 / 31.2 / 30.3, with double-talk suppression within 1.5 dB (11.2 vs 12.6); WebRTC AEC3 at 48 kHz shows its telephony trade again (42.3 dB ERLE, 7.8 dB near-end SDR). DTLN-aec sits this section out — its models are inherently 16 kHz. webrtc_aec3_infer gained the sample-rate argument this section uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
The prose had accreted section by section (three systems, then four, then the 48 kHz addendum) and read like it. Rewritten as one arc: - The intro announces the two-part structure and why it exists (DTLN pins Part 1 to the 16 kHz benchmark geometry; Part 2 is the 48 kHz shipping geometry without it), and that 'MuTap learned' appears in two generations — the v1 prototype in Part 1, the shipping v2 model in Part 2. - One name and one color per engine identity everywhere: 'MuTap classical' / 'MuTap learned (v1|v2)' in prose, legends and tables alike (the chain/hybrid drift is gone). - Section transitions written for the arc: the coupling-loss reading interprets both charts together, the residual view is introduced as what the deployable meter cannot see, Part 1 closes with a per-system summary that sets up Part 2, and Part 2 states its realism deltas (real C ABI deployment path, AEC3 at 48 kHz) explicitly. - The verdict now accounts for everything measured, including the v1 -> v2 off-domain repair and the licensing asymmetry. Measurements unchanged (deterministic seeds; tables verified identical to the previous execution). Re-executed and committed with outputs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Owner
Author
|
CI status note: the Generated by Claude Code |
tap
pushed a commit
to tap/MuTap-Max
that referenced
this pull request
Aug 8, 2026
tap/MuTap#33 merged (rebase); the pin moves from the branch head to the identical tree on main (1d258d3), per the submodule rule in the PR notes. Package builds and tests green against the pinned submodule itself (no MuTap_ROOT override). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
tap
pushed a commit
to tap/MuTap-Max
that referenced
this pull request
Aug 10, 2026
tap/MuTap#33 merged (rebase); the pin moves from the branch head to the identical tree on main (1d258d3), per the submodule rule in the PR notes. Package builds and tests green against the pinned submodule itself (no MuTap_ROOT override). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds a learned (neural) residual-echo suppressor as a selectable post-filter engine —
tap::mu::nn_suppressor+aec_chain_nn, withaec_chaingrowing a pluggablePosttemplate parameter — plus thetools/ml/benchmark and clean-licensed training pipeline that measured the design into existence, the trained models (16 kHz benchmark v1, 48 kHz shipping v2), C ABI support, two executed comparison notebooks, and book/README/HANDOFF documentation. Companion PR: tap/MuTap-Max (themutap.aec~ @postfilter 0/1/2+@modelsurface).Why
The originating question: is there an ML approach to echo cancellation that is not IP-encumbered? Measured answer: end-to-end neural AECs are (DTLN-aec deletes out-of-domain near ends at 0–5 dB SDR; its weights carry mixed-license training data), but a small learned band-gain post-filter over MuTap's own linear canceller wins single-talk ERLE at equal transparency — and every input to its training is MIT / CC BY 4.0 / synthesized here, so the weights are clean to ship. The post stage is also the one classical stage that is a heuristic, i.e. the one place a learned rule plausibly beats DSP.
Verification
tests/branchless_parity_check.cppfingerprintd515415bee83b420before and after thePosttemplate refactor, bothMUTAP_SUPPRESSOR_BRANCHLESSvalues.tests/test_nn_suppressor.cpp(transparency at forced unit gains to −140 dB, silence at zero gains, comfort-noise floor level,echo_explainedextremes, geometry rejection, chain composition).tools/ml/test_parity.py, 2.3e-8 relative on identical frames; re-verified with the shipped weights (1.0e-7).notebooks/ml_aec_comparison.ipynb,notebooks/aec_head_to_head.ipynbincl. the 48 kHz shipping-geometry section) and tabulated with provenance intools/ml/README.md. Headline (48 kHz speech, medians, C ABI chain): learned v2 53.4 dB single-talk ERLE / 34.3 dB near-end SDR vs classical 34.5 / 30.3, double-talk within 1.5 dB; off-domain music double-talk 4.0 vs 9.1 dB — classical remains the default engine.nn_suppressorare untested on target (host float32 instantiation is exercised; int8/CMSIS-NN filed).Notes for the reviewer
aec_chaingains a third template parameter with a default — source-compatible for every existing consumer (aec_chain<double>unchanged, verified by fingerprint);residual_suppressoruntouched. New MUNN weights format is versioned (0002 self-describing geometry; 0001 still loads).pretrained/*.munnweights were trained exclusively on LibriSpeech (CC BY 4.0) + material synthesized in-repo. Survey intools/ml/README.md.tools/ml/is host-side tooling behindMUTAP_BUILD_ML_TOOLS(default OFF) — nothing enters the emulated-target builds.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y3Y23Xavpd31igC1mDDcWg
Generated by Claude Code