Xtts faster inference - #4439
Closed
yunusemretom wants to merge 4 commits into
Closed
Conversation
Both only trigger when a GPU is present, which is why they survived CI. `_forward_encoder` cast speaker ids with `.type(torch.LongTensor)`, which is the CPU long tensor type, so the ids were moved away from the CUDA `emb_g` embedding and training multi-speaker FastPitch/FastSpeech2/DelightfulTTS died on a device mismatch. Use `.long()`, which keeps the device. DelightfulTTS's local `id_to_torch`/`embedding_to_torch` called `.cuda()` on the `None` they had just decided not to convert. `numpy_to_torch` next to them and the copies in `TTS/tts/utils/synthesis.py` already return early instead; do the same here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Profiling XTTS v2 on an RTX 4050 Laptop puts 93% of inference in the autoregressive GPT loop. That loop is ~100% GPU-busy with no launch-overhead gap, and its step time barely moves with batch size (7.0 ms at B=1, 9.5 ms at B=4), so it is bound by reading the GPT's 379M weights rather than by arithmetic. Two changes follow from that; swapping GPT2's eager attention for SDPA was also measured and gave exactly 1.00x, so it is not included. Half precision halves the traffic that sets the pace. Three things blocked it: - `null_position_embeddings` returned hardcoded float32 zeros, and GPT2 adds those to the input embeddings, promoting every activation back to float32. It is now a module carrying a non-persistent buffer, so its dtype follows `.half()` / `.to()` on the parent. - `GroupNorm32` cast its input to float32 but not its own weights, so half precision hit it as a mixed-dtype error. It now casts both, which is what normalizing in float32 was meant to mean. - Conditioning latents and vocoder inputs did not follow the model's dtype. `use_half_precision()` leaves the conditioning encoder in float32, where it overflows to NaN in half precision and costs nothing to keep, and the vocoder likewise. `load_checkpoint(half=True)` applies it at load time. Batching decodes several sentences in one pass, so the weights are read once for the whole batch. `inference(batch_size=N)` groups sentences by length to waste as few decoding steps as possible; grouping does not affect results, so the original order is restored afterwards. Verified by decoding greedily both ways: the token sequences are identical. Paragraph-length text, RTX 4050 Laptop: RTF 0.259 -> 0.178 with half precision, -> 0.092 with batch_size=4, and peak VRAM drops from 2.21 to 1.49 GB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolving requirements.txt against current package versions produces an environment that imports but does not work: cu130 torch wheels against a CUDA 12.4 driver, librosa reaching for the removed pkg_resources, numpy 2 against a cython extension built for 1.x, and transformers 5 missing the generation symbols stream_generator.py imports. KURULUM.md records the version set that passes the tests, the espeak-ng symlink the phonemizer already expects, how to run commands when ROS leaks into PYTHONPATH, which test failures are environmental rather than bugs, and how to turn on the half precision and batched decoding paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.