Trung Dang, David Aponte, Dung Tran, Tianyi Chen, Kazuhito Koishida
Applied Sciences Group, Microsoft Corporation

[arXiv]

Abstract

Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce LiveSpeech 2, a stream-aware model that supports infinitely long speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process an infinite text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios.

Model Architecture

Model Architecture

Samples (Short)

Transcript Enrollment Reference LiveSpeech v2 YourTTS SpeechX XTTS-v2 MetaVoice LiveSpeech v1
the girl entered and gave an involuntary cry of surprise
he would be like a philologist refusing to examine a fact in language a philosopher hesitating to scrutinize a fact in humanity
the beggar is plea the politician is scepter and the drummer is ablest assistant
the lord who has given us power to teach and to hear let him also give us the power to serve and to do
i could not see that in either case holmes had come upon the clue for which he was searching
the boolaroo stared a moment and then he fell back in his throne laughing boisterously
every word fell distinctly in perfect harmony and eloquence upon louis xiv is ears
rodolfo thanked her and supper being ready they went to join the rest of the party at table
he heard that she was downstairs and ran to meet her with beating heart
i wanted to talk with thee a little about thy plans

Samples (Long)

Transcript Enrollment Reference LiveSpeech v2 YourTTS XTTS-v2
and immediately on his sitting down there got up a gentleman to whom i had not been introduced before this day and gave the health of missus neverbend and the ladies of britain eula
the definition of truth as the correspondence between beliefs and facts seems peculiarly evident in the case of memory as against not only the pragmatist definition but also the idealist definition by means of coherence
if you stay it must be to sit down here in the shadow and take the part of uncas until such times as the cunning of the indians discover the cheat when as i have already said your times of trial will come
henry was about 23 years of age of an active turn brown skin and had given the question of freedom his most serious attention as his actions proved
this was the call of life to his soul not the dull gross voice of the world of duties and despair not the inhuman voice that had called him to the pale service of the altar
they might be deaf to the equalities proclaimed in the declaration of independence or blind to the moral sin of slavery but they comprehended a rifle which could be fired 10 times a minute and kill a man at a 1000 yards
under the simple test of effectiveness for advertising we should expect to find leisure and the conspicuous consumption of goods dividing the field of pecuniary emulation pretty evenly between them at the outset
the english commissioners who some days after came to take him under their custody were admitted to kiss his hands and he received them with the same grace and cheerfulness as if they had traveled on no other errand than to pay court to him
but violent as were the internal and alimentary fluids the tide became still more violent when the body came into contact with flaming fire or the solid earth or gliding waters or the stormy wind the motions produced by these impulses pass through the body to the soul and have the name of sensations
both the count and the captain expressed themselves reassured by his representations and observed that they must now do all in their power to discover what had become of the vast continent of africa of which they were hitherto failing so completely to find a vestige