How to Clone a Voice Locally with Audio8 TTS 0.1B

Voice cloning sounds like one of those AI tasks that should require a huge model and a complicated setup.

That is why Audio8 TTS Preview 0.1B caught my attention.

It is a compact text-to-speech model that supports zero-shot voice cloning: you provide a short reference recording, tell the model exactly what was spoken in that recording, and then ask it to synthesize new text in a similar voice.

The model card describes the main generative model as roughly 170 million parameters, with a separate codec decoder of about 120 million parameters. The repository also includes the codec, tokenizer, processor, and custom Hugging Face code needed for inference.

For someone learning local AI, this makes Audio8 interesting for a simple reason: voice cloning no longer has to start with a multi-billion-parameter speech model.

This guide covers reference preparation, local synthesis, and adjusting the speaking speed of the generated WAV file.

Use voice cloning only with your own voice or with clear permission from the speaker. Synthetic speech can be convincing enough to create real impersonation risks.

What zero-shot voice cloning actually means

You do not fine-tune the model on hours of recordings.

Instead, the model receives three important inputs:

reference audio
+
reference transcript
+
target text

The reference audio tells the system what the speaker sounds like.

The reference transcript tells it what words were spoken in that audio.

The target text is the new sentence you want the cloned voice to say.

Conceptually:

reference.wav
"This is exactly what the speaker said."
            ↓
     Audio8 TTS
            ↓
"A completely new sentence"
            ↓
      output.wav

That is why the transcript is not optional decoration. The official Audio8 instructions explicitly say that the reference transcript should match the spoken content of the reference audio.

Install the model

Audio8’s 0.1B Hugging Face model uses custom Transformers code, so it is loaded with trust_remote_code=True.

The model card currently recommends Python 3.11 or newer and a CUDA-capable GPU, with these core packages:

pip install "torch>=2.5.0" "torchaudio>=2.5.0" \
  "transformers>=4.57.0,<5" "soundfile>=0.12" "safetensors>=0.4"

Use the CUDA build of PyTorch appropriate for the machine, then confirm that torch.cuda.is_available() returns True before the first GPU run.

A minimal cloning script

The official Hugging Face interface is straightforward:

import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Edge0/Audio8-TTS-Preview-0.1b"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).eval().to(device)

inputs = processor(
    text=["Keep practicing a little every day. Small improvements add up."],
    reference_audio=["reference.wav"],
    reference_text=["This must exactly match the words spoken in reference.wav."],
    return_tensors="pt",
)

inputs = {name: value.to(device) for name, value in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        do_sample=True,
        return_dict_in_generate=True,
    )

waveforms, waveform_lengths = model.decode_audio(output.codes)

audio = waveforms[0, : int(waveform_lengths[0])].float().cpu().numpy()
sf.write("output.wav", audio, model.config.codec_sample_rate)

Those sampling values match the defaults shown in the model’s current generation code and documentation.

Keep these settings for the first run.

Changing five sampling controls before you know what the baseline sounds like makes troubleshooting harder.

The reference recording matters more than people expect

The easiest mistake is to spend all your time tuning temperature while using a poor reference sample.

A useful reference clip should be clean, easy to transcribe, and representative of the voice you actually want to reproduce.

The Audio8 documentation warns that very long, noisy, or inaccurate reference clips can reduce stability and speaker similarity.

Prepare the reference recording in this order:

1. Clear speech. Avoid music, another speaker, television noise, strong echo, or aggressive noise reduction artifacts.
2. Exact transcription. Do not “correct” what the speaker said. If the recording contains a hesitation or a particular wording, the transcript should describe the actual audio as closely as possible.
3. Natural delivery. A conversational sentence is usually more useful than a flat sequence of isolated words.
4. Useful phonetic variety. A reference that contains different vowels, consonants, sentence rhythms, and punctuation gives the model more information about how the speaker sounds across different speech patterns.

A reference with varied sounds is more informative than a short greeting such as:

Hello, this is my voice.

That sentence is valid, but it gives the system very little variety.

A better reference-text example

For a new English reference recording, a varied passage could read:

Good morning. I wanted to check in before we leave, because the weather changed quickly overnight. I packed the blue notebook, the small charger, and a bottle of water. If we have enough time, we can stop for coffee on the way; otherwise, let's go straight there and come back before dinner.

This is better than a short greeting because it naturally contains:

statements
commas
short pauses
longer phrases
questions of emphasis
different consonant clusters
numbers of syllables per phrase

But there is one critical rule:

Do not paste this text into reference_text unless the reference recording actually contains these words.

The transcript and audio must stay paired.

Multilingual reference audio follows the same rule

The model card identifies Chinese and English as the primary languages. German, Spanish, French, Italian, Japanese, and Korean are listed for experimental multilingual evaluation.

The same reference principle applies regardless of language: use a clean recording with natural phrasing, enough phonetic variety to represent the speaker, and a transcript that matches the recording exactly.

If you want to test cross-language cloning, keep the reference sample controlled first. It is much easier to diagnose speaker-similarity problems when you know the source recording itself is clean and correctly transcribed.

Test with target text that is different from the reference

If your target sentence is almost identical to the reference recording, it is difficult to tell whether you are testing genuine zero-shot cloning or simply giving the model an unusually easy continuation.

A better test changes both wording and sentence structure.

For example:

You do not need to improve everything today. Practice one skill carefully, repeat it a few times, and then come back tomorrow. Consistency matters more than one perfect session.

This kind of target is useful because it contains multiple phrases and natural changes in pacing without copying the reference sentence.

Why the same reference can produce slightly different audio

Audio8 uses sampling during generation by default.

The current implementation exposes familiar controls such as:

temperature = 0.7
top_p = 0.9
top_k = 50
do_sample = True

That means two generations do not have to be acoustically identical.

This is normal.

Save several outputs using the same reference and target text before comparing sampling settings.

If the goal is simply to make a usable clip, listen for speaker similarity, pronunciation, rhythm, artifacts, and whether the emotional tone fits the text.

What if the cloned voice talks too fast?

If the generated voice has the desired identity but the delivery is too fast, adjust the finished audio before changing generation settings.

The simplest solution does not require regenerating the voice at all.

Use FFmpeg’s atempo audio filter after synthesis:

ffmpeg -i output.wav -filter:a "atempo=0.7" output_slow.wav

atempo=1.0 is the original tempo.

atempo=0.7 plays the speech at 70% of the original tempo, which is a straightforward interpretation of making it about 30% slower.

FFmpeg’s atempo filter is specifically designed to adjust audio tempo rather than simply changing the sample rate.

This is useful because it lets you separate two problems:

voice generation
      ↓
Do I like the cloned voice?

post-processing
      ↓
Do I like the final speaking speed?

That separation makes experimentation much easier.

There is a limitation, though. Slowing finished audio is not the same thing as asking the TTS model to perform a naturally slower speaking style. Post-processing stretches the timing of the whole clip. If you need expressive control over pauses, emphasis, or emotional delivery, generation-time prosody control would be preferable when the model supports it.

Keep target sentences reasonably short

The Audio8 project recommends keeping individual synthesis inputs within about 150 characters for best quality in its current preview tooling. Longer text can be split into shorter segments.

That is a useful habit even outside Audio8.

Instead of asking a small TTS model to synthesize an entire page in one generation, break the text into sentences or short paragraphs and join the WAV files afterward.

This gives you more control over retries and makes it easier to replace one bad sentence without regenerating everything.

The “0.1B” name deserves context

The model name makes Audio8 sound almost impossibly small.

The current model card clarifies that the main generative model is approximately 170M parameters, not literally 100M parameters, and that the audio codec decoder is another roughly 120M parameters.

So the complete stack is larger than the name alone suggests.

It is still compact compared with many modern neural TTS systems.

That is the real reason this model is interesting: it pushes usable zero-shot TTS into a size range where local experimentation becomes much easier.

Check speaker similarity and speech quality

Compare several sentences rather than judging a clone from one output.

Use several target texts that stress different characteristics: short statements, longer sentences, numbers, names, questions, different supported languages if relevant, and sentences with natural punctuation.

Then listen for two different things.

First, speaker similarity: does it sound like the reference speaker?

Second, speech quality: is it clear, natural, correctly pronounced, and free of strange pauses or codec artifacts?

A system can score well on one and badly on the other.

That distinction matters because “this sounds like the person” and “this is good speech synthesis” are not identical tests.

A repeatable first run

Audio8 TTS Preview 0.1B makes zero-shot voice cloning unusually approachable.

The Python interface is small, the reference workflow is easy to understand, and the model is compact enough that local experimentation does not require datacenter hardware.

Use a clean reference recording and an exact transcript before tuning generation parameters.

Start with a clean reference. Make the transcript match. Use a target sentence that is genuinely different. Generate a few samples. Only then start tuning sampling or post-processing the tempo.

And if the clone sounds good but simply talks too fast, a one-line FFmpeg command may solve the problem more cleanly than another hour of parameter tweaking.

If you are still separating speech-recognition concepts from speech generation, Speech-to-Text vs Text-to-Speech is a useful starting point. For the recognition side of the stack, How Does Whisper Work? explains how audio is converted in the opposite direction: from speech back into text.