Speech you can hear: how text-to-speech works

Start here

How this works

Most clips come from Bulbul V4 Flash, Sarvam’s text-to-speech model for Indian languages. On many slides Bulbul also explains the idea out loud, in its own voice.

Each slide plays by itself the first time you open it: Bulbul’s explanation, then the clips in order. Press Play this slide to hear it again, or any clip on its own.

Where Bulbul gets everything right, there would be nothing to hear, so some slides use another model that gets it wrong: Kokoro, a small open model I run on my laptop, or OpenAI’s TTS. Each slide says which.

Clips are labelled A, B, C. Within a group, only one thing changes: the script, the reader, the punctuation, the pace, or the style.

Headphones help, especially on the phone-line slide.

Reading the text · Text normalisation

Numbers, dates and acronyms

Before a model can speak, it rewrites symbols as words. This is called text normalisation, and the rules are local. An Indian reader says ₹5,00,000 as “five lakh rupees”. A reader built on American rules has never seen commas in those places.

The same digits also change with context. 1800 is a phone number, a year, a price or a time, depending on the words around it.

Same text, two readers

Same digits, four contexts

Reading the text · Spelling to sound

Devanagari vs romanised Hindi

A model turns letters (graphemes) into sounds (phonemes). Devanagari spells Hindi almost one to one. Latin letters bring English habits with them: main becomes “mane”, do becomes “doo”, ruko becomes “roo-koh”.

Same voice, same model, same sentence. Only the script changes. Under each clip are the sounds the model decided to say.

Listen

Clips from Kokoro-82M with its Hindi voice. The romanised text goes through English spelling rules.

Reading the text · Code-mixing

Hinglish needs a reader who knows both

Indians mix Hindi and English inside one sentence. For every word, the reader has to decide: Hindi rules or English rules? A reader that knows only one language gets the other half wrong.

The same romanised Hinglish line, read by an English-only voice and by a Hinglish voice. Listen to kal, ho gayi and karte hain.

Listen

Reading the text · Heteronyms

Same spelling, two sounds

Some English words are spelt the same but said two ways: read (reed / red), record (REK-ord / ri-KORD), wind. The letters cannot decide. The model has to work out the grammar of the whole sentence.

Each clip says one spelling two ways, a few words apart.

Listen

Reading the text · Schwa deletion

The hidden vowel Hindi drops

Devanagari writes a short a after most consonants, but spoken Hindi drops many of them. कमला is kamlaa, not kamalaa. नमकीन is namkeen. The model must know which ones to drop.

Here the model is given the sounds directly, so only the vowels change. B keeps every written vowel and sounds like someone reading letter by letter.

Listen

Clips from Kokoro-82M, given phonemes instead of text.

Reading the text · Pronunciation dictionaries

Fixing a name

Every voice gets some names wrong. The usual fix is a pronunciation dictionary: a list of “write this, say that” pairs applied to the text before the model sees it. Respelling the word is the simplest version.

Listen

Clips from OpenAI tts-1.

Prosody · Pauses and melody

Move a comma, change the meaning

Prosody is the melody of speech: pitch, pauses and timing. Bulbul has no pause tags, so punctuation is the control. Move one comma and the meaning flips. Change a full stop to a question mark and the end of the sentence should rise.

Where the comma goes

Statement or question

Prosody · Emphasis

Stress one word, change the meaning

Text has no mark for stress, but stress changes meaning. “I didn’t say she stole it”: someone else did. “I didn’t say she stole it”: I only hinted. Some models take a spoken instruction for this. Bulbul has no stress control.

Same words, stress moves

Clips from OpenAI gpt-4o-mini-tts, with an instruction naming the word to stress.

Prosody · Duration

Speaking rate

Models like FastSpeech predict how long each sound lasts. A pace setting scales those predictions, here from half speed to double. Because the model re-plans the speech, fast speech stays clear. Compare it with D, the normal clip simply played twice as fast.

Listen

The voice · Speaker and style

One line, four styles, two ways

A voice has two parts: who is speaking, and how. Bulbul ships one speaker recorded in several styles; you pick the style by picking the voice. OpenAI takes the style as a free-text instruction. Each row below is the same line and the same style, made both ways.

Sports

Late-night radio

Devotional

Documentary

The voice · Languages

Ten languages, one hello

Each language has its own sounds and its own script, and a model is only as good as its data in each. The gap between languages is wide: letter-to-sound error is about 0.5% for English and Czech, but about 38% averaged over 507 languages.

One greeting in all ten languages that have Bulbul V4 voices. Hindi and English have over a hundred voices; Tamil, Telugu and Kannada have four each.

Listen

Delivery · The phone channel

On a phone line

Wideband speech is sampled at 16 kHz or more. A classic phone line samples at 8 kHz and passes only about 300 to 3400 Hz. The high frequencies that make "s" and "sh" crisp are cut off. Telephone speech also roughly doubles speech-recognition error.

Most Indian voice bots live on this narrow channel, so a voice has to stay clear on it. Clip B is clip A, band-limited and resampled to 8 kHz the way a phone line would.

Listen

Delivery · Streaming

Smaller chunks, sooner sound, up to a point

A voice agent wants to start speaking before the language model has finished writing. So it cuts the reply into chunks and sends each chunk to TTS as soon as it arrives. Smaller chunks mean the first sound comes sooner. I measured this with Kokoro-82M on a 67-second essay.

Two things matter: how soon the first sound comes (time to first audio), and whether the speech still flows once it starts.

The fastest start is token by token, at 474 ms. But the gain flattens early. Going from the whole reply to clauses cuts the wait from 9 s to 0.6 s. Going on from clauses to tokens saves only another 158 ms, and it wrecks the speech. The next chunk always arrives in time: no run ever stalled. The problem is that each fragment is spoken like a sentence of its own, with its own rise, fall and pause. The voice sounds like a toddler reading word by word, and the essay takes 331 s to say instead of 67.

The sweet spot is a short first chunk, then whole sentences: 516 ms to first audio, only 42 ms slower than token by token, from 6 calls, with speech just 4% longer and no audible breaks. Clauses are close behind.

Each clip is the essay’s first sentence, spoken by joining the chunks

“In 1952, at Bell Labs, Dr. Davis and his colleagues built Audrey, a machine that recognised spoken digits from a single speaker.”

Measured on an RTX 4090 Laptop, synthetic token stream at 40 tokens/s.

Limits · Emotion and expression

Laughs, sighs and whispers

Laughs, sighs and whispers are not words. A model needs a separate way to make them, such as a tag or an instruction. Without one it reads the letters. Each row is the same line: typed into Bulbul V4 Flash, which has no tags, and given to OpenAI as an instruction.

Laugh

Whisper

Sigh

The end, for now

That's all for now

That is the tour of what text-to-speech does between the text and the sound: reading the text, prosody, the voice, and delivery.

The clips were made once and are stored as small files with this page, so nothing here calls a TTS service when you press play.

← Back to all posts