
"The answers we seek are never where we first look. They hide in the architecture of the machine."
TL;DR: I set out to turn a Spanish ebook into a full-cast audiobook with code. Along the way I fought an LLM that forgot its own characters, a TTS engine that mistook Spanish for English, and a server that turned out to be dead weight. This is the case file: what broke, what the clues were, and how I fixed it.
🧩 The Case Board
Before we open the file, meet the suspects, the witnesses, and the tools of the trade.
| Component | Technology | Role in the Case |
|---|---|---|
| Prosody Engine | Google Gemini Flash | Reads each paragraph, injects voice and emotion tags |
| Narrator TTS | OmniVoice (local GPU) | English-taxonomy voice design via instruct strings |
| Character TTS | Microsoft Edge-TTS | Regional Spanish neural voices over a direct WebSocket |
| Premium TTS | Google Gemini GenAI | High-fidelity prebuilt voices (Aoede, Charon, etc.) |
| Audio Assembly | PyDub | Stitches mixed WAV/MP3 segments into chapter files |
| Audio Mastering | PyDub + Mutagen | −20 dBFS normalization, mono conversion, ID3 tagging |
| Code Intelligence | Graphify | Knowledge graph and codebase map |
| Agent Skills | Custom .md skill protocols |
Specialized AI assistant behaviors per task type |
| Orchestration | orchestrate_book.py |
End-to-end pipeline lifecycle |
| Voice Memory | voice_memory.json |
Persistent cross-chapter character voice registry |
And here is the pipeline as it stands at the end of the investigation:
flowchart LR
A[EPUB] --> B[Markdown]
B --> C[Chapters]
C --> D["Prosody tagging<br/>(Gemini)"]
D <--> M[("voice_memory.json")]
D --> E["Synthesis<br/>(Edge-TTS · Gemini · OmniVoice)"]
E --> F["Assembly<br/>(PyDub)"]
F --> G["Mastering<br/>(-20 dBFS + ID3)"]
🌒 Prologue: The Book That Could Not Be Read
11:47 PM. The hard drive hummed.
On the screen, a digital copy of 1984 sat inside a folder, perfectly intact and perfectly silent. George Orwell's haunting masterpiece. The Spanish translation, painstakingly preserved. The text had been obtained.
But the voice did not yet exist.
Not just any voice. The living, breathing, emotionally complex voice required to carry five hundred pages of dystopian prose, the kind that could do justice to Winston Smith's despair. Somewhere between the raw bytes of an EPUB file and the rich audio of a professional narrator lay a gap. A cipher. One that only code, machine intelligence, and a very stubborn engineer could crack.
The developer leaned back, studying the empty waveform canvas.
It would need to be built. Not recorded.
Built.
This is the account of what happened next.
📂 Chapter One: The Codex Begins
Every great investigation starts with a simple question. This one was deceptively simple:
"How hard could it be to turn a Spanish ebook into an audiobook?"
The answer, as it turned out, required months of architectural design, three different text-to-speech engines, an AI-powered character memory system, a Gemini LLM running prosody analysis on every paragraph, and enough refactoring to make a seasoned engineer weep.
But that was all to come.
The initial plan, sketched on a digital whiteboard, looked like this:
- Convert an EPUB file to clean Markdown.
- Split the Markdown into chapters.
- Analyze each chapter with an AI to assign expressive voice tags.
- Synthesize each tagged paragraph with a TTS engine.
- Stitch the segments together.
- Master the audio with proper normalization and metadata.
Six steps. It sounded reasonable. It was reasonable in the way that disarming a bomb sounds reasonable when someone hands you the instruction manual.
The first script was born: src/segment_ebook.py. Simple and elegant, it split a Markdown document into chapter-sized chunks using header detection. The output was clean:
📁 segmented_output/
├── 1984 - George Orwell_CAPÍTULO_I.md
├── 1984 - George Orwell_CAPÍTULO_II.md
└── ...
Clean. Orderly.
A false sense of security had been established.
👻 Chapter Two: The Ghost in the Prompter
The LLM was the first co-conspirator to be recruited.
Gemini would serve as the Script Adaptor. It would read raw Spanish prose and transform it into a tagged script, annotating every paragraph with voice directives:
[voice: narrator (model: es-CO-SalomeNeural)] [neutral] Era un día luminoso y frío de abril...
[voice: Winston (model: es-AR-TomasNeural)] [enojado] —No puedo seguir haciendo esto.
The first version of this script, reliable_prosody.py, was a marvel of prompt engineering. The system instruction was a carefully structured directive, half film director and half casting agent:
system_prompt = f"""
You are an expert Voice Actor and Script Adaptor for "{book_title}".
Rewrite the provided paragraph into a tagged script for the MioTTS engine.
STRICT VOICE ASSIGNMENTS (YOU MUST FOLLOW THESE):
{chr(10).join(assignments)}
EMOTIONAL PROSODY:
Available Emotions: neutral, feliz, triste, enojado, susurro, grito, miedo, autoritaria.
"""
It worked beautifully. For a while.
There was a fundamental problem hiding in the architecture, one that would not reveal itself for weeks.
🔎 The Evidence
Exhibit A: Chapter 1. A minor character named Parsons appears. Gemini assigns him
es-CL-LorenzoNeural, a deep, working-class Chilean accent. Perfect.Exhibit B: Chapter 6. Parsons reappears. Gemini has forgotten him entirely. He is now
es-VE-SebastianNeural, a Venezuelan voice with a completely different timbre.
The same character. Two different voices. Two different souls.
In a live recording studio, a director would catch this instantly. In an automated pipeline, the inconsistency lived silently inside the audio files for days. It surfaced only during a quality review of Chapter 10, when Winston's neighbor suddenly sounded like he'd had an accent transplant.
🗂️ Case Note: The clue had been there all along. The LLM's context window ended at the chapter boundary. Everything it knew about Parsons died there.
🧠 Chapter Three: The Memory Vault
The solution arrived not as a thunderbolt but as a quiet architectural realization.
A file. A simple JSON file. It would serve as the pipeline's long-term memory, a voice casting ledger that persisted across every chapter boundary.
voice_memory.json:
{
"Parsons": {
"role": "supporting_character",
"voice": "es-CL-LorenzoNeural"
},
"Winston": {
"role": "main_character",
"voice": "es-AR-TomasNeural"
}
}
But simply having a memory file was not enough. Someone, or something, had to fill it.
The old update_memory function was, in a word, empty. A ghost. It had been written with good intentions:
def update_memory(tagged_text, memory):
"""
LIMITATION: The current tag format doesn't explicitly link
"Character Name" to "Voice".
For now, we rely on the improved INPUT memory.
"""
pass
pass.
One of the most haunting words in programming. It means: I know this needs to be done. I don't know how to do it yet.
💡 The Breakthrough
The breakthrough came from changing the instructions given to the LLM itself. Instead of asking for generic, role-based tags:
[voice: supporting_character (model: es-CL-LorenzoNeural)]
The LLM was now instructed to use the character's actual name:
[voice: Parsons (model: es-CL-LorenzoNeural)]
Now the memory function had something to parse. A regex became the cipher key:
def update_memory(tagged_text, memory):
pattern = re.compile(r"\[voice:\s*([^\(]+?)\s*\(\s*model:\s*([^)]+)\)\]")
updated = False
for char_name, model_name in pattern.findall(tagged_text):
char_name = char_name.strip()
if char_name.lower() in ["narrator", "supporting_character", "main_character"]:
continue
if char_name not in memory:
memory[char_name] = {"role": "supporting_character", "voice": model_name}
updated = True
return updated
Every paragraph was now processed like evidence at a crime scene. The moment a new character name appeared in a voice tag, they were captured, catalogued, and added to the permanent record.
The system had evolved from amnesia to eidetic memory.
🎙️ Chapter Four: The Three Voices of the Machine
By mid-production, the pipeline had assembled an unlikely cast of three distinct text-to-speech engines. Each had its own personality, its own strengths, and its own dark secrets.
Microsoft Edge-TTS: The Workhorse
Fast, reliable, and rich with regional Spanish accents that no other provider could match. It spoke through a direct WebSocket connection. No proxy, no intermediary.
Google Gemini GenAI: The Oracle
High-fidelity prebuilt voices (Aoede, Charon, Fenrir, Kore, Puck) that conveyed emotion with remarkable nuance. The tradeoff was the loss of regional accent authenticity.
OmniVoice: The Phantom
A neural TTS model hosted locally on the developer's own GPU at http://localhost:5002. Its unique power is instruct-based voice design, the ability to describe a voice in a plain-English taxonomy:
OmniVoice-female, young adult, low pitch, whisper
OmniVoice-male, elderly, very low pitch
Instead of choosing from a fixed catalog of named voices, you could design the voice from its constituent attributes.
⚠️ The Catch
OmniVoice spoke English.
When presented with Spanish text, it would occasionally look at the words, run its internal language detection, and confidently decide: "This is English. I will use my English phonetic dictionary."
The result was a narrator who sounded like a Midwesterner attempting to speak Spanish at gunpoint.
🕵️ Chapter Five: The Language Detective
The OmniVoice language-fallback bug followed a chilling pattern.
Short phrases with no Spanish-specific characters, like "No.", "Sí.", and "Ven.", triggered the English detector, because they could be English-adjacent. Nothing in the text said "this is Spanish," so the engine guessed. It guessed wrong.
The solution was elegant: inject Spanish DNA into ambiguous phrases.
# OmniVoice language auto-detection mitigation
if len(omni_text.split()) <= 5 and not any(c in omni_text for c in '¡¿áéíóúñÁÉÍÓÚÑ'):
omni_text = f"¡{omni_text}!"
The Spanish-exclusive inverted exclamation mark, ¡, became the password. By wrapping short, ambiguous phrases in ¡ !, the pipeline gave OmniVoice an unmistakable linguistic fingerprint.
It worked.
🗂️ Case Note: When a model has to guess, don't make it guess. Hand it a signal it can't misread.
🪦 Chapter Six: The Forgotten Server
Among the most significant discoveries was the realization that the pipeline had been carrying dead weight from the very beginning.
MioTTS, a local Python microservice on port 5000, had served as a proxy between the pipeline and its TTS providers.
- It had to be started manually before any synthesis could begin.
- It added latency to every call.
- It complicated debugging.
Most critically: it was unnecessary.
The forensic audit revealed that every provider could be reached directly:
# Edge-TTS: Direct WebSocket
communicate = edge_tts.Communicate(text, voice=voice_name)
await communicate.save(str(output_path))
# Gemini: Direct SDK
response = genai_client.models.generate_content(...)
# OmniVoice: Direct REST
resp = await self.client.post("http://localhost:5002/v1/audio/speech", json=omni_payload)
The server was mourned briefly, then forgotten.
🗺️ Chapter Seven: The Map of Code
Every great investigation benefits from a map. For this project, the map came from Graphify, a tool that ingested the entire codebase and generated a navigable knowledge graph.
The initial graph was revealing:
| Metric | Value |
|---|---|
| Nodes | 199 |
| Edges | 226 |
| Communities | 32 |
The God Nodes, ProductionProcessor, CharacterDirector, and TTSRouter, were the spine of everything.
But the graph also exposed ghost communities: "Goldstein Analysis & Voice Tests" and "Regional Voice Samples." These were phantom nodes from code that had been deleted but not yet purged from the graph's memory.
The graph needed a purge and a rebuild:
Remove-Item -Path "graphify-out" -Recurse -Force
graphify extract .
| Metric | Before | After |
|---|---|---|
| Nodes | 199 | 60 |
| Edges | 226 | 88 |
| Communities | 32 | 9 |
Every node now represented something that actually existed.
⛓️ Chapter Eight: The Path of Hardcoded Sins
During the forensic cleanup, a disturbing pattern emerged.
Everywhere in the codebase, in every corner, hard evidence of one specific book had been embedded like original sin:
"title": f"1984 - Capítulo {ch_num}",
"artist": "George Orwell",
self.role_map = {
"Parsons": "supporting_character",
"Julia": "secondary_character",
"O'Brien": "secondary_character",
}
The pipeline had been written for 1984, not for any book.
Every hardcoded value was traced and replaced with dynamic resolution:
# Before: Hardcoded
"title": f"1984 - Capítulo {ch_num}",
"artist": "George Orwell",
# After: Dynamic
default_title = project_root.name if project_root else book_path.stem
book_title = metadata.get('title', default_title)
book_artist = metadata.get('artist', 'Unknown Artist')
When the last hardcoded string was removed, the pipeline finally became what it had always been meant to be: a book-agnostic production system.
🔐 Chapter Nine: The Architecture of Isolation
Audio files from every synthesis test had piled up in a single flat directory on the D: drive. Chaos. Chapter 1 sat beside OmniVoice experiments. The distribution folder was one shared container for everything.
The solution was structural isolation. Every book would receive its own territory on the D: drive, identified by its unique slug:
D:/generated_audiobooks/
└── 1984/
├── chapter_1984_CAPÍTULO_I/ # Raw segments
└── distribution/ # Final masters
# Dynamic slug resolution
if self.project_path:
self.audio_output_dir = AUDIO_OUTPUT_DIR / self.project_path.name
The --project argument now served double duty: it isolated the pipeline's source files within the repository and the audio output on the D: drive. The slug was extracted from the project path automatically.
Future books would each get their own vault. Their audio would never commingle.
⚖️ Chapter Ten: The Phase Separation Protocol
The single most consequential architectural change came from a deceptively simple observation.
The pipeline processed the book chapter by chapter, with each chapter running a complete cycle of Prosody → Synthesize → Assemble → Master. This worked. But it contained a fatal flaw.
The memory was being built and consumed in the same pass. A character introduced in Chapter 3 would be on file. But a character who first appeared in Chapter 8 hadn't been catalogued yet when the earlier chapters were already being recorded. The ledger was still being written while the studio was already rolling.
The solution: decouple prosody from synthesis entirely.
parser.add_argument("--prosody-only", action="store_true",
help="Only generate prosody tags, skip synthesis")
parser.add_argument("--synthesis-only", action="store_true",
help="Skip prosody tagging, only run synthesis/assembly/mastering")
The production workflow became a two-phase protocol:
# Phase 1: Tag the ENTIRE BOOK first (memory compounds across all chapters)
python src/orchestrate_book.py --book "source/Book.md" --project "book_conversion/Book" --prosody-only
# Review the prosody files...
# Phase 2: Synthesize with complete character memory
python src/orchestrate_book.py --book "source/Book.md" --project "book_conversion/Book" --synthesis-only
By the time synthesis began, voice_memory.json contained every character who appeared anywhere in the book, with their voices locked in permanently.
The mystery of the shifting accents was solved.
🌅 Epilogue: The Pipeline Lives
On the final day of the investigation, the codebase was clean. The documentation was current. The voice memory system was operational. The phase separation protocol had been tested and committed. The knowledge graph had been rebuilt from a clean slate.
The pipeline was ready.
Not for 1984. For any book. Any story. Any voice.
The output arrives in D:/generated_audiobooks/<slug>/distribution/, chapter by chapter, normalized to −20 dBFS, tagged with proper ID3 metadata, and formatted as MP3 and Opus for maximum compatibility. Every character speaks with their assigned voice, from Chapter 1 and never wavering through to the final page.
The voice cipher had been cracked.
📝 Lessons from the Case File
If you take only five things from this story:
- LLMs forget at the context boundary. If something has to persist, write it to disk. A simple JSON file beat any amount of prompt cleverness.
- Make your model's output parseable. Switching from role-based tags to character names turned an empty
passinto a working memory system. - Don't let a model guess. One
¡was enough to stop a TTS engine from mistaking Spanish for English. - Audit your middlemen. The proxy server added latency and complexity, and nothing else.
- Separate phases when state compounds. If step B needs the complete output of step A, don't run them in the same loop.
🔭 Open Case: What's Next
Like any good mystery, the final pages reveal that the case is never truly closed.
Phase 6 is already underway.
- OmniVoice returns inconsistent output formats.
- Its non-verbal expression tags,
[laughter],[sigh], and[dissatisfaction-hnn], remain untested in Spanish. - Sixteen voice profiles are waiting to be validated.
The developer closed the terminal. The hard drive kept humming.
Somewhere in the silence, a new book was waiting to be read.
— End of Case File —
About the author
- Solutions, Workarounds, Challenges, Workflows while working with Artificial Intelligence.
Comments
Join the discussionNo comments yet. Be the first to comment.



