TTS cache

The single biggest lever for making voice interviews feel responsive instead of chatty. Ear3 ships a two-tier TTS cache — RAM for sub-millisecond playback, persistent storage for popular phrases — that sits inline in the Pipecat pipeline and eliminates the LLM→TTS→audio latency on any phrase we’ve said before.

Ships in the worker image

This runs in production today inside pipecat-cloud-server. When we extract it as a standalone ear3-tts-cache package (Apache 2.0), you’ll be able to drop it into any Pipecat pipeline. Self-hosters already get it — it’s baked into the image.

Why this matters

A typical voice interview turn:

User stops speaking
   ├── VAD confirms utterance end          ~200 ms
   ├── LLM generates response text         ~600 ms  (main cost)
   ├── TTS renders audio                   ~400 ms  (main cost)
   └── First audio frame plays             ~50 ms
                                          ─────────
                                          ~1250 ms

Under a second is the conversational threshold — past it, users start interrupting the bot and the flow feels off. TTS eats a third of that budget. Cache the TTS output and you claw 400 ms back on any repeated phrase — which turns out to be most of them (opening greeting, “got it”, “can you tell me more”, question templates).

Architecture

Two tiers, one hash:

                    ┌──────────────────────────────┐
                    │  DIALOG START                │
                    └──────────────┬───────────────┘
                                   │
                                   ▼
                    ┌──────────────────────────────┐
                    │  PostgreSQL query            │
                    │  WHERE survey_id + voice_id  │
                    │  ORDER BY hit_count DESC     │
                    └──────────────┬───────────────┘
                                   │
                                   ▼
                    ┌──────────────────────────────┐
                    │  RAM warm-up                 │
                    │  HashMap<sha256, audio_url>  │
                    └──────────────┬───────────────┘
                                   │
              ┌────────────────────┴────────────────────┐
              │           LLM produces "…"               │
              └────────────────────┬────────────────────┘
                                   │
                                   ▼
                    ┌──────────────────────────────┐
                    │  SHA-256(text)               │
                    └──────────────┬───────────────┘
                                   │
                     ┌─────────────┴──────────────┐
                     │                            │
                     ▼                            ▼
        ┌────────────────────┐      ┌────────────────────┐
        │ RAM hit  ⚡          │      │ RAM miss            │
        │ ~0.5 ms → play      │      │ → check S3 cache    │
        └────────────────────┘      └─────────┬──────────┘
                                              │
                                    ┌─────────┴──────────┐
                                    │                    │
                                    ▼                    ▼
                        ┌────────────────────┐ ┌────────────────────┐
                        │ S3 hit             │ │ Full miss          │
                        │ ~40 ms → play      │ │ → Cartesia TTS     │
                        │ + warm RAM         │ │ ~400 ms → play     │
                        └────────────────────┘ │ + write S3         │
                                               │ + increment hit    │
                                               └────────────────────┘

RAM tier — plain HashMap, per-worker, warmed at dialog start with the top-N phrases for this survey + voice. Sub-millisecond lookup.
Persistent tier — S3 bucket keyed by sha256(text + voice_id). Each hit increments a Postgres counter; entries used ≥ 2 times in the last 7 days survive eviction.
Cache key — SHA-256 of text + voice_id (voice matters — Cartesia Sonic and ElevenLabs Turbo v2 sound different for the same text).

What gets cached

Everything the LLM produces, aggregated per-sentence. That covers:

Opening greeting — same on every session (“Hi, this is Ear3. I have a few questions…”)
Question templates — even personalised ones share phrase-suffixes (“Great, and now I’d like to ask about…”)
Backchanneling — “got it”, “makes sense”, “one more thing”
Closing — “Thanks for taking the time. Have a good day.”

In production on our largest workspace, cache hit rate is ~68 % on sessions past the first 100 — cold surveys have to build the cache, warm ones fly.

Metrics to watch

Metric	What it tells you
`tts_cache.ram_hits`	Best case — worker had the audio ready
`tts_cache.s3_hits`	Second best — S3 fetch adds ~40 ms
`tts_cache.misses`	Full TTS render — 400 ms latency, budget on Cartesia bill
`tts_cache.hit_rate`	`(ram + s3) / total`. Aim > 60 % on warm surveys
`tts_cache.eviction_rate`	If high, warm pool is too small or LRU is too aggressive
`tts_cache.write_latency`	S3 write should be non-blocking; alert if > 100 ms p99

Exported as OpenTelemetry via the same pipeline as the rest of the worker — see Self-host → Observability for wiring to your OTel collector.

Configuration (self-host)

The cache is on by default. To tune it, set these env vars on the worker:

env:
  # Disable entirely (not recommended — you'll notice)
  - { name: EAR3_TTS_CACHE_ENABLED, value: "true" }
 
  # RAM tier size per worker (default 500 entries)
  - { name: EAR3_TTS_CACHE_RAM_MAX_ENTRIES, value: "1000" }
 
  # S3 bucket for the persistent tier (required if enabled)
  - { name: EAR3_TTS_CACHE_S3_BUCKET, value: "your-tts-cache-bucket" }
  - { name: EAR3_TTS_CACHE_S3_REGION, value: "us-west-2" }
 
  # Persistence threshold (default: 2 hits in 7 days)
  - { name: EAR3_TTS_CACHE_MIN_HITS,   value: "2" }
  - { name: EAR3_TTS_CACHE_TTL_DAYS,   value: "7" }

Ear3-managed customers don’t touch any of this — we run it with tuned defaults across all workspaces.

Cost impact

Each cache hit is a Cartesia call not made. At Cartesia Sonic pricing, ~$0.03 per 1k characters, a workspace running 10k interviews/month with a ~65 % hit rate saves roughly $400–600/month in TTS spend, plus the harder-to-price latency win in end-user perceived quality.

For Ear3-managed workspaces this saving is already priced into your subscription. For self-hosters it drops straight to your Cartesia bill.

Self-host — how to run the worker (cache included)
Open source — the roadmap for extracting ear3-tts-cache as its own package
pipecat-cloud-server / tts_cache_simple — the current source

Open source