A Voice Agent Is Mostly Not the Model_

The demo is easy. A model answers the phone, sounds human, books an appointment. Everyone nods.

The system behind that demo is where the work is, and almost none of it is the model. I learned this building Bryy, a voice and SMS automation platform for home services businesses that qualifies leads, books appointments, and handles customer messaging.

The model is the most replaceable part

Bryy routes language model calls across Groq, Together, and OpenRouter. Speech to text is Deepgram. Text to speech is ElevenLabs. Telephony runs through Telnyx or Twilio.

Every one of those is behind an interface, because every one of them will change. Providers deprecate models, adjust pricing, or have a bad afternoon. Treating any of them as load bearing is a mistake you make once.

What does not change is everything around them: who this caller is, what they already told you, what the business actually offers, whether that slot is free, and what happens when the call drops halfway through booking. That is the system. The model is a component inside it, and the honest framing is that you are building a state machine that occasionally consults a language model, not an AI product with some plumbing attached.

Conversation state is the hard problem

Text and voice are the same conversation to a customer and completely different problems to a program.

SMS is asynchronous and durable. A lead replies four hours later, mid-thread, having forgotten what you asked. Voice is synchronous and unforgiving. Latency is a felt experience, silence reads as a broken call, and there is no scrollback.

Both have to write into the same view of the lead, because a customer who texts and then calls is one person with one history. Getting that wrong produces the specific failure that destroys trust in these systems: the agent asking a question it already has the answer to.

Postgres with Drizzle holds the durable record. DragonflyDB handles the fast path where per-turn latency matters. The split is deliberate. The conversation is the source of truth and it is not allowed to live only in a cache.

Multi-tenant, with the operator as a first-class user

Bryy runs a database per client rather than a tenant column on shared tables.

The reason is blunt: this data is customer phone numbers, call recordings, and transcripts belonging to somebody else’s business. A missing WHERE tenant_id in a query is not a bug, it is a disclosure. Separate databases make that class of mistake structurally impossible rather than something a reviewer has to catch.

The part I underestimated was the operator surface. Running this platform is not just running the tenants. It needs its own dashboard: tenant onboarding, a prompt template library with version control, service tree configuration, platform-wide logs, and the ability to impersonate a tenant to debug what a client is actually seeing.

Prompt versioning deserves emphasis. Prompts are configuration that changes behavior in production, which makes them closer to code than to content. Editing a live prompt with no version history is deploying to production with no rollback. Every prompt change is versioned, and the operator can see which version a given conversation ran under.

What holds up

The architecture is Bun, Hono, and GraphQL through Yoga and Pothos, over Postgres and Drizzle, with JWT auth using refresh token rotation.

None of that is exotic, and that is the point. The interesting risk in this system is concentrated in telephony, conversation state, and tenant isolation. Spending novelty budget on the framework layer would have been spending it in the wrong place.

If you are building something similar, the advice is short. Assume every model and vendor is temporary. Decide where conversation state lives and let nothing else claim to own it. Isolate tenants structurally rather than by convention. And build the operator tooling early, because you will spend more time in it than any single customer will spend in theirs.