Voice AI is hard. Here's what nobody tells you.

Everyone's launching a voice AI product right now. The demos look incredible. A smooth, natural-sounding agent handles a customer query, cracks a joke, resolves the issue in 30 seconds. The audience claps. The LinkedIn post gets 10,000 impressions.

Then someone actually deploys it.

We've been building voice AI at CloudInteract for the past few years, shipping it into real contact centres handling real calls with real customers who are often frustrated, confused, or in a hurry. And I can tell you, the gap between a voice AI demo and a voice AI product is enormous.

Here's what we've learned.

---

Latency kills the illusion

Humans are absurdly sensitive to conversational timing. A pause of 300 milliseconds feels natural. A pause of 800 milliseconds feels like the other person has checked out. Most voice AI architectures chain together speech-to-text, an LLM, and text-to-speech. Three separate round trips, each adding latency. By the time your agent responds, the caller has already said "hello?" twice and is reaching for the zero key.

We built Nova 2 Sonic as a speech-to-speech system specifically to collapse that pipeline. No intermediate text step. The model processes audio directly and generates audio directly. It sounds simple on paper. It was anything but.

---

Accents, dialects, and the real world

Most speech models are trained predominantly on American English. That's a problem when your caller is from Glasgow, or Mumbai, or rural Wales. Background noise makes it worse. A caller ringing from a hospital ward, a building site, a car with the windows down. These aren't edge cases. In a contact centre, they're the majority of calls.

Getting accuracy right across the full spectrum of how people actually speak, in the conditions they actually call from, requires continuous tuning. There's no "set and forget" with voice AI. If you're not constantly feeding real-world audio back into your models, accuracy degrades fast.

---

Emotion isn't optional

Text-based AI can get away with ignoring tone. Voice can't. A customer saying "that's fine" in a flat monotone means the opposite of "that's fine" said with relief. If your voice agent can't detect frustration, sarcasm, or distress, it will blunder through sensitive moments with the emotional intelligence of an automated car park barrier.

We've invested heavily in emotional detection. Not as a nice-to-have, but as core infrastructure. An AI agent that escalates to a human when it detects rising frustration is infinitely more useful than one that cheerfully ploughs through a script while the caller gets angrier.

---

Context is everything

Voice conversations aren't stateless. Callers reference things they said 30 seconds ago, or in a previous call last week. They change their mind mid-sentence. They interrupt. They go on tangents and then say "anyway, what I was saying was..."

Handling this requires more than a big context window. The model needs to understand conversational structure: what's a digression, what's a correction, what's new information versus a restatement. We've found this is where most off-the-shelf solutions fall apart. They can handle a clean, linear conversation. Real calls are rarely clean or linear.

---

Integration is the unsexy bit that matters most

A voice agent that can have a nice conversation but can't actually do anything is a parlour trick. The hard work is connecting it to your CRM, your knowledge base, your booking system, your identity verification, and doing it fast enough that the caller doesn't notice. In contact centres built on Amazon Connect, we have an advantage because we're building natively on AWS. But even then, every integration is its own project. Every data source has its own quirks.

---

So why bother?

Because when it works, the economics of customer service change completely.

We're seeing 40%+ of enquiries resolved without a human agent. Not deflected. Resolved. Callers get answers faster. Agents handle the complex stuff they're actually good at. Costs drop by 30-50%.

But getting there requires real engineering, not just prompt engineering. It requires understanding telephony, not just transformer architectures. And it requires an obsession with the messy, unglamorous reality of how people actually communicate.

If someone tells you voice AI is easy, they haven't shipped it yet.

---

We're building voice AI at CloudInteract that works in the real world, not just on stage. If you're thinking about AI in your contact centre, drop me a message.