How to build a voice AI agent with Claude

Most "AI agent" demos are a chat window bolted onto a language model with a clever prompt. That's fine for a demo. It falls over the moment someone asks it to actually do something — check a calendar, look something up, take an action — because there's no real system behind the words.

This is the architecture we actually run in production, on this page right now. Ask the assistant in the bottom-right corner what times are open this week — it's not roleplaying an answer, it's calling a real API against a real calendar.

01The three pieces

A voice AI agent that does real work is three separate systems wired together, not one big prompt:

• Reasoning — Claude, via the Anthropic API. This decides what to say and, critically, when to call a tool instead of just talking.
• Voice — a text-to-speech engine (we use ElevenLabs) that turns Claude's reply into audio, with a graceful fallback to the browser's built-in speech synthesis if the real voice call ever fails.
• A backend — a server between your frontend and Claude. This is not optional. More on why in section 3.

02Step by step

1. Get API access

Create an API key at console.anthropic.com. Keep it server-side from the very first line of code — never in anything that ships to a browser.

2. Write the system prompt like a job description, not a personality

The system prompt should state, in priority order, what the agent is actually for. Ours reads roughly:

1. EDUCATE — answer questions in plain language
2. QUALIFY — learn what the visitor actually needs
3. PUSH — once they're a real fit, move toward a
   concrete next step using real tools, not a link

Vague personas ("be friendly and helpful!") produce vague agents. A prioritized job description produces one that knows what to do next.

3. Give it tools for anything that needs to be real

Claude's tool-use (function calling) lets the model request that your backend run a specific function — check a calendar, create a record, send a notification — then continues the conversation with the real result. This is the difference between an agent that describes booking you an appointment and one that actually does it.

const CHECK_AVAILABILITY_TOOL = {
  name: 'check_availability',
  description: 'Get real open appointment slots.',
  input_schema: { type: 'object', properties: {} }
};

Two things bite people here:

• Don't trust the model's memory of a tool result across turns. If your frontend only stores the visible text (not the raw tool output), the model can misremember or invent a value on a later turn. Validate on the backend: if a value the model wants to act on doesn't match something a tool actually just returned, reject it and make the model re-fetch, instead of acting on a hallucinated value.
• Tell the model what it doesn't know about its own product. Claude has no idea your frontend has a speaker icon unless the system prompt says so — left unstated, it will confidently tell visitors "I'm text-only," even while your TTS pipeline is live three lines of code away.

4. Put a backend between the browser and Claude

This is the one non-negotiable. A serverless function (we use Azure Functions) receives the conversation from the browser, calls Claude with your API key attached server-side, runs any tool calls, and returns only the reply. The API key never reaches the client. This also gives you one place to rate-limit, log, and cap cost.

5. Add the voice

A second small backend endpoint takes text and returns audio from your TTS provider — again, server-side, so that key stays private too. Play it with a plain Audio element, and catch failures:

try {
  const res = await fetch('/api/tts', { method: 'POST', body: ... });
  const audio = new Audio(URL.createObjectURL(await res.blob()));
  await audio.play();
} catch {
  // fall back to the browser's built-in voice — never let a
  // voice-provider outage take down the whole chat experience
  speechSynthesis.speak(new SpeechSynthesisUtterance(text));
}

6. Add speech input (optional, and free)

The browser's SpeechRecognition API handles microphone-to-text entirely client-side, no API call needed. It's not available in every browser, so feature-detect it and hide the mic button if it's missing rather than showing a broken control.

7. Test the failure paths, not just the happy path

What happens when the TTS provider is down? When the model calls a tool with a malformed argument? When someone asks it something entirely off-topic? An agent that only works when everything goes right isn't production-ready — it's a demo with better lighting.

03Mistakes that actually break these in production

• API keys shipped to the browser. If your frontend calls Claude or your TTS provider directly, your key is sitting in plain text in the network tab of anyone's dev tools. Always proxy through your own backend.
• No cost ceiling. A public chat endpoint with no rate limit is an open invoice. Cap message length, add basic abuse throttling, and keep-warm your backend deliberately rather than pinging your own paid LLM endpoint on a timer.
• Tools that trust the model completely. Validate everything a tool call is about to act on — especially anything with a real-world side effect, like creating a calendar event or sending an email — against ground truth your backend just fetched, not just what the model claims.
• No fallback for the voice layer. Treat TTS as an enhancement, not a dependency. If it fails, the agent should still be fully usable as plain text.

04This is the one running on this page

Everything above is the actual architecture behind the assistant on cciaiml.com — right down to the tool-use guard against hallucinated booking times, which we found and fixed by watching it fail in production. It checks a real Microsoft Bookings calendar, creates a real confirmed appointment, and emails our team the moment it happens.