Building an AI phone receptionist a multi-location service business can actually answer calls with

*Can a voice agent answer the front-desk phone for a business with many locations, book real appointments, and fail safely — without the business finding out it failed from a one-star review?*

Missed calls are the cleanest problem in a service business. The caller has intent, a budget and a vehicle, a boiler or an appointment window. Nobody picks up. They call the next name on the list. The revenue is gone and nothing in the CRM records that it existed.

Multi-location makes it worse. Calls land on a shared number, get routed by an IVR nobody listens to, and a branch that is quiet cannot take overflow from a branch that is drowning.

Latency is the whole product. A voice agent that is correct but slow feels broken. Humans start to feel the gap at about 500ms of silence and start talking over the agent at about 1 second. Every architectural decision below is a latency decision first and a quality decision second.

The rejected alternative: one big model call. The obvious build is speech-to-text, one large model prompt containing the full booking logic, then text-to-speech. It is a two-day prototype and it fails in production. Time-to-first-token on a long prompt with tool definitions pushes past the interruption threshold, and the model happily invents availability that does not exist. We rejected it in Blueprint.

What broke in Prototype: barge-in. Callers interrupt. Early builds kept generating audio while the caller was talking, so the agent and the human spoke over each other and the transcript became unusable. Fixing it meant real voice-activity detection with an endpointing policy tuned per language, plus the ability to cancel in-flight generation and discard buffered audio mid-sentence. That is unglamorous plumbing and it took longer than the booking logic.

Accents, noise and code-switching. Service-business callers phone from workshops, roadsides and building sites. Transcription word error rate on a quiet studio recording tells you nothing about performance on a hands-free call in traffic.

1. Blueprint — scoped the agent down to bookings, hours, status and transfer. The agent does four things: book an appointment, answer opening hours and location questions, give the status of an existing job, and transfer to a human. Everything else is an explicit transfer. Scope discipline is what makes a voice agent evaluable — an agent with unbounded scope has no measurable failure rate because you cannot enumerate what it is supposed to do.

2. Prototype — built the call loop before the intelligence. We built the streaming loop first: telephony in, transcription streaming, partial results, interruption handling, audio out. No booking logic at all — the agent only read back what it heard. This exposed the latency and barge-in problems in week two instead of week eight.

3. Forge — split the model work into a fast path and a slow path. The turn-by-turn conversation runs on a small, fast model with a short system prompt and a tightly-scoped tool list. Anything requiring real reasoning — ambiguous scheduling, a caller describing an unusual problem — escalates to a larger model, and the agent covers the extra latency with natural speech ("let me check that for you") rather than silence.

4. Harden — made every write operation idempotent and auditable. Bookings go through a single idempotent endpoint keyed on call ID plus slot. A dropped call mid-booking cannot create a duplicate. Every call produces a stored transcript, the tool calls made, and the reasoning trace — because the first serious question anyone asks about an AI receptionist service is "what did it say to my customer".

The call never leaves a streaming path. Nothing waits for a complete utterance before starting work.

flowchart TD
    A["Inbound call (SIP / PSTN)"] --> B["Telephony gateway<br/>media streaming"]
    B --> C["VAD + endpointing<br/>barge-in control"]
    C --> D["Streaming ASR<br/>partial transcripts"]
    D --> E{"Turn router"}
    E -->|"routine turn"| F["Fast model<br/>short prompt, 4 tools"]
    E -->|"ambiguous / complex"| G["Reasoning model<br/>filler audio covers latency"]
    F --> H{"Tool call needed?"}
    G --> H
    H -->|"yes"| I["Tool layer"]
    H -->|"no"| K["Streaming TTS"]
    I --> I1["Branch directory<br/>hours, address, services"]
    I --> I2["Scheduling API<br/>idempotent booking"]
    I --> I3["Job status lookup"]
    I1 --> K
    I2 --> K
    I3 --> K
    K --> L["Audio out to caller"]
    E -->|"escalation triggered"| M["Warm transfer<br/>to branch or overflow desk"]
    M --> N["Human agent<br/>receives transcript summary"]
    F -.-> O[("Call log store<br/>transcript + tool trace")]
    G -.-> O
    I -.-> O
    M -.-> O

Three things in that diagram matter more than the model choice.

The turn router. It decides, per turn, whether this is a fast-path turn or a reasoning turn. Most turns — "which branch", "what time", "yes that works" — never touch the large model.

The tool layer is narrow and real. Three tools, each hitting a live system. The agent has no ability to state availability that the scheduling API did not return. That single constraint removes the most damaging failure mode in booking agents.

Everything writes to the call log. Transcript, tool calls, model chosen, escalation reason. This is the substrate for evaluation, dispute resolution and every improvement after launch.

These are engineering targets, not measured client results. We budget the round trip before writing code, then hold each stage to its allocation.

Reasoning-path turns run longer. We cover them with acknowledgement audio that starts within the routine budget, so the caller hears a response on time even when the answer arrives late. Silence is the thing that reads as broken; a spoken "one moment, checking that" does not.

Endpointing is the biggest single lever and the most dangerous one. Tighten it and the agent interrupts callers who pause to think. Loosen it and every turn feels sluggish. We tune it per deployment against recorded calls, not against a default.

An AI phone receptionist earns trust by knowing when to stop. We define escalation triggers explicitly, before launch, and each one is testable.

Two failed transcription turns. If the agent has asked for a repeat twice, it transfers. It never asks a third time.

Explicit request. Any phrasing resembling "speak to a person" transfers immediately, with no retention attempt.

Out of scope. Complaints, refunds, disputes, anything involving money already paid — straight to a human.

Distress or safety language. Immediate transfer with priority routing.

Tool failure. If the scheduling API errors or times out, the agent does not guess. It takes a callback number and transfers.

Low confidence on a booking write. If the agent cannot confirm the caller repeated back the correct slot, it does not write.

Every transfer is a warm transfer: the human receives a one-paragraph summary and the transcript before the call connects. A cold transfer that makes the caller repeat themselves destroys the value the agent just created.

Out of hours, when no human is available, the agent says so plainly and takes a callback. It never pretends someone is coming.

The agent does not take a real call until it passes an eval set. This is the part most automated phone answering deployments skip, and it is why so many of them get switched off in month two.

The eval set. 150–300 scripted call scenarios, built from the client's own recorded calls where available and from the branch staff's description of their worst days where not. Each scenario has a defined correct outcome: booking created with these attributes, or transfer triggered, or information given. Scenarios cover accents, background noise, mid-call interruptions, callers who change their mind, callers who name the wrong branch, and callers asking things deliberately outside scope.

The metric that mattered. Not transcription accuracy. Task completion rate with zero incorrect writes. A call is a pass only if the outcome matches the defined correct outcome *and* no wrong booking was created. We weight a wrong booking as far worse than a failed booking, because a customer who arrives to no appointment is a much larger loss than a customer who was transferred.

Latency percentiles, not averages. We hold p50 and p95 of response-start time, per turn. The average is meaningless — a single 4-second turn is what the caller remembers from an otherwise fast call. Regression on p95 blocks a release.

Failure modes and what happens when it is wrong. Three named failure modes, each with a defined containment:

*Mishears the branch.* Containment: the agent always reads back branch and time before writing, and the confirmation SMS names the branch in the first line.

*Books a slot that has just been taken.* Containment: the scheduling API is the source of truth, the write is conditional, and a rejected write triggers an immediate re-offer rather than a silent failure.

*Confidently answers an out-of-scope question.* Containment: narrow tool list plus an explicit refusal-and-transfer instruction. Caught in eval by adversarial scenarios that invite the agent to speculate.

Shadow mode before live. For 1–2 weeks the agent runs on real calls with its audio muted — it listens, decides, and logs what it would have said, while a human handles the call. We compare its decisions to what the human actually did. That comparison, not the eval set, is what tells you whether to go live.

For the wider argument about where voice AI for business pays off and where it does not, see AI agents for business, explained.

We would build the eval set in week one, not week five. On builds where evaluation came late, we shipped a prototype that felt impressive in a demo and then discovered a whole class of failure — callers naming a branch by its old name — that a proper eval set would have surfaced immediately. Demos select for the happy path. Eval sets do not.

We would instrument the human baseline first. Nobody knows their current miss rate, average handle time or booking conversion on the phone. Without that baseline the business cannot judge the agent, and the conversation drifts into anecdote after launch. One week of measurement before the build is worth more than an extra feature.

We would push harder on the confirmation SMS. It is the cheapest possible containment for every mishearing failure mode and it is always the last thing to get built, because it is not the interesting part.

We over-invested in voice quality early. Callers care much less about the naturalness of the voice than about whether it interrupts them and whether the booking is right. We would spend that time on endpointing.

It cannot handle complaints. It cannot negotiate. It cannot take payments without a separate, scoped, compliance-reviewed workstream. It will not fully replace a receptionist, and any virtual receptionist AI vendor promising that is selling a demo. What it reliably does is answer every call, handle the routine majority, and route the rest to a human with context attached.

What we are not claiming. We do not claim a revenue figure for this scenario. Recovered calls are not recovered revenue until they convert, and that conversion depends on the business, not on the agent.

Reference builds like this stay in our own environment so we can keep testing against new speech and language models as they ship. Current open work: multilingual endpointing, better handling of callers who give a postcode instead of a branch name, and reducing reasoning-path latency so the acknowledgement audio is needed less often.