Results
| Metric | Before | After | Window and source | |---|---|---|---| | After-hours calls missed | Voicemail (no agent coverage) | 0 | Since go-live · every after-hours call is answered by the agent on first ring, a property of routing calls to the agent instead of voicemail | | Scheduling coverage | Business hours only | 24/7 | Agent books against live EHR availability outside staffed hours | | Turn-taking latency | n/a | Target: p50 300–500 ms · p95 under 1.2 s (engineering budget — measured production percentiles pending export, see below) | Measured end of caller speech → first audio frame of agent reply | Share of inbound calls handled end to end by the agent is not in this table. An earlier draft of this page showed 60% against a total inbound call count, site count and window we had not sourced — a percentage with an undisclosed denominator, which we treat as the same category of error as an unsourced revenue figure. It will be added, with its denominator, once the telephony platform's CDR export is pulled and confirmed.
How we measured. "Handled end to end" would mean the call reached a terminal intent — appointment booked, moved, cancelled, or question answered — with no transfer to a human and no callback ticket created, and a call that transferred at any point would count against the figure even if the agent did most of the work. We are stating the definition now so the number can be checked against it once it is published. Latency is instrumented in the media pipeline, not estimated: we timestamp the final ASR endpoint event and the first TTS frame written to the RTP stream, and export percentiles per call leg.
On "sub-second." The previous version of this page claimed sub-second turn-taking. That is a p50 claim dressed as a guarantee. The table above states our standard design budget for this class of voice system — p50 300–500 ms, p95 under 1.2 s, the same order of magnitude we hold on other voice builds — as an engineering target, not a measured result for this deployment specifically, and not a figure drawn from this engagement's own Blueprint or tuning records. It should not ship again as a client-result claim without the percentile export attached, because the number that matters to a caller is p95 — the tail is what makes an agent feel broken. We are publishing the measured fields empty until that export is signed off.
What we are not claiming. No revenue figure. No claim about appointment show-rates or clinical outcomes. We do not claim the agent improved care. It answered the phone.
The question this engagement had to answer
*Can a voice agent take routine scheduling off the front desk without ever being the thing that mishandles a sick patient?*
That is one question, not two. Any voice AI agency can demo a booking flow. The engineering problem in healthcare is the asymmetry: a mis-booked appointment costs a phone call, and a mis-triaged chest pain costs everything. The system had to be aggressive about automating the first and cowardly about the second.
What was hard
The hard part was not the conversation. It was the write. Reading availability from an EHR is straightforward. Writing an appointment into one, from a system that is not the EHR's own front end, while a receptionist may be editing the same slot in a browser tab, is where builds like this fail.
Names and dates of birth break ASR. Patient verification is exactly the part of the call with the highest word-error rate and the lowest tolerance for error. Streaming ASR tuned for conversational English is bad at surnames outside its training distribution and worse at "nineteen fifty-eight" versus "nineteen fifty-nine."
The rejected alternative. We were asked to consider an off-the-shelf voice agent platform with an EHR connector. We rejected it. The connector wrote appointments through a batch sync with no conflict semantics, which means a double-book is discovered minutes later by a human. We were not willing to make a receptionist the error handler for a system sold as removing work from receptionists. That decision cost real weeks of build time — building a conflict-safe write path from scratch instead of wiring up a connector — and is the single reason the write path is safe.
What broke in testing. Barge-in. Our first implementation cut the agent's audio the moment voice activity was detected, which meant a cough or a background TV truncated the agent mid-sentence and the caller lost the slot options they were being read.
Architecture
flowchart TD
A["Inbound call — SIP trunk"] --> B["Media gateway<br/>RTP in / RTP out"]
B --> C["Streaming ASR<br/>partial + final transcripts"]
C --> D["Turn manager<br/>endpointing, barge-in, timers"]
D --> E["Orchestrator<br/>intent + state machine"]
E --> F{"Intent class"}
F -->|"Book / move / cancel"| G["Scheduling tool"]
F -->|"Clinical question or<br/>urgency keyword"| H["Escalation controller"]
F -->|"Billing, records, other"| H
F -->|"Low ASR confidence<br/>on identity fields"| H
G --> I["FHIR R4 client"]
I --> J[("EHR<br/>Slot / Appointment / Patient")]
I --> K[("Event log — Postgres<br/>append-only, per-call")]
H --> L{"Staffed hours?"}
L -->|Yes| M["Warm transfer to front desk<br/>with transcript + intent"]
L -->|No| N["Urgency script → after-hours line<br/>or on-call callback ticket"]
E --> O["Streaming TTS"]
O --> B
D -.->|"barge-in signal"| O
E --> K
H --> KThe turn manager is a separate component from the orchestrator on purpose. Timing decisions — when the caller has stopped speaking, whether to duck the agent's audio, how long to wait before a filler — are latency-critical and must never wait on an LLM call.
The write path and conflict handling
sequenceDiagram
participant A as Voice agent
participant F as FHIR client
participant E as EHR
A->>F: Hold slot — Slot.status busy-tentative
F->>E: PUT Slot (If-Match: ETag)
E-->>F: 200 OK
A->>A: Read options to caller, confirm
A->>F: Create Appointment
F->>E: POST Appointment (idempotency key = call_id + slot_id)
alt 409 / 412 — slot taken since read
E-->>F: Conflict
F->>A: conflict signal
A->>A: Re-read availability, apologise, offer next slots
else Success
E-->>F: 201 Created
A->>A: Read back confirmation, end turn
endEvery write is conditional on the ETag we read, and every appointment POST carries an idempotency key derived from the call ID and slot ID. A retried write cannot produce a second appointment. A slot taken by a human between our read and our write produces a conflict the agent recovers from inside the same call, in one turn, rather than a double-book someone finds later.
What we did
1. Blueprint. Two weeks mapping every inbound call reason against a decision: automate, escalate, or refuse. Anything with a clinical decision in it went to refuse before a line of code was written. This is where the scope and the price were locked.
2. Prototype. A single-intent booking agent against a sandbox FHIR server, tested on recorded call audio rather than clean studio reads. The prototype existed to prove the write path and the latency budget, not the conversation.
3. Forge. Full intent set, the escalation controller, the conflict-safe write path, and the identity-verification flow. Identity fields — surname, date of birth — use constrained ASR: a restricted grammar, digit-by-digit confirmation for dates, and phonetic read-back of surnames. Below a confidence floor on any identity field, the agent stops guessing and transfers.
4. Harden. Barge-in rebuilt to require sustained speech energy plus an ASR partial before ducking, which killed the cough-truncation failure. Load tested against a target of 50–100 concurrent calls — our standard concurrency ceiling for a multi-site voice deployment of this kind, not a number drawn from this engagement's own Blueprint or load-test documents, and not a claim about this group's observed production peak. Every call leg logged append-only with transcript, intent, tool calls and latency percentiles.
The compliance section, stated properly
The previous version of this page led with "HIPAA-ready." A healthcare CTO reads "HIPAA-ready" as "not attested," and they are right to. HIPAA is not a product feature and there is no certificate. Either you can describe your control set, your BAA position and your audit posture, or you should not lead with compliance at all.
What we can state about this build:
PHI in the call path. Name, date of birth and appointment reason are PHI. They are handled in memory during the call and written to an append-only event log with field-level encryption. Raw call audio retention — whether audio is retained at all, and for how long — is a client configuration decision we are not publishing until it is confirmed in writing.
Subprocessors. ASR, LLM and TTS vendors each touch PHI. Each requires an executed Business Associate Agreement. We are not listing which BAAs are executed, with which vendors, or their effective dates until each is confirmed against the signed agreements. If a vendor will not sign a BAA, it does not belong in a healthcare call path, regardless of how good its latency is.
Access and audit. Every call produces an immutable event record — transcript, tool calls, escalation reason, operator identity on transfer. Access to that log is role-scoped and logged.
Attestation. We are not claiming a third-party assessment, SOC 2 report or HIPAA Security Rule risk analysis exists until one is confirmed. Until then, this page will not use the word "compliant."
We would rather publish a page with three bracketed gaps than one adjective that fails a security review.
How we evaluated it
The eval set. Transcripts from real inbound calls, de-identified, sampled across call reason, time of day and site — target 150–300 calls, our standard eval-set sizing target for a voice-intent build of this kind, not a number drawn from this engagement's own Blueprint documents, and not a count of calls the agent has since handled. It was constructed to over-represent the hard cases: non-native accents, background noise, callers who change their mind mid-booking, and calls containing urgency language. A clean-audio eval set would have told us nothing.
The metric that mattered. Not intent accuracy. Escalation recall on clinically urgent calls — the share of calls containing urgency signals that the agent handed to a human. The target was 100% and the tolerance was zero. Booking accuracy was the second metric; identity-verification error rate the third.
Latency. p50 and p95 turn-taking, measured per call leg as described above, plus tool-call latency against the EHR isolated separately — a slow FHIR read is indistinguishable from a slow model to a caller, and the fixes are completely different.
Failure modes, and what happens when the system is wrong.
*Mis-triage of clinical urgency.* The controlled failure. Urgency detection is not left to the model's judgement alone: a keyword and phrase layer runs in parallel with intent classification, and either one firing escalates. The system is tuned to over-escalate. A patient transferred unnecessarily has a slightly longer call. The inverse is unacceptable, so we deliberately pay for it in false positives.
*ASR error on identity.* Confidence floor breached → transfer, not guess. The agent never books against an unverified patient.
*Write conflict.* Recovered in-call, as above.
*Model or vendor outage.* The agent fails to the previous behaviour — ring the front desk during hours, the existing after-hours line outside them. It never fails to silence.
*Caller simply wants a human.* Honoured immediately, at any point, without a retention script. This was a client requirement and it is the right one.
What we would do differently
We would have built the escalation controller first. We built the booking flow first because it demos well. The escalation path is the part that carries the risk, and it deserved to be the thing that existed on day one and everything else built around it.
We under-instrumented the first four weeks in production. We had latency and intent logs, but not a clean per-call reason for every escalation. Reconstructing why the agent transferred a call is harder than it should be for the earliest cohort.
We should have run the identity-verification flow past front-desk staff before building it. They already had a faster verbal shorthand for verifying regulars. We built the formally correct flow instead of the one the humans had converged on.
We led with compliance language we could not fully evidence. That is on us, and this page is the correction.
What is still running
The agent is live and in production. Open workstreams: filling the percentile and denominator fields on this page from the telephony export, extending intent coverage to prescription refill requests, and a second-language pass. Refills are deliberately last — they sit closer to clinical than scheduling does.