RAG support for a DTC brand, with a citation-accuracy eval

Ticket deflection and CSAT are not in this table. An earlier draft of this page showed 58% deflection against a total contact count and window we had not sourced, with a baseline and after CSAT both marked as gaps — a percentage with an undisclosed denominator and no satisfaction figure beside it. We will not publish it in that state. This is the definition it will be measured against once it is:

> A contact is deflected when the customer opens a conversation, receives an agent answer, and does not open a new contact on the same issue within 72 hours, and does not request a human at any point in the conversation.

That excludes abandons. A customer who gives up and leaves is not deflected — that is a failure counted as a failure. It also excludes any conversation that reached a human, even if the agent resolved 90% of it first. It is a stricter definition than "no-human-touch" and much stricter than "session ended without escalation." If a vendor quotes you a deflection rate, ask which of those three they mean; the same system scores wildly differently under each.

Deflection alone is a metric CX buyers distrust, correctly. A system can deflect 100% of tickets by being impossible to escalate from. That is why CSAT on agent-handled contacts is a required field before a deflection rate ships on this page at all, not a footnote to add later.

How we measured. First response time comes from the helpdesk export, not from our own logs — the client's system of record, not ours. Deflection, once published, will use the same source, with repeat-contact matching on customer ID plus issue class, not on ticket text similarity.

What we are not claiming. No revenue figure. No cost-per-ticket saving. No headcount reduction — the client did not reduce their support team, they moved it onto the harder half of the queue.

*Can a support agent be trusted to answer from our own policies, when getting a returns window wrong costs us a chargeback and a public review?*

The commercial risk in e-commerce support is not an unanswered question. It is a confidently wrong answer about a refund, a shipping restriction or a delivery date — because the customer will hold you to it.

Two kinds of truth, two completely different freshness requirements. Policy text changes a few times a year. Order status changes every few minutes. A single retrieval system that treats both as documents will happily serve a customer a cached shipping status from yesterday, with a citation, which makes it worse than no answer at all.

The headline claim was accuracy, and accuracy was never measured. The previous version of this page led with "cited answers" and never quantified whether the citations were right. A citation that points to the wrong paragraph is a trust device attached to a wrong answer. That is the gap this rewrite exists to close.

The rejected alternative. We were asked to fine-tune a model on the help centre instead of building retrieval. We refused. Fine-tuning bakes the policy into weights, so a returns-window change requires a retraining cycle, and the model cannot cite a source because it no longer has one. Retrieval was slower to build and is the only version that stays correct when the policy changes on a Tuesday.

What broke. Our first chunking strategy split on fixed token windows. It cut a shipping-restrictions table across two chunks, so the agent retrieved the country list without the exclusions attached to it and answered that a restricted destination was fine. Caught in eval, not in production, which is the entire argument for having an eval.

flowchart TD
    A["Customer message"] --> B["Intent + entity extraction<br/>order id, email, SKU, issue class"]
    B --> C{"Needs live account data?"}

    C -->|Yes| D["Tool call layer"]
    D --> D1[("Commerce platform API<br/>order, fulfilment, tracking")]
    D --> D2[("Returns / RMA system")]
    D1 --> H
    D2 --> H

    C -->|No| E["Hybrid retrieval"]
    E --> E1["BM25 — lexical<br/>SKU codes, policy names"]
    E --> E2["Dense — pgvector<br/>semantic paraphrase"]
    E1 --> F["Fuse candidates — top 40"]
    E2 --> F
    F --> G["Cross-encoder rerank → top 6"]
    G --> H

    H["Answer composition<br/>grounded, citation-bound"] --> I{"Groundedness check<br/>every claim spans a source?"}
    I -->|Pass| J["Answer + source links to customer"]
    I -->|Fail| K["Escalate — no answer sent"]

    B -->|"Refund, complaint, damage,<br/>legal, low confidence"| K
    K --> L["Human agent<br/>transcript + retrieved docs + order state attached"]

    J --> M[("Eval + telemetry log<br/>query, chunks, citations, outcome")]
    K --> M
    M --> N["Weekly eval replay<br/>+ retrieval gap review"]

The groundedness check is a hard gate, not a score in a dashboard. If a sentence in the drafted answer cannot be traced to a retrieved span, the answer is not sent. The agent escalates instead. Refusing to answer is always available to it.

Chunking. Structure-aware, not fixed-window. Help centre articles split on heading boundaries; tables are never split — a table is one chunk with its caption and preamble attached. Each chunk carries its article title, section path and last-updated date as a header, so the retriever sees the context a bare paragraph loses. Long sections use a sliding window with overlap; short adjacent sections are merged up to a floor so single-sentence chunks stop polluting the candidate set.

Embeddings. A general-purpose commercial embedding model, chosen after a bake-off against two alternatives on the client's own eval set rather than on a public benchmark. The specific embedding model and vector dimension are withheld pending confirmation against the deployed configuration. Benchmark leaderboards do not predict performance on one brand's returns policy.

Hybrid, because SKUs are not semantic. Dense retrieval alone loses exact identifiers — product codes, order prefixes, policy names. BM25 catches those. Fusing both and reranking with a cross-encoder over the top 40 gave the largest single accuracy gain in the build, larger than any prompt change.

Freshness. Two paths, deliberately. Documents reindex on a webhook from the CMS at publish time, with a nightly full reconciliation to catch missed hooks. Order data is never indexed. It is fetched live at answer time through the commerce API. Nothing about an individual order is ever embedded, cached or retrieved from a vector store, which removes the entire class of stale-order-status failures by construction.

Escalation path. Refunds, damage claims, anything with complaint or legal language, any groundedness failure, any low-confidence identity match, and any explicit request for a human. The human receives the transcript, the retrieved chunks and the live order state, so they do not restart the conversation.

1. Blueprint. Two weeks reading twelve months of the client's actual tickets and classifying them: automate, escalate, or fix upstream. A meaningful share were not support questions at all — they were product page failures. Those went to the client's own backlog, not to us. Scope and price locked here.

2. Prototype. Retrieval only, no conversation. We proved we could pull the right passage for the top question classes before we let a model write a sentence.

3. Forge. Hybrid retrieval, reranking, tool calls to live order data, the groundedness gate, and the escalation handoff. The eval harness was built in this phase, not after it.

4. Harden. Weekly eval replay against a growing held-out set, retrieval-gap review on every escalation caused by a groundedness failure, and content fixes fed back to the help centre. A meaningful share of early failures were fixed by writing a missing help article, not by touching the model.

The eval set. Built from real historical tickets, not written by us. Target size 150–300 questions — our standard eval-set sizing target for a retrieval-grounded support build, not a number drawn from this engagement's own Blueprint documents, and not a count of tickets the agent has since answered — sampled to match the live issue-class distribution, then deliberately over-weighted on the classes that hurt: shipping restrictions, returns-window edge cases, promotion stacking, and questions whose answer changed during the period — those test whether the system is retrieving current policy or an old one.

Every item has a human-written reference answer and the specific source span that answer must come from. That second field is what makes citation accuracy measurable at all.

The metrics that mattered.

Citation accuracy — does the cited source actually contain the claim it is attached to? Adjudicated by a human, not by a model grading itself. This is the headline metric for this build, and the previous page's failure to report it is the reason for this rewrite. Our standard release-gate target for builds of this kind is ≥98% — an engineering acceptance threshold we design toward, not a number drawn from this engagement's own sign-off documents, and not the ongoing production rate. The measured result from the held-out eval run is not yet published on this page; it will be added once the adjudicated run is sourced.

Groundedness — share of generated sentences fully supported by a retrieved span. Enforced live by the gate; measured offline as a rate. Our standard release-gate target is ≥98%, same basis as citation accuracy — an engineering target, not a figure drawn from this engagement's own documents. The measured result is withheld for the same reason as citation accuracy above.

Hallucination rate — share of answers containing at least one unsupported factual claim that reached the customer. Target is not "low." Target is zero, with escalation absorbing the difference. The measured result is withheld for the same reason as citation accuracy above.

Retrieval recall@k — did the correct span appear in the top 6 at all? Separates retrieval failures from generation failures. They have different fixes and conflating them wastes weeks.

CSAT on agent-handled contacts, alongside deflection, always.

Latency. Median and p95 to first token, and to full answer. Live tool calls to the commerce API dominate the tail, so we measure them separately — a slow order lookup and a slow model look identical to a customer and are fixed in completely different places.

Failure modes, and what happens when the system is wrong.

*Retrieved the wrong passage.* Groundedness gate usually catches the mismatch and escalates. When it does not, the citation is visible to the customer, who can see the source does not say what the answer says. Visible citations are a safety feature, not decoration.

*No passage exists.* The agent says it does not know and hands over. It does not improvise from general knowledge about how returns usually work.

*Stale policy.* Prevented by webhook reindex plus nightly reconciliation; each chunk's last-updated date is in the retrieval context.

*Stale order status.* Structurally impossible — order data is never indexed.

*Confident and wrong anyway.* This is the residual risk and we do not claim it is zero. It is bounded by the gate, surfaced by visible citations, sampled weekly by human review, and every occurrence goes into the eval set permanently.

We would have built the eval set in week one. We built it in the Forge phase. Every retrieval decision before that point was made on intuition and had to be re-litigated once we could measure. That cost real time and it was avoidable.

We would have set the deflection definition with the client before quoting any number. We agreed it mid-build. It should have been in the Blueprint, in writing, because it changes the headline figure by tens of points and everyone's expectations with it.

We should have shipped the content-gap report as a product from day one. The most valuable output for the client was not the agent — it was the ranked list of questions their help centre could not answer. We surfaced that as a side effect for months before making it a first-class deliverable.

We under-weighted CSAT early. Deflection was instrumented on day one; satisfaction was retrofitted. That ordering tells you what we were optimising for, and it was the wrong order.

The agent is live. Open workstreams: filling the eval figures on this page from the adjudicated run, extending coverage to pre-purchase product questions — a different risk profile, because a wrong answer there costs a return rather than a chargeback — and a second-language pass on the help centre corpus.