AI Agent Analysis · Candidate Exercise
Roadside Assistance · Voice Agent Upgrade

Move the two numbers that matter.

Before we get into the analysis, let's see what the future can look like — one fixed call, start to finish (2:25).

Where the agent stands today

A 30-day, 1,000-call stratified sample — the rates below are real rates.
41.6%
Resolved without a human
47.9%
Escalated or transferred
9.7%
Caller hung up
3.67
Avg CSAT · 362 scored

The ask, and the target

Two numbers Replicant wants to move. Everything here ladders to them.
41.6%→47–52%resolution
~33%→~29%low CSAT scores (0–3)
The payoff

Three changes, none of them a rebuild — and every number here is measurable in an A/B test that's possible today, because calls already split between two flow versions.

Read each line as a floor, not a ceiling. Every fix above is measured on its own, with the other leaks left in place. Highest upside is doing them all.

One idea runs through all three fixes: confirm what we already know and get a yes — don't make a stressed caller recite it. The bot already holds the phone number, the car on file and the home address, so it should lead with them: "Am I talking to Jane?" instead of "spell your first name"; "Is this your blue Audi?" instead of "make, model, year, colour?"; "Is your car at 47 Wales St?" instead of an open "where are you?"

The plan · highest impact first

Three fixes, in the order they pay off

Each is grounded in specific calls you can open. Expand any card for the effort breakdown and the tradeoff it carries.

01

Find the car the way a person would

Location is the single largest leak once a caller is verified.
+4.0 ptsresolution · best case
The leak

82 of 478 located-step callers are never located — the biggest leak after verify. The map link is a last resort (median 228s in), and a good pin gets rejected when its auto-generated address reads oddly.

The fix — and the bet behind it
  • A phone GPS pin is the most accurate location we can get
  • Most callers have data and can judge if the link will work where they are, so let them pick: "text you a link, or tell me?"
  • No data or a landline? They decline and stay on voice
  • At a home on file, skip it all: "Is your car at 47 Wales St?"
  • Pin back → trust it, confirm one landmark, don't re-read a generated address
call_id 8613959324 · resolved · CSAT 5/5
173sBOTCan I send you a text with a map to pin your location?
231sBOTGreat! I received your location.
271sBOTService request submitted. Confirmation 980, a vehicle within 60 minutes.
Link → pin → one confirmation → dispatched. The caller scored it 5/5. The pin works; today it is just offered too late.
+4.0 pts best case · ~+2 realistic ~1 hr saved / 1,000 calls Effort 2.0 / 5

How we reach +4.0. 47 located-step calls are recoverable (20 address capture + 27 link failures). Once a car is located, 87% resolve today, so 47 × 87% ≈ 40 calls ≈ +4.0 pts. Realistic ~+2 at today's link-accept and pin-return rates.

Verified · car confirmedProblem classified · tow or not is knownThis fix82 calls today“Is your car at 47 Wales St?”Address on fileYes: pickup doneTow? Destination nextNo“Text you a link, or tell me?”Caller choosesLink: needs a tow?Known from the problemWhat the SMS form asksTow: location + destination · else locationForm back: trust the pinNo form in ~60s: switch to voiceTell me: landmark or business“The Esso at Main and 5th?”Found: confirm onceNot found: agent, told whyTow? Ask the destinationNo tow: location onlyLocation capturedPlus destination only for a towEvery transfer says why and passes on the address, pin or form.
Teal = new or changed step. The problem type decides whether a destination is asked for.
What has to work
  • Dispatch accepts a map pin (coordinates), not only a street address · client
  • The address on file is current; "No, I moved" takes the new address and updates the record
  • The failing SMS links are fixed, and landlines are detected before a link is offered
  • Every transfer passes on the pin, address or form so no one repeats themselves
What makes it fail
  • A GPS pin can be off in rural areas or garages → the driver confirms on arrival
  • Not reading the full address back could hide a wrong spot → confirm one landmark
  • A business lookup may pick the wrong branch → confirm the cross street or city
  • The 8 silent transfers are a rule we can't see; no gain is claimed until the client shares it
02

Stop losing recognised callers at the door

More callers drop during verification than at any later step.
+3.2 ptsresolution
The leak

238 of 994 connected calls are lost before the caller even explains the problem. In 59 of them the phone number was already recognised, but the caller was still asked to spell a name, the spelling failed, and they were transferred. One caller swore when asked to read out a 16-digit membership ID.

The fix
  • Number recognised? Confirm with one question: "Am I talking to Jane Doe?" (or "Jane or Mark?")
  • Number not recognised? Ask first: "Do you have a membership ID?"
  • Has an ID → verify on the ID alone; skip name-spelling once it's found
  • No ID → name + address; no match → a human, with the reason
call_id 8613304811 · transferred · no CSAT
0.7sBOTI found your membership. Could you please spell your first name?
18.4sCALLER[NAME]
27.2sBOTPlease hold while I transfer your call to an Auto Club agent.
"I found your membership" — then dropped over a name spelling it did not need. The number was already a match.

Assumption. The 59 name-mismatches are most likely a speech-to-text miss — "Tony" heard as "Toni" — or other causes the masked data can't show. Confirming the name with a yes sidesteps the guess either way.

+3.2 pts resolution Low scores 32.6% → 29.9% Effort 2.0 / 5 Removes ~27s of name-spelling
“Welcome” + 911 noticeProblem first: “How can I help?”Classify against rulesSafety → a person; the rest continueThis fixidentificationPhone number recognised?YesNo“Am I talking to Jane Doe?”Several? “Is this for Jane or Mark?”Not verified recently?Text a 4-digit code to the number on fileVerified — no name spelling“Do you have a membership ID?”Number not recognisedYesNoSay or key the IDFound → verifiedName + addressMatch → verifiedNo ID and no match → a human, with the reason. Recognised callers skip name-spelling.
Teal = new or changed step. Top boxes are the existing flow, shown for context; the dashed area is this fix.
What has to work
  • A "verified in last 3 months" flag stored per member and read by the bot · client
  • An SMS sender for the code; landline callers fall back to name + address
  • Security sign-off to relax verification for low-value requests; legal clears the text · client
  • Loose name + address matching against member records for the no-match group
What makes it fail
  • Reading a name aloud lets anyone holding the phone say yes → low-value requests only; the roadside ID check is the backstop
  • "Jane or Mark?" reveals names on the account → acceptable for family accounts, the usual case
  • An SMS code needs the caller to check messages mid-call; landlines can't receive it
  • Verified callers still leak later, so the gain is capped until fixes 01 and 03 land
03

Ask the problem first, then route

Safety and out-of-scope calls should reach a person in seconds, not minutes.
+0.9 ptsresolution · +time +CSAT
The leak

94 verified callers were sent to an agent before the agent ever asked where the car was — including emergencies. Callers who scored low were asked "Is that correct?" far more often than happy ones, and rigid confirmation dropped calls that were really fine.

The fix
  • Classify right after the problem; safety cases go to a human immediately
  • Verify next — so even calls the bot can't book reach a human already verified
  • Out-of-scope (RV, keys, impound) routes with notes, no re-verifying by the agent
  • Confirm the car on file with one yes/no; accept "Mhmm" and an unknown year
  • One read-back at the end, not a check after every answer
call_id 8613514334 · transferred as urgent
80sCALLERHe's locked in… a truck with a dog inside.
86.9sBOTThis is urgent — I'm transferring you to an agent immediately.
Right call, but 86 seconds too late. A pet locked in a hot car should route the moment it's heard, before any verification or car questions.
+0.9 pts resolution Abandonment 9.7% → 9.2% ~1 hr saved / 1,000 calls ~50 handoffs reach agents pre-verified Effort 2.5 / 5
“Welcome” + 911 noticeKnown number, open ticket?Status / changeOr call goes to agentProblem first: “How can I help?”This fix94 calls todayClassify against rulesRight after the problemSafety: agent nowCrash, child, highwayVerify the memberfix 02 · confirm, ID, or name + addressCan the bot book it?Before any car detailsNo: agent, notesRV, keys — sent verifiedTow for later?“Tomorrow”, “Monday”Schedule itOr agent with notes“Your blue 2004 Audi?”Or a different car? Else make, modelYear unknown: fine“Mhmm” counts as yesOne read-back at the endNot a check after every answerFind the car (next section)
Teal = new or changed step. Verification (fix 02) runs right after classification, so even a call the bot can’t book reaches a human already verified. Only safety cases skip it.
What has to work
  • The routing rules already exist — they just run right after the problem · Replicant
  • Vehicles are kept on file for "Is this about your blue 2004 Audi?"; if not, ask make + model · client
  • Scheduled tows need a new dispatch capability and a policy decision · client
  • The handoff carries the verified identity, the problem and what failed, so the caller doesn't repeat it
What makes it fail
  • One read-back at the end catches a wrong detail late → re-ask only the wrong part
  • Accepting an unknown year could send the wrong truck → keep the drivetrain and height questions
  • The member may be in a different car → always add "or a different car?"
  • Missing an emergency is serious → when unsure, treat it as an emergency
How I got to these numbers. I read about 100 calls by hand to find the patterns, then measured all 1,000 with regex on the transcripts and turn-timestamp analysis, so the rates are computed across the full sample rather than a handful of anecdotes. The three flows above are the exact conversation redesigns behind each fix — teal marks every step that is new or changed.