Which model hears your callers?

Test it on your own calls

One Tuesday evening this summer, a caller phoned [company], which delivers parcels for online shops, and told its AI agent: "Hi, it's about parcel PX 4827 3315. Please don't leave it with the neighbor next door. I'm away until Thursday and I'll pick it up myself." The transcript had every word right except "don't". Without that word the sentence asked for the opposite, and the parcel was left with the neighbor. On Thursday the same caller was back on the line, angry.

Irene, who runs [company]'s customer line, read the transcript the next morning. One word was missing, and it was the word the call was about. The agent had been taking the evening calls all summer in a pilot, with the speech-to-text model that came with the vendor's setup, which this post calls Model A. The vendor's page gave Model A a high score for words right overall. [company], Irene, the pilot, the calls, the two models and every count in this post are an example, not real data.

A score like that would not have warned Irene. It counts every word alike, and a phone call depends on a few of them: the tracking number, the surname, the day, the address and any "not" that turns a request around. The model that hears your callers is the one that gets those words right. When you choose speech-to-text for phone calls, or speech recognition as it is also called, test the candidates on your own recorded calls. Mark the words that carry the call in each recording, count how many of them each model gets exactly right, and look at where each miss ends up. Base the choice on what a word list cannot fix later, and have the agent read back every request that changes something.

Can you trust the vendor's score?

The vendor's page sums a model up in one number, how many words it got right in 100. Some pages count the misses instead and quote the word error rate. Either way it is an average over every word the model was tested on, so an "um" counts exactly as much as the third digit of a tracking number.

Scored that way, the Tuesday transcript looks excellent. The model wrote down the tracking number, the neighbor, the day and everything else the caller said, and lost "don't", the word that turned the request around.

The score was also measured on someone else's recordings. Your callers are in the car, on speaker or out on the street, their voices reach the model through a phone line, and they read out a tracking number from a message on their phone, fast. Before you rely on the vendor's number, ask how close its recordings are to calls like these.

• Whose voices were recorded, in which language and which regional variety?
• Did the speakers read from a script or talk freely?
• Did the sound come through a phone line or a studio microphone?
• Was there noise around them, or a quiet room?

Which words carry the call?

Irene took 50 recorded calls from one ordinary week of the pilot, the week of the Tuesday call, so that call is one of them. They sound like her evenings, with callers in the car, on speaker and outside, older callers, callers with an accent, and some who said a street name in another language. Choose recordings your team is allowed to use for this, and keep the transcripts with the same care as the recordings.

Model A's transcripts of those calls were already there from the pilot. Irene asked the vendor to run Model B, the second candidate, on the same recordings, and to score both models for words right overall on these calls. Two reps who answer calls like these every day then listened to each recording once, wrote down the words that carry the call as the caller said them, and checked both transcripts on those words alone.

The check was strict. A tracking number counted as right only when every letter and every digit was right, because a number that is almost right finds no parcel, or finds someone else's. A surname counted when it matched the name on the parcel, and a "not" counted when the transcript kept it.

One call can hold a tracking number, a surname and a day at once, so the reps' total of 150 counts words, not calls.

Which model got them right?

On the vendor's score for these 50 calls, Model A was ahead, with 94 words right in 100 against 92 for Model B. The reps' check gave the lead to Model B.

Model B got 131 of the 150 words that carry the call right, and Model A 112. The biggest gap was in tracking numbers, 40 of 44 against 30. When callers read the number fast, Model A dropped a digit or swapped two.

Model A also lost 3 of the 18 "not"s, and one of them was the Tuesday caller's "don't". Model B kept all 18 in these 50 calls. Surnames were hard for both. Model A got 20 of 36 right and Model B 23, so neither transcript should be trusted with a surname on its own.

Most of the words that put Model B behind on the overall score were small words and filler words, such as "um", "so" and "the", which nobody acts on. Irene chose Model B for the customer line. If two models come out close on the words that carry the call in your own test, add more calls before you decide.

What happens when it mishears?

A miss can end in one of three ways. A check against your records catches it, the agent catches it by reading the request back, or it goes through.

Model A missed 38 of the 150 words that carry the call. Its 14 wrong tracking numbers matched no parcel, so the agent asked again, and its 16 wrong surnames did not match the name on the parcel, so the agent asked the caller to spell it. The other 8 misses had nothing to catch them: 4 addresses, 1 day and 3 "not"s. In the pilot the agent did not read requests back, so those 8 went through as the model heard them.

Model B missed 19 of the 150. On its transcripts, the check against records would have caught 4 wrong tracking numbers and 13 wrong surnames. Its 2 wrong addresses would have had nothing to catch them without a read-back.

When the model heard the tracking number right, the agent found the parcel on the first try. In these calls no misheard number matched someone else's parcel, but one could, which is why the agent checks the surname too.

Your records cannot check a new address, a new day or where the caller wants the parcel left. So the agent reads back every request that changes something, and says in its own words what it is about to do before it asks whether that is right. That way the caller hears the action, which is the part they need to check. For a "not", the read-back is the only check there is, since no record can say whether the caller said "don't".

If the agent had read the request back, the Tuesday call would have gone like this, with the same transcript from Model A.

Caller: "Please don't leave it with the neighbor next door. I'm away until Thursday and I'll pick it up myself."

Agent: "So we leave your parcel with your neighbor. Is that right?"

Caller: "No, please don't. I'll pick it up myself when I'm back on Thursday."

Agent: "Sorry about that. So we keep your parcel at the depot until you pick it up on Thursday. Is that right?"

Caller: "Yes, thanks."

Model A's dropped "don't" becomes a correction on the same call instead of a second, angry call on Thursday. The agent apologizes once and goes straight to the corrected action, so the caller checks the new plan too. A read-back will not catch every miss. This one would have kept the Tuesday parcel at the depot.

What can a word list fix?

After you choose, you can still give the model a word list. For [company] that means the street names and towns it delivers to, its tracking number format of two letters and eight digits, and the names of its services. Vendors call such a list "custom vocabulary" or "keywords", and it helps any model with words it has not met.

Irene added the streets and towns of [company]'s area and had the same 50 calls run through Model B again. On the second run it got all 22 addresses right.

So compare the models on what a word list cannot fix. No list teaches a model to keep a "not", or to hear your callers' language the way they speak it, with its regional speech, its accents and its older callers. A noisy phone line from a car, a speaker or the street is just as noisy with a list. Picking the model also decides whether it writes while the caller is still talking or only after they stop. A model that waits makes every answer slower, and our post on why your AI agent makes callers wait looks at that pause.

Surnames stay hard too. No list holds every caller's surname, so the agent keeps checking the name on the parcel.

What happens to the transcript after the model, from cleaning up numbers and dates to working out what the caller wants, is covered in our post on how voice AI fixes what it mishears.

If your customer line takes calls in more than one language, run the test again for each of them. The model that hears your German callers best may not be the one that hears your Turkish callers best.

Where does DRING come in?

DRING does not tie an agent to one speech-to-text model. In DRING's speech core, the models for speech, language and voice are chosen per agent and per language, and a workflow can route to a different model without a re-platform. Callers in German and callers in Turkish can be heard by different models. The call is transcribed in real time, then corrected and normalized before the agent reasons.

In Agent Factory, where DRING builds, tests and improves its agents, your calls and workflows become a working agent and hard conversations become tests. A call like the Tuesday call can become one of them. A stronger speech-to-text model goes live only after clearing the regression suite the whole stack runs.