The call that fails one time in twenty
The worst voice problems are not the ones where nothing works. They are the ones where almost everything does. A call fails, the user redials, and the second call goes through. By the time anyone looks, the evidence is gone.
These problems stay open for weeks, and they stay open for the same reason every time: each team checks its own equipment, finds it healthy, and concludes the fault is elsewhere. Everyone is partly right. The equipment is healthy most of the time.
Stop asking what is broken. Ask what is different.
An intermittent failure means that some calls take a path, or carry a setting, that the others do not. The work is finding that difference. Reading one failed call by itself rarely shows it, because a failed call looks mostly normal. Putting a failed call beside a successful one does.
A method that works
- Capture before you theorize. Turn on logging at the gateway or session border controller and leave it on until the failure happens again. Log to a buffer or a log server, never to the console, and check the device's load first. Record the time, the calling number and the called number for each failure users report.
- Collect one bad call and one good call through the same path. Same site, same direction, same carrier if you can.
- Compare the same message in each. Start with the first request and the final response. Ignore the fields that always differ, such as identifiers and timestamps. Look at the addresses, the codecs offered, and which device answered.
- Count. If one call in twenty fails, look for something that is true of one call in twenty. Two gateways sharing load unevenly. One server in a pool of several. A setting applied to one trunk and not its twin.
- Prove it. Force a test call down the suspected path. If it fails every time, you have found where the fault is. If it does not, you have saved yourself a wrong fix.
What the pattern usually turns out to be
- Two devices that should be identical and are not. A pair of gateways where one was upgraded or edited and the other was not.
- One member of a pool. A carrier or cloud service answers from several addresses, and one of them behaves differently.
- Time. Failures that follow the busy hour point to capacity: a call limit on a trunk, or a congested link.
- A specific destination. Calls to one carrier, one country or one number range take a different route out.
Bring evidence, not a description
A carrier or vendor can act on a call identifier, an exact time with its time zone, and the message that was wrong. They cannot act on "calls fail sometimes." The quality of the evidence decides how long the ticket stays open.
If a second attempt works, the system is telling you it has two behaviors. Find the second one.
This comparison is the reason we built CollabPlane Workbench: it puts two calls side by side and marks the fields that differ. The method works with any tools, including a text editor and patience.