/projects/dell-automated-caller

Dell Automated Caller

Automated end-to-end testing for a phone system.

Conception, architecture and implementation · 2020

.NET CoreRabbitMQEntity FrameworkTwilioxUnit

Private

Overview

An internal tool that tests an interactive voice system by actually calling it — the test suite dials the phone menu, listens to what it says, and checks it against what was expected, then files the result alongside the rest of the suite.

What I did

I conceived the tool and built it, and later mentored the junior engineer who joined the project.

The test scripting language and the validator that rejects a bad script before it costs a call.
Similarity-based assertion, with the threshold declared per step.
The queue between the request and the call.
The telephony integration and the webhook that carries each transcribed response back.
Reporting results back into the test-management tool.

The problem

Testing a phone menu meant someone dialling it, pressing the keys, listening to what the system said, and writing down whether it was right — once per scenario, per language, per route. A full cycle was over twenty thousand calls placed by hand across the team, which in practice meant the full cycle almost never ran. Automating it brought the cycle down to about three hours.

By the numbers

Call volume and cycle time as recalled from the project; the command count is verifiable in source.

20k+calls per test cyclepreviously placed one at a time, by hand across the team

~3hto run the full cycleit had taken about a month

9commands in the test DSLthe script is validated before anything is dialled

Architecture

A .NET Core service in DDD layers. The API accepts a script; a validator rejects a malformed one before a call is placed; the run is dispatched over a RabbitMQ publish/subscribe queue; a telephony provider places the call and posts each transcribed response back by webhook; the response is scored against what the script expected; and the outcome is written back to the test-management tool against its plan, suite and work-item identifiers.

Test scriptAn ordered list of commands describing one call.
ValidatorRejects a malformed script before anything is dialled.
QueuePublish/subscribe, so a slow call never blocks the request.
Telephony providerPlaces the call and posts each transcribed response back.
Test managementReceives the outcome against its plan, suite and work item.

A test script

Setup Language="en-US"
Dial +1 (000) 000 0000
Wait 3
Hear [Confidence=85%] thank you for calling, please say or enter your service tag
Enter (serialnumber) 1234567#
Hear [WaitBefore=2] one moment while I look that up
Hang

The grammar is checked before the call: a missing Dial or Hang, a repeated step where only one is allowed, or a step out of order fails the script rather than the phone bill. Validation steps run after the call ends. The number above is a documentation placeholder.

One test cycle

By hand~1 month

Automated~3 hours

Durations as recalled from the project; the repository does not record them.

What a step records

Expected	Heard	Similarity
please enter your service tag	please enter your service tag	100%
one moment while I look that up	one moment while i look that up	97%
transferring you to support	transferring you to sales	78%

Structure from the real model — every spoken response is stored with what was expected, what was transcribed, and how closely the two matched. Values here are illustrative.

What it does

A test script is a short list of ordered commands: dial, wait, enter digits, listen, validate, hang up.
Placeholders in the script are substituted at run time, so one script covers many data sets.
Every spoken response is stored with what was expected, what was heard, and how closely they matched.
Results are written back to the test-management tool against the plan, suite and work item they belong to.
A malformed script is rejected with a readable list of errors before any call is placed.

Engineering decisions

Assert on similarity, with the threshold declared per step
Speech transcription is never character-exact, so comparing for equality fails good tests. Each assertion carries its own tolerance in the script, because how close a transcription lands depends on what was said — a stock prompt transcribes reliably, a product name does not.
The script is a small language, validated before anything is dialled
A real call costs time and money and cannot be undone. The validator checks that the required commands are present, that single-use commands appear once, that the order is legal, and that each line matches its grammar — reporting every error in plain language before the first digit is dialled.
A queue between the request and the call
A phone call takes minutes and fails for reasons outside the caller’s control. Publish/subscribe decouples whoever asked for the run from whatever executes it, so a slow or failed call never blocks the request that started it.
Checking more than the audio
Hearing the right words does not prove the call was routed correctly. Separate validation steps check the voice menu, the telephony routing, and the records both left behind — which is what makes it an end-to-end test rather than an audio assertion.