A single typographical error triggered a disruption in telephone service to more than 12 million customers nationwide last summer, a report by a communications firm and Bell officials said.
DSC Communications of Plano, Texas, and Bell Communications Research issued a report that a DSC employee typed the wrong character in a line of instructions distributed nationwide, knocking out service for various periods between June 26 and July 2."We're very, very embarrassed that it happened," DSC Chairman James Donald told the Dallas Morning News. "It's just such a little bitty difference there, but such a large effect upon other things."
The report said the error consisted of entering the numeral 6 instead of the letter D on a 23-character line known as a "patch" that was supposed to fix an earlier mistake. The new error resulted in the company's software sending the wrong signal to telephone transfer sites, breaking down the system's ability to handle traffic on the telephone networks.
The employee who committed the gaffe is not being identified because he is considered a top programmer of the company, Donald said.
"He's conscientious, he works hard, he wants everything to be right, " Donald said of the employee.
The glitch June 26 caused an eight-hour outage to 5 million Atlantic Bell customers in Virginia, West Virginia, Maryland and Washington, D.C. , and a three-hour outage for 3 million Pacific Bell customers in the Los Angeles area.
On July 1, 1 million Bell Atlantic customers lost service in Pittsburgh for six hours while 2.4 million Pacific Bell customers in San Francisco lost service for just four minutes. The same Pittsburgh customers lost service for three hours July 2.
DSC had field-tested a major update of its software in June, but found a "deficiency" in the creation of a so-called "routing database," which tells signal points how to route calls through a telephone network.
To correct the deficiency, DSC patched its software but the typo caused the widespread outages. A half-dozen company engineers were sent to each problem area while 200 others searched for the problem at DSC headquarters in Texas.
Finally, on July 4, the system's "fault isolation" feature zeroed in on 200 lines of software which then had to be examined by eye. The next day, a DSC error was positively identified as the source of the problem.
"We just wish it had never happened," Donald said.