No enemy required: Are all our systems really just this unreliable?
Recent widespread service disruptions at Microsoft, the FAA, and Chase Bank suggest that internal procedural failures during planned technical work pose a greater systemic risk than external cyberattacks.
In the span of five days, Americans lost access to some of the systems they depend on most. Microsoft 365 suffered an outage that disrupted email and collaboration tools for about a day . The Federal Aviation Administration halted departures nationwide after equipment on its telecommunications network became unstable. Three days later, Chase banking customers found themselves unable to move their own money .
So far, none of these events has been attributed to a cyberattack. Microsoft says it found no sign of one . Chase says its trouble was not cyber-related . The FAA says the evidence shows no indication of a cyberattack , and names a remote radio site where a contractor had been working as one of two potential sources while its investigation continues. Federal officials say they see no evidence that the outages are connected or that an adversary was behind them.
That should concern us far more than it seems to.
Over the last two decades, organizations have poured money into protecting their systems from hostile actors. We have built entire industries around cybersecurity. We run penetration tests, monitor hostile traffic, buy threat intelligence and employ teams whose sole purpose is keeping adversaries outside the perimeter. Those investments matter as the threats from the outside are real.
But by the organizations’ own accounts, at least two of these disruptions did not start outside the fence. They began during planned technical work. The danger may not have been a sophisticated attacker — it may have been merely a routine upgrade.
I spent 23 years in FAA air traffic control and retired as an air traffic manager. Since 2006, I have practiced and taught the FAA’s safety risk management process. I have watched organizations succeed because changes were evaluated before they went live. I have also watched them struggle when responsibility for a change became diffused and nobody owned the outcome.
Consider what has been disclosed. The FAA says a contractor had been working at a remote communications site, including deploying a new router , and that the site is one of two potential sources of the trouble. The root cause remains under investigation.
Microsoft said an earlier network capacity upgrade caused trouble after a later system change , and its preliminary post-incident report describes a latent defect that left servers overwhelmed under load . By the company’s own account, planned work exposed a hidden flaw. It was not an attack.
Chase has attributed its outage to a software bug and has not said how the bug reached its systems.
That makes two of three incidents tied to planned work by their owners’ own accounts. I won’t speculate on the third.
A firewall can block hostile traffic. An intrusion detection system can flag suspicious activity. A security operations center can watch for adversaries around the clock. But none of those controls asks whether maintenance should happen in the middle of a Friday. None asks whether a rollback plan exists, whether a dependency was overlooked, or whether the people approving a change understand what happens if it fails. Those questions belong to management, not technology.
Aviation learned that lesson long ago. The agency’s latest safety management order took effect in September. Its requirement is plain: A planned change cannot proceed without a risk acceptance signature from a management official, and the decision is recorded in the agency’s risk register. Federal bank examiners have guidance of their own — a handbook that calls for a formal process for controlling and approving changes .
On paper, the process is simple, but a procedure on paper is not accountability in practice. The FAA has described the Nov. 13 router replacement as routine work of a kind done for decades without impact. I understand why. In my experience, though, “routine” might be the most dangerous word in any operation. Novel changes get scrutiny because everybody sees the risk. Routine changes get a nod because they worked the last hundred times. The hundred-and-first is the one that makes the front page of the newspapers.
I don’t know whether the router replacement was handled as a planned change under the FAA’s orders; the agency hasn’t said. The Washington Post reported on Nov. 17, citing a person close to the investigation, that an FAA official supervised the work, tested it and signed off shortly before the ground stop. If so, a signature exists. I still don’t know whether anyone accepted the risk before the work began, what assessment supported that decision or what testing was done against the live network. As members of Congress promise oversight on the ground stop, those are the questions to put to the FAA.
The same ones should be put to Microsoft and Chase: What changed? Who approved it? What testing did it pass before it touched a live system?
Those aren’t technical questions. They are leadership questions. Microsoft has released a preliminary post-incident report to its customers. The FAA and Chase should do the same, and all three should answer in language that customers, passengers and regulators can all understand.
More money will not fix this by itself. Neither will a new vendor, a new platform or a taller firewall. The next major disruption may not require an enemy at all. It may require nothing more than a work order and nobody clearly accountable for the result.
Vincent E. Bianco III is principal consultant of Marivin Consulting Services LLC and spent 23 years in FAA air traffic control, retiring as an air traffic manager.
Topics in this story
Gathered from external sources. Rights to this text belong to whoever originally published it.