Safety & OperationsClearance sheet

FAA Halts Router Installations After Four-Hour Outage

The FAA has halted Juniper router installations after a multicast misconfiguration flooded its network with 1,000 requests every four seconds, grounding flights nationwide for over four hours.

Read-back

  1. Friday's outage grounded flights nationwide for more than four hours, with days of follow-on disruption.
  2. A Juniper router installed by L3Harris in Arkansas generated roughly 1,000 connection requests every four to five seconds.
  3. L3Harris' predecessor Harris Corp. has held the FAA telecom contract since 2002, creating what Duffy called a vendor-lock.
  4. The FAA has disabled the multicast setting on other Juniper routers but cannot yet explain how it was turned on.
  5. Senate Democrats demanded a deployment pause on Tuesday; the FAA's halt effectively grants it.
FAA Pauses Installation of Gear Tied To Messy Flight Halt
PlateFAA Pauses Installation of Gear Tied To Messy Flight Halt — bernswaelz / pixabay

The FAA has paused all further installations of the Juniper routers tied to Friday's nationwide ground stop, which halted flights for more than four hours and disrupted travel for days afterward. The suspension announced Tuesday is open-ended; the agency gave no date for resumption.

The failure traces to a single router installed at an FAA facility in Arkansas. Contractor L3Harris Technologies carried out the work as part of the program switching air traffic control telecommunications from legacy copper wire to fiber. The router, supplied by Hewlett Packard Enterprise's Juniper Networks, shipped with a broadcast — multicast — setting turned on, according to a person familiar with the matter.

What did the misconfiguration actually do?

The setting flooded the FAA network with connection requests, roughly 1,000 every four to five seconds by the agency's own count. The traffic degraded Cisco Systems routers across the country, taking down the telecom backbone controllers rely on.

HPE defended its hardware. "Based on what we know and have been told by others, our equipment was working as configured," the company said in a statement. "The FAA is assessing the configuration and performance of all infrastructure involved to determine the cause." Cisco did not respond to a request for comment.

FAA chief Bryan Bedford said the agency believed the multicast function had been disabled. "We do know that this router could be enabled to do what it did," he told reporters on Capitol Hill Tuesday. "We thought we had that technology turned off." The FAA has now disabled the setting on the remaining Juniper routers in the system.

What remains unexplained is who or what re-enabled it. "We've seen nothing in how the configuration on that device was changed by the installation," Bedford said. The uncertainty is the story: an audit finding, not a fix, is what will determine accountability.

Bedford also said an L3Harris employee worked under the supervision of an FAA technician in Arkansas. That account contradicts an earlier statement from the union representing FAA technicians — a dispute that now sits unresolved in the record.

Can L3Harris be replaced?

Transportation Secretary Sean Duffy, asked directly whether the contractor would be fired, said no — because it cannot be. The company holds what he called a "vendor-lock" on the government: it essentially owns and monitors the entire telecommunications network. L3Harris' predecessor, Harris Corp., won the original FAA contract in 2002.

"I'm going to have to work with them," Duffy said. He expects improved performance, and for the first time suggested Washington may enlist an additional firm to monitor the network controllers depend on. "It took longer than I would have liked to identify that router with L3Harris," Duffy told reporters. "We have to have them perform better, and then we're going to look at bringing some outside help in."

A representative for L3Harris did not respond to a request for comment. Duffy has previously said officials are examining whether "incompetence" played a role in the outage.

What does this change for the modernization program?

Senate Democrats on a key committee had called earlier Tuesday for an immediate pause on router deployments until the FAA demonstrates the copper-to-fiber cutover can be done safely. The agency's own halt effectively grants that demand. The practical consequence: the telecom replacement program, already 24 years under one contractor's control, now proceeds without a schedule, under an unresolved root-cause investigation, and with a second vendor possibly layered on top of the first.

Until the FAA explains how the multicast setting came to be active, the pause protects the network but answers nothing about whether the next installation is safe.

via bloomberg.com (Original)

Filed under

  • faa
  • ground-stop
  • air-traffic-control
  • juniper-networks
  • l3harris
Share this article:

More from James Calloway

James Calloway

Show full bio

Staff writer covering people, launches and day-to-day practice in Commercial aviation industry coverage at Flightdeck Report.

726 articles

Same bay

« Previous articleNext article »