Recovering from the SolarWinds hack could take 18 months

Fully recovering from the SolarWinds hack will take the US government from a year to as long as 18 months, according to the head of the agency that is leading Washington’s recovery.

Brandon Wales, the acting director of CISA, the US Cybersecurity and Infrastructure Agency, says that it will be well into 2022 before officials have fully secured the government networks compromised by Russian hackers. The list includes at least nine federal agencies, including the Department of Homeland Security and the State Department. Even fully understanding the extent of the damage will take months.

“I wouldn’t call this simple,” Wales says. “There are two phases for response to this incident. There is the short-term remediation effort, where we look to remove the adversary from the network, shutting down accounts they control, and shutting down entry points the adversary used to access networks. But given the amount of time they were inside these networks—months—strategic recovery will take time.”

When the hackers have succeeded so thoroughly and for so long, the answer sometimes can be a complete rebuild from scratch. The hackers made a point of undermining trust in targeted networks, stealing identities, and gaining the ability to impersonate or create seemingly legitimate users in order to freely access victims’ Microsoft 365 and Azure accounts. By taking control of trust and identity, the hackers become that much harder to track.

“Most of the agencies going through that level of rebuilding will take in the neighborhood of 12 to 18 months to make sure they’re putting in the appropriate protections,” Wales says. 

The hack on SolarWinds, a US software firm with customers around the world, was first discovered in November 2020. But American intelligence agencies say Russian hackers first infiltrated in 2019. Subsequent investigation has shown that the hackers started using the company’s products to distribute malware by March 2020, and their first successful breach of the US federal government came early in the summer. That’s a long time to go unnoticed—longer than many organizations keep the kind of expensive forensic logs you need to do the level of investigation required to sniff the hackers out.

SolarWinds Orion, the network management product that was targeted, is used in tens of thousands of corporations and government agencies. Over 17,000 organizations downloaded the infected back door. The hackers were extraordinarily stealthy and specific in targeting, which is why it took so long to catch them—and why it’s taking so long to understand their full impact.

The difficulty of uncovering the extent of the damage was summarized by Brad Smith, the president of Microsoft, in a congressional hearing last week. 

“Who knows the entirety of what happened here?” he said. “Right now, the attacker is the only one who knows the entirety of what they did.”

Kevin Mandia, CEO of the security company FireEye, which raised the first alerts about the attack, told Congress that the hackers prioritized stealth above all else.

“Disruption would have been easier than what they did,” he said. “They had focused, disciplined data theft. It’s easier to just delete everything in blunt-force trauma and see what happens. They actually did more work than what it would have taken to go destructive.”

“This has a silver lining”

CISA first heard about a problem when FireEye discovered that it had been hacked and notified the agency. The company regularly works closely with the US government, and although it wasn’t legally obligated to tell anyone about the hack, it quickly shared news of the compromise with sensitive corporate networks.

It was Microsoft that told the US government federal networks had been compromised. The company shared that information with Wales on December 11, he said in an interview. Microsoft observed the hackers breaking into the Microsoft 365 cloud that is used by many government agencies. A day later, FireEye informed CISA of the back door in SolarWinds, a little-known but extremely widespread and powerful tool. 

This signaled that the scale of the hack could be enormous. CISA’s investigators ended up working straight through the holidays to help agencies hunt for the hackers in their networks.

These efforts were made even more complicated because Wales had only just taken over at the agency: days earlier, former director Chris Krebs had been fired by Donald Trump for repeatedly debunking White House disinformation about a stolen election. 

While headlines about the firing of Krebs focused on the immediate impact on election security, Wales had a lot more on his hands. 

The new man in charge at CISA is now faced with what he describes as “the most complex and challenging” hacking incident the agency has come up against.

The hack will almost certainly accelerate the already apparent rise of CISA by increasing its funding, authority, and support. 

CISA was recently given the legal authority to persistently hunt for cyber threats across the federal government, but Wales says the agency lacks the resources and personnel to carry out that mission. He argues that CISA also needs to be able to deploy and manage endpoint detection systems on computers throughout the federal government in order to detect malicious behavior. Finally, pointing to the fact that the hackers moved freely throughout the Microsoft 365 cloud, Wales says CISA needs to push for more visibility into the cloud environment in order to detect cyber espionage in the future.

In the last year, supporters of CISA have been pushing for it to become the nation’s lead cybersecurity agency. An unprecedented cybersecurity disaster could prove to be the catalyst it needs.

“This has a silver lining,” said Mark Montgomery, who served as executive director of the Cyberspace Solarium Commission, in a phone call. “This is among the most significant malicious cyber acts ever conducted against the US government. The story will continue to get worse for several months as more understanding of what happened is revealed. That will help focus the incoming administration on this issue. They have a lot of priorities, so it would be easy for cyber to get lost in the clutter. That’s not going to happen now.”