Exclusive: This pizza box-sized equipment could be key to Ukraine keeping the lights on this winter – CNN

CNN
 — 

Staring down another frigid winter and desperate to keep the lights on, Ukraine’s power grid operator has surreptitiously imported custom-built equipment designed to withstand Russian electronic warfare attacks with the help of US officials, CNN has learned.

Engineers at US tech giant Cisco spent weeks building and stress-testing the new gear in a lab in Austin, Texas, and delivered a prototype to Ukraine in the spring with the help of a US Air Force plane carrying humanitarian aid, according to Cisco.

After Ukraine’s state-owned grid operator, Ukrenergo, quietly confirmed the new equipment worked despite Russian attacks on its GPS systems, Cisco shipped dozens of the pizza box-sized hardware kits worth an estimated $1 million to Ukraine, where they were installed across the country, Ukrenergo executives told CNN.

The new equipment, which has not been previously reported, could offer a crucial lifeline to Ukraine’s electricity grid, which remains a key target of Russian attacks as the Kremlin’s war enters its second full winter. Russian missile and drone strikes over the last two years have destroyed about 40% of the power substations and related equipment that Ukrenergo operates across the country, the grid operator told CNN.

In a rare cyberattack that has only just been made public, hackers connected to Russia’s military intelligence agency, the GRU, caused a power outage in Ukraine in October 2022, according to US experts.

“We are anticipating them to continue, especially this winter,” Illia Vitiuk, head of cybersecurity for the Ukrainian security service SBU, said of attempted Russian hacks on power plants.

The issue that Cisco aimed to help fix, however, is caused by Russian radio-jammers that interfere with the GPS systems Ukrenergo also relies on to manage the flow of power in Ukraine.

The stealthy operation, which was described to CNN by sources inside Cisco, Ukraine and the US government, is the latest example of how the Biden administration has leaned on US corporations to help defend Ukraine while trying to keep Washington at arm’s length from a direct confrontation with Russia.

SpaceX has provided satellite coverage used by the Ukrainian military. Microsoft helped move huge Ukrainian government data centers out of the country ahead of the invasion. The CEO of Denver-based data analytics firm Palantir has boasted that the firm’s software has been used for “most of the targeting” by the Ukrainian military in Ukraine.

Officials from multiple US agencies played a quiet role in getting the Cisco equipment into Ukraine, sources say. The Pentagon handled the flights, the Department of Energy helped coordinate the equipment’s delivery, and, according to Ukrenergo, the Department of Commerce arranged crucial meetings earlier this year between a handful of US tech executives and managers with Ukrenergo who were eager for new ways to defend their grid from Russian attacks.

Over dinner at an upscale steakhouse near Stanford University in February, Ukrenergo executives shared war stories with their contacts at Cisco, which has done business in Ukraine for years.

Ukraine’s grid operators were facing a serious but underreported problem, they told their dinner companions: The constant GPS jamming that both the Russian and Ukrainian militaries use to interfere with guided missiles was also disrupting visibility for Ukraine’s power grid operators, who relied on GPS-based clocks to relay information about power flow from one location to another.

Sitting at the table that night was Joe Marshall, a veteran researcher at Talos, Cisco’s cyber-intelligence unit, who listened intently as the Ukrainians explained their problem over steaks and drinks. Marshall has spent years protecting electric power systems in Ukraine and elsewhere from sabotage, but he’d never dealt with a problem like Ukrenergo’s.

After dinner, Marshall went back to his hotel and racked his brain for a potential solution.

“Time was a factor,” he said. “These were people’s lives we were discussing here.”

Marshall spent hours watching YouTube videos posted by an electronic-warfare expert, and also got tips from US officials and industrial cybersecurity experts at Cisco and elsewhere.

As the world’s largest maker of computer networking equipment, Cisco had resources to spare. Marshall and his team of more than a dozen engineers got to work molding a very common piece of equipment, called an industrial ethernet switch, to fit the specific needs of the Ukrainian grid.

Cisco estimated the cost of building materials and shipping of the switches to be $1 million, but the company said it donated the equipment to Ukrenergo for free.

Taras Vasyliv, who oversees power dispatching for Ukrenergo, likened the custom-built switches to a “flashlight” for a surgeon who is trying to operate in the dark.

The switch allows an electric substation – which has the crucial task of converting power from high to low voltage – to communicate with other parts of a power grid. Critically, these switches needed to be outfitted with their own internal clocks that could calculate accurate time measurements, providing an element of redundancy and giving grid operators visibility even when GPS systems are down.

Otherwise, “you’re blind,” Vasyliv said in a phone interview from Kyiv.

Several of his colleagues have been killed during the war, Vasyliv told CNN, as the Russian military has pummeled Ukrenergo infrastructure. But keeping the lights on, and avoiding the next air strike, keeps him going.

“Just do your job, and do it very good,” he says he tells himself.

Within a few weeks of the dinner in Silicon Valley, Marshall and his team had a prototype developed. To see whether it actually worked, Cisco had to figure out how to get them into Ukraine.

Marshall, a former Pentagon IT contractor from Alabama, turned to a US official to find a flight that was leaving from a military base on the East Coast in April. The flight went to Germany before arriving in Rzeszów, Poland, a hub for humanitarian and military support about 60 miles from the Ukrainian border.

From there, the prototypes were loaded onto a train to go into Ukraine, where they were quietly delivered to Vasyliv and his team of Ukrenergo engineers.

With their offices in Kyiv partially destroyed by shelling, Vasyliv said his engineers tested the switch in a drab office in western Ukraine.

“This looked like the startups in California from 1970 [rather] than some very fancy laboratory,” he said.

The switches worked, and Cisco ramped up production so that dozens more could get to Ukraine.

US officials familiar with the Cisco project were reluctant to discuss specific shipments out of fear of tipping off Russia’s ability to thwart them. The same GRU cyber-sabotage team that has cut the lights in Ukraine, after all, previously damaged computers at logistics companies in Poland that were servicing Ukraine, according to Microsoft.

But over the course of three months last winter, the Department of Energy “identified, procured and shipped” nearly 20 tons of electrical equipment to Ukraine on US Air Force cargo planes, the department said in February.

Behind the scenes, US officials are often coordinating the delivery of key technology to Ukraine. The US Department of Defense is now paying SpaceX to provide its Starlink satellite service in Ukraine, the department said in July, without disclosing the price of the contract.

US officials charged with protecting the US electric sector have also been studying Russia’s digital sabotage of Ukraine’s grid for close to a decade – to help Ukraine but also to ensure US power companies know how to defend against the hacking techniques.

When the GRU first used hacking tools to cut power for about 225,000 Ukrainians in the winter of 2015, according to a US indictment and private experts, the Department of Homeland Security flew a team to Ukraine to study the forensics of the attack. Another power-disrupting cyberattack in Ukraine in 2016 showed the Russians were evolving their techniques.

On October 10, 2022, the GRU targeted an unnamed Ukrainian electric facility, “causing an unplanned power outage” at the same time the Russian military launched air strikes on electric infrastructure across Ukraine, according to US cybersecurity firm Mandiant, which responded to the hack. The extent of any power outage from the hacking was unclear. Ukrainian officials have told CNN it can be difficult to distinguish whether air strikes or hacking causes an outage.

But the incident raised the possibility that the Russian hacking unit was getting quicker at developing new tools to disrupt power in Ukraine, hastened by the tempo and demands of war.

That cyberattack last year in Ukraine “demonstrates the evolution of improved and faster [operational technology] threat capabilities that could be leveraged in North America,” NERC, the North American grid regulator, said in a statement to CNN, referring to cyber capabilities that target industrial equipment.

At least one of the Department of Energy’s elite research labs – which spend millions of dollars annually anticipating new hacking threats to the US grid – will be closely studying the methods the GRU used in the October 2022 hack in Ukraine, sources familiar with the matter told CNN.