Feature It was a sunny morning in late April when a massive power outage suddenly rippled across Spain, Portugal, and parts of southwestern France, leaving tens of millions of people without electricity for hours.
Cities were plunged into darkness. Trains stopped and metro lines had to be evacuated. Flights were cancelled. Mobile networks and internet providers went down. Roads were gridlocked as traffic lights stopped working.
It took 10 hours for power to be restored and 23 hours before the entire national grid in Spain was back up and running, with the incident being deemed the [most severe blackout to have affected Europe](https://eepublicdownloads.blob.core.windows.net/public-cdn-container/clean-do…
Feature It was a sunny morning in late April when a massive power outage suddenly rippled across Spain, Portugal, and parts of southwestern France, leaving tens of millions of people without electricity for hours.
Cities were plunged into darkness. Trains stopped and metro lines had to be evacuated. Flights were cancelled. Mobile networks and internet providers went down. Roads were gridlocked as traffic lights stopped working.
It took 10 hours for power to be restored and 23 hours before the entire national grid in Spain was back up and running, with the incident being deemed the most severe blackout to have affected Europe in the last two decades.
Is critical infrastructure prepared for OT ransomware?
This incident was not caused by a cyberattack, but by a series of complex cascading failures where power generation components disconnected at the same time that multiple overvoltages simultaneously occurred, overwhelming the national power grids (human error is probably involved too, but the grid operators and power plants are still pointing fingers at each other).
However, the Spanish power outage brings back unpleasant memories of the devastating cyberattack in 2015 that took down Ukraine’s electric grid for six hours, which was traced back to Russian online attackers.
Most worryingly, it has shown how delicate the balance is when it comes to keeping national grids stable, and how failures in one country in Europe can cause an instant domino effect in neighboring nations reliant on energy imports.
Nick Haan, the field chief technology officer for strategic partners at international industrial operational technology (OT) systems security vendor Claroty, told The Register:
“Currently, incident handling across Europe’s power sector is too fragmented. Every operator and country has its own way of running things, which makes coordination difficult when things go wrong. The European grid is unique in the world given how tightly connected it is. One disruption can spread across borders within minutes.”
The 2015 Ukraine power grid attack was the first serious blackout to be attributed to an online attack, but it is unlikely to be the last, given that there has been a steady ramp-up in attacks against utility companies across the Western world, which are built on an ecosystem that prioritizes just-in-time, on-demand, and decentralized power generation.
Cyberpunks mess with Canada’s water, energy, and farm systems
Recent cyberattacks have revolved around ransomware affecting financial systems, but there is a serious risk that criminals and nation-state attackers could either incidentally, or deliberately, bring down substations or halt fuel supplies, such as in the case of the Colonial Pipeline cyberattack in May 2021.
White hair-inducing IT infrastructure
The picture gets even worse when you take a look inside power plants at their IT infrastructure – a sprawling, complex mishmash of random software, aging hardware and a multitude of operating systems controlling different bits of equipment supplied by a variety of vendors, none of whom want cybersecurity teams taking a closer look inside.
For instance, within a single gas turbine, you can usually find up to seven different systems, and each of these systems controls about 10 devices, all of which come with their own separate IP address. Or you could have a substation being controlled from many miles away by a control room. If someone were to attack it, you’d get an alert, but you might not be able to do anything to stop it.
Power stations need a shared language for incident response which covers common processes
Back in 2015 when I covered the fallout from the Ukraine power grid attack, experts told me that at some sites they had visited, there were IBM machines covered in dust dating back to the 1970s. That sounded terrible. Maybe modern viruses wouldn’t work on them, but there’s a reason the world isn’t using retro tech.
The picture today is even more frightening. The experts I spoke to for this article said they’ve seen “very exotic operating systems” integrated inside supervisory control and data acquisition (SCADA) systems in substations, everything from Windows XP, Windows 7 and Windows NT4, to BeOS (a failed 1990s operating system) and 30+ year-old networking software GE JungleMUX, which was recently discontinued.
“In many cases, they’re still using dial-up [internet] in really rural areas, and the most worrisome thing is that many of these control systems rely on protocols that are insecure, like DNP3, for example, which has no security control, no access controls, no data encryption, no privilege management, no keys or passwords,” said Roman Arutyunov, the co-founder and senior vice president of product at Palo Alto-based cybersecurity software firm Xage Security.
DEF CON hackers plug security holes in US water systems amid tsunami of threats
“So you end up with systems that will literally take any command that’s sent to them and act upon it. It’s very easy to compromise substations, and what [power stations] rely on [as a defense] today, is to close them up, because once a malicious actor is inside a power station or substation, it’s too late. It’s gone.”
He added that it’s a complex problem operating a power station – being stuck with vendor lock-in isn’t great, but at the same time, vendors guard their equipment fiercely because they’re responsible if something goes wrong.
However, this makes improving a power plant’s cyber defences a frustrating and often thankless task, let alone digital transformation. A solution is sorely needed.
This headache is one the European Commission is focused on. It is funding several projects looking at making electric grids more resilient, such as the eFort framework being developed by cybersecurity researchers at the independent non-profit Netherlands Organisation for Applied Scientific Research (TNO) and the Delft University of Technology (TU Delft).
TNO’s SOARCA tool is the first ever open source security orchestration, automation and response (SOAR) platform designed to protect power plants by automating the orchestration of the response to physical attacks, as well as cyberattacks, on substations and the network, and the first country to demo it will be the Ukraine this year. At the moment, SOAR systems only exist for dedicated IT environments.
The researchers’ design includes a SOAR system in each layer of the power station: the substation, the control room, the enterprise layer, the cloud, or the security operations centre (SOC), so that the SOC and the control room work together to detect anomalies in the network, whether it’s an attacker exploiting a vulnerability, a malicious device being plugged into a substation, or a physical attack like a missile hitting a substation.
The idea is to be able to isolate potential problems and prevent lateral movement from one device to another or privilege escalation, so an attacker cannot go through the network to the central IT management system of the electricity grid.
Ransomware scum have put a target on the no man’s land between IT and operations
“If you look at the OT world, it’s mostly about availability – so electricity should always work, but in the IT world, confidentiality and integrity is more important,” Reinder Wolthuis, senior project manager and consultant, cybersecurity for TNO, said at the One Conference in The Hague.
“That’s why we have developed an interaction between the SOC and the control room, so the SOC asks for the authority to switch off these machines, and the control room approves it. But if the SOC detects a cyberattack in a substation, then the control room can do a real time digital modelling. It can do simulations to see what would be the impact and that is then reported back to the Security Operating Centre (SOC).”
The SOARCA tool is underpinned by CACAO Playbooks, an open source specification developed by the OASIS Open standards body and its members (which include lots of tech giants and US government agencies) to create standardized predefined, automated workflows that can detect intrusions and changes made by malicious actors, and then carry out a series of steps to protect the network and mitigate the attack.
Would this system be useful?
Experts largely agree the problem facing critical infrastructure is only worsening as years pass, and the more random Windows implementations that are added into the network, the wider the attack surface is.
There is a tendency for power plants and grid operators to stick their heads in the sand, perhaps due to a perception that what’s happening in Ukraine isn’t going to happen to them, but they do need to take the threat more seriously, the experts added.
One cybersecurity expert, who is researching critical infrastructure in Ukraine and does not wish to be named, told us that things have changed since the start of the Ukraine-Russia conflict – now power stations have built “redundancies, within redundancies, within redundancies” of additional cables between substations, and therefore it is difficult to overwhelm the whole national grid when a missile attack hits a substation. Things are playing out in a different way to how Russia and other governments thought they would.
This is the FBI, open up. China’s Volt Typhoon is on your network
“This SOARCA tool would definitely be useful with just-in-time networks in Canada, the US, parts of Europe and even Australia and New Zealand. But Ukraine? Not really. And even if the grid were to go down for a few hours, we’re in a war. Who cares?”
Ukraine’s state-owned power grid operator JSC NEK Ukrenergo, which will be simulating the tool on a digital twin of its grid, said it sees many benefits to implementing such a system into their national grid. However, JSC added that it probably wouldn’t be implementing the tool anytime soon: “Even in ‘peaceful times’, a deployment of this nature would require capital investment, staffing, training, perhaps new hardware or integrated systems, and ongoing maintenance.”
That’s quite an honest answer, and its views are probably echoed by much of the global energy industry.
“There’s obviously room for improvement, but there’s always going to be a cost benefit analysis. In the UK, there are two industries that have always been terrible at adopting new technology – the national grid and healthcare,” said Sam Barker, vice president for telecoms market research at Juniper Research.
“When it comes to the smart grid networks and smart grid operators, they have often been slow to adopt new technologies that not only make the grid more efficient, but can also lower costs, and that may be down to issues such as vendor lock-in. [Plus] there’s different systems, and unifying them is going to be a big challenge.”
Han tolds us: “What’s missing is a consistent pan-European approach to crisis management. Power stations need a shared language for incident response which covers common processes, communication standards, and escalation paths. This isn’t just for cyber incidents either. Without uniform standards, we’ll keep relying on ad hoc cooperation in moments when time is critical.”
“We need to see change come through smart legislation and regulation that enforces baseline standards across the continent.”
Iran-linked crew used custom ‘cyberweapon’ in US critical infrastructure attacks
Claroty tries to help power plants work out exactly what their IT infrastructure looks like and inventories just how bad the state of their cybersecurity patching is, and then adds monitoring tools to the network to sniff packets and detect anomalies in device behavior.
Haan also advises his clients to renegotiate vendor contracts when they come up for renewal, to enable cybersecurity oversight on these proprietary systems.
More standards and regulations will help
TNO’s Wolthuis said the energy industry is likely to be pushed soon to take action by regulators, particularly once the Network Code on Cybersecurity (NCCS), which lays out rules requiring cybersecurity risk assessments in the electricity sector, is formalized.
Bret Jordan, a cybersecurity expert and specifications author for standards relating to networking protocols and cybersecurity, is strongly in favour of more standardization in the cybersecurity industry.
He believes the CACAO Playbooks will really help both critical infrastructure and regular enterprises by dramatically improving threat intelligence, so power stations will in future be alerted to potential threats at the same time that governments know about them.
But it can sometimes be better to allow a nation-state attacker to think they have a foothold in your network, so you can monitor them until you have enough information about the attack, so you want to be able to isolate them, not kick them out.
- CISA roasts unnamed critical national infrastructure body for shoddy security hygiene
- UK’s Sellafield nuke waste processing plant fined £333K for infosec blunders
- FrostyGoop malware shut off heat to 600 Ukraine apartment buildings
- US critical infrastructure cyberattack reporting rules inch closer to reality
- US task force aims to plug security leaks in water sector
- Utility security is so bad, US DoE offers rate cuts to improve it
- EPA flushes water supply cybersecurity rule after losing legal fight with industry, states
“You have to assume that the threat actors are in the network, and they’ve compromised it, then they’re just going to wait until they need to do something. It could be till 2038,” Jordan told The Register.
“We need to understand what works and what doesn’t, and how to prevent, detect, mitigate and remediate, so the combination of threat intelligence with workflows enables you to know what to look for.”
Jason Keirstead, a cybersecurity operations expert and head of engineering at LangGuard.AI, who previously worked on the CACAO specification, agreed: “If we can enable real collective defense – enable organizations to know when one is attacked and share that information in near real time, and have the defenses pushed out to other organizations, it could really move the needle against some of these adversaries.
“The only way to do that is through a combination of standardization and also letting some of the guard down and sharing more information more openly. We should be making the attackers’ lives harder, so they have to change tactics more often.” ®