It’s been 13 years since the meticulously planned and executed 20-minute sniper attack on the Metcalf substation in California disabled 17 massive transformers. This article in the New York Times describes the attack and its implications, especially in light of the current huge increase in transformer demand driven in large part by – what else? – the buildout of new data centers.
I want you to read the article for yourself, so I won’t try to summarize it. However, here are my most significant takeaways from the article:
1. Early in the article, the reporters point out that “Electrical engineers had long assumed that the patchwork nature of the grid would protect it from failure.” Fortunately, the grid was protected from failure in this attack, since there was never any outage. The grid’s built-in redundancy, along with human grid controllers who rehearse continually for contingencies like this, worked as intended. That’s good, because no group ever claimed responsibility for the attack and the perpetrators were never identified.
2. However, the article goes on to describe why the lack of an outage in this case is cold comfort: a coordinated attack like Metcalf on a small number of transmission substations in each of the three US Interconnects – Eastern, Western, and ERCOT (most of Texas)– could lead to literally a total US blackout of the continental US. This was revealed to us in 2014, when Rebecca Smith of the Wall Street Journal wrote a widely-quoted article about a FERC memo that probably shouldn’t have been written. I didn’t know Rebecca at the time, but we became good friends when she published an even-more-explosive story about Russian penetration of the US grid (which still has never been officially investigated, even though other government sources have implied this happened). She passed away far too soon in 2023 and I miss her terribly.
3. Fortunately, the response to the Metcalf attack has gone a long way to mitigate the risk. NERC developed and rapidly enforced CIP-014, which is so far the only NERC physical security standard.[i] Not only did NERC entities (mainly electric utilities and independent power producers) move as quickly as possible to comply with the standard, but in many cases they over-complied (even though compliance for just one substation was very expensive).
4. That is, a lot of substations that weren’t in scope for the standard were treated as if they were.[ii] While it was up to each NERC entity to make the final decision on how to mitigate the risk of such an attack, most entities who complied put up ballistic barriers around the entire substation, as well as greatly increased monitoring capabilities (one problem at Metcalf was that most of the security cameras faced inwards to catch copper thieves – which were until 2013 considered to be the biggest threat to a substation. Besides squirrels, of course).
5. However, it’s too early for “Mission Accomplished” signs. Near the end of the article, the reporter describes his own on-site investigation, including his finding that there are multiple low hills close to the Metcalf substation where it is still possible to get an unobstructed sight line – from about 150 yards away - to the transformers that were the targets of the snipers. You might wonder why substations are always built in the open. This is because the transformers generate a lot of heat that needs to be dissipated, which is unfortunate since the advent of cheap and destructive drones means almost nothing exposed to the open air is completely safe. We must live with the fact that the only way we can limit sniper and drone threats to substations is to move all transmission substations underground, where they will be cooled with sulfur hexafluoride. Of course, the cost of doing that would be tremendous, and it’s usually only done in very crowded cities.
The second part of the article discusses the even bigger problem that was made clear by the Metcalf attack. It starts with this passage: “It was hard to replace transformers quickly back then; it’s nearly impossible now. Since 2021, the average wait time for a large power transformer — the kind you’d find at a major substation, capable of handling tens or hundreds of thousands of volts — has climbed from less than a year to 128 weeks, with some lead times growing as long as five years.” The article goes on to describe how serious the consequences would be from a widespread and prolonged outage (and three days is “prolonged”), especially if it includes a major city.
For a highly readable book that describes in vivid detail how unprepared the US is for such an outage, I recommend you pick up Ted Koppel’s book, “Lights Out”. It’s dated, but still very relevant. But don’t be fooled by the blurb that says the book is about what a cyberattack could do the US grid – it’s about what would happen if there were a widespread and prolonged outage due to any cause. Of course, such an outage could be caused by a physical attack like Metcalf, although I think unleashing swarms of drones to simultaneously attack key substations would be more likely to succeed today. There’s also the very remote (but nonzero) possibility of an EMP attack, which could literally destroy the entire grid by frying transformers and other equipment that would take years to replace. But probably the most likely cause of a widespread and prolonged outage would be a superstorm like Sandy, only bigger.
Ironically, the one type of grid threat we don’t have to worry about – besides a zombie apocalypse – is a cyberattack that brings down all three Interconnects in the US, or even just the smallest of the three, ERCOT. This simply.can’t.happen.
While I think the Times reporter did an excellent job, I want to point out that he fell for a lie that’s been told repeatedly since 2020. It’s in this paragraph:
Disabling nine substations simultaneously might be beyond the reach of a domestic terror group, but it is well within the capabilities of a state actor. For years, intelligence analysts have warned that the grid represents an inviting “attack surface” for adversaries including China, North Korea, Russia and Iran. Already, China has been caught installing a software backdoor in a large power transformer being shipped to a federal utility in Colorado. The Pentagon has warned that the Chinese could target critical infrastructure specifically to “undermine the will of the U.S. public.”
The sentence in red is a complete fabrication that keeps resurfacing in the media; it was dreamed up by a very well-known ICS security expert in 2020, who unfortunately has come to believe that the best way to promote his business is to scare the public with lies and get them repeated again and again by the media. The fact is that transformers operate entirely according to the laws of physics. They don’t have a microprocessor at all, so hacking a transformer would be as difficult as hacking a tree or rock in the forest (transformers can have add-on devices, usually from different manufacturers, that do have microprocessors - including load tap changers and dissolved gas analyzers. But those devices don’t control the transformer itself).
I’ve been playing whack-a-mole with this lie for years, including here and here, but it refuses to die. In fact, last year I received two or three emails from college students who had a writing assignment on a topic that sounded something like “In light of the recent successful cyberattack on a large transformer, what can be done to protect transformers from cyberattacks?”
To summarize, if you insist on worrying about a widespread grid disaster, you should worry about a widespread physical attack on substations - Metcalf on steroids – or a huge superstorm. What’s not likely at all is a successful cyberattack on transformers or the zombie apocalypse. You don’t have to lose sleep over either of those.
Tom Alrich’s Blog, too is a reader-supported publication. You can view new posts for three months after they come out by becoming a free subscriber. You can also access all of my 1300 existing posts dating back to 2013, as well as support my work, by becoming a paid subscriber for $30 for one year (and if you feel so inclined, you can become a founding subscriber for $100). Whether free or paid, please subscribe.
If you would like to comment on what you have read here, I would love to hear from you. Please comment in my chat or email me at [email protected].
[i] CIP-006 deals with physical security of control systems but isn’t intended to protect the assets where the systems are located: Control Centers, transmission substations and generating facilities.
[ii] The article doesn’t mention this, but I know it to be a fact.