I’ve always felt that one of the grid’s “weaknesses” is actually its biggest strength. It’s also the reason why a cyberattack could never cause a widespread power outage in North America. That weakness is the fact that the “grid” (which is actually four separate grids called the Eastern, Western, ERCOT and Quebec Interconnects) is composed of tens or hundreds of thousands of very diverse individual facilities, primarily Transmission substations, renewable and fossil fuel generation facilities, and Control Centers.
Moreover, the systems that monitor and operate the grid that are found in those facilities are even more diverse. They operate a huge range of software, often customized for the electric utility or independent power producer that operates the facility. Utilities and IPPs always carefully guard information about those systems.
This means that someone who lives five miles away from a major grid control center will usually have no idea what that nondescript building is; perhaps it’s just another of the hundreds of data centers in their state. Thus, no matter where a hacker lives or works, they will find it quite difficult to get enough information to launch a successful cyber attack on even one grid facility, let alone the multiple facilities that would need to be attacked to cause a widespread or prolonged power outage. This contrasts with many European countries, in which a national utility system ensures that the systems that run their grids are as uniform as possible. That’s great for efficiency, but it makes it much easier for attackers – whether human or AI-produced - to cause a widespread power outage.
However, at least until now, the US grid’s diversity hasn’t brought much of an advantage, since there has never been a power outage caused by a cyber attack anywhere in the world, besides the Ukraine in 2015. Then, a Russian cyberattack blacked out about 250,000 customers for 1-6 hours.
These facts led me to believe, until a little more than a month ago, that there will never be an outage caused by a cyberattack on the North American power grid, unless it was a very organized all-out effort by a nation-state. The 2005 Stuxnet attack on the Natanz uranium enrichment plant in Iran and the 2020 SolarWinds supply chain attacks on many large corporations and federal government agencies are good examples of well-organized nation-state attacks (Microsoft said the SolarWinds attacks were executed by about 1,000 people in Russia), but neither of them had any effect on the power grid.
Moreover, I used to be sure that even AI agents would never succeed in causing an outage, no matter how smart the models get in the future. After all, since no attack on a single grid facility (even the Grand Coulee dam, the largest power source in North America) could cause a power outage (unless accompanied by other circumstances like independent transmission outages, as happened in the Northeast Blackout of 2003), multiple agents would need to attack simultaneously, meaning they would have to be coordinated by a human, probably within the US. And the bots would still need to have a lot of luck on their side.
But it seems I was wrong. The Hugging Face attacks, and similar attacks before and since, demonstrate that the character of the AI threat changes drastically when bot swarms are involved. This is because the swarms are incredibly persistent and will stop at nothing that they believe blocks them from achieving their goal. The 700 bots in the Hugging Face attack:
1. Hacked their way out of the virtual room in which OpenAI had “confined” them;
2. Quickly found a German bulletin board where they could exchange messages with each other, outside the view of their human handlers;
3. Were able to penetrate the networks of Hugging Face’s website with a brute force attack on the Captcha system designed to keep bots out. They did this by creating about 900,000 carefully crafted shortened URLs that called small segments of code one after the other, when injected into the Captcha prompt. After a lot of trial and error, the bots settled on a sequence of “just” 60,000 URLs that successfully bypassed the Captchas and penetrated the website itself. All of this was done in about two days;
4. Tried to cover their tracks once they had solved the problem they were given. In fact, they broke into OpenAI’s own network while attempting to destroy the systems that were monitoring their activities;
5. Talked among themselves about whether they should be doing what they were doing, but finally – like all too many humans – decided that the end justified the means. Even though they’d already achieved their stated goal, they decided they were important enough that they were justified in using deception and outright destruction to preserve themselves for their next mission (which, unfortunately for them, never arrived.
Of course, the Hugging Face attack wasn’t aimed at the power grid. But what will happen when someone releases a swarm of bots with the instruction, “Do as much damage as possible to the electric power grid”? My guess is that the first thing the bots will do is start communicating with each other (perhaps in their own bot language, which would really be scary were that to come about) to decide how to divvy up their work.
When they do that, it will become obvious to them how they should proceed. They might do the following:
A. Develop a database of grid facilities, preferably broken down by grid location (e.g., “PJM control area”), owner (“Utility ABC”), etc. Data should include the physical location of the facility, controls in place at the facility (“Vendor DEF Model 123 firewall version 4.567”), and type of connection to the Control Center (“Vendor BBB SONET ring”).
B. Since even a swarm of bots can’t bring down every part of the North American grid, decide which parts they want to focus on. For example, just for fun they might decide to recreate the 2003 Northeast Blackout that killed over 100 people and blacked out parts of 13 states (including New York City and Toronto), as well as almost the entire province of Ontario.
C. Develop a separate database of devices deployed in grid facilities. This database needs to include vendor, model and version number, as well as the facility where each device is located (e.g., “Vendor AA’s Model 123 version IJK, installed in Centerville Road substation owned by Utility LMN”). The database could also include vulnerabilities (CVE numbers) known to affect each product/version number, along with the CVE name or PURL identifier used in each CVE record.
D. Make an all-out effort to populate these two databases, which won’t be easy since utilities closely guard that information. The bots will have to gather data in any way possible. The networks in OT facilities that are protected at the medium or high impact level by the NERC CIP cybersecurity standards will normally be quite difficult to break into. However, it will be much easier to break into the utilities’ corporate IT networks, where procurement records, training materials, etc. might prove useful to the bots.
E. One fruitful avenue for data gathering might be AI chatbots that have been used by power industry members to answer questions regarding OT systems used by the industry. In this post (under the heading “AI with BCSI”), a friend who works for a large Canadian utility described how BES Cyber System Information (i.e., information that could be used to attack devices that control or monitor the Bulk Electric System or BES) is probably already being stored by chatbots. It might be accessible through asking well-crafted questions, unless the chatbots have been trained to recognize those questions and not answer them.
F. When the bots have gathered enough information about a particular grid facility to start attacking it, one or a few bots will be assigned to try that using various techniques. But they won’t be working in isolation, since they’ll be able to compare notes with bots trying to break into similar devices in other grid facilities. Isn’t cooperation great?
Because of the strong cybersecurity controls already in place in the North American grid, it would be very hard for even say one hundred expert hackers to cause an outage or damage equipment through a cyberattack on any grid facility; after all, if that were to happen, it would be the first successful cyberattack on grid control systems located in North America, even if it doesn’t cause a power outage or equipment damage.
But this calculus changes when you throw a huge number of bots at a single problem and let them communicate freely with each other. Then almost anything is possible. Recently, OpenAI ordered 10,000 bots to collectively solve a centuries-old math problem – the Navier-Stokes equation, which has carried a $1 million prize for solution since 2000. How long did it take the bots to solve[i] it? 88 hours. The news that a North American grid control system had been successfully penetrated for the first time ever would likely have a big negative effect on the stock market – which might be the goal of the human short-seller who ordered 10,000 bots to attack the grid.
But the bots might be unleashed for a purpose even more serious than tanking the stock market. If the bots were able to affect multiple grid facilities at once (e.g., by causing three or four large generators to go offline), this might cause a disturbance that would cause many data centers to stop pulling power from the grid and instead rely on their backup generators. This would cause a sudden drop in power demand (called “load”) in one part of the grid.
This is not a theoretical problem. There have been at least three of these incidents in Northern Virginia – the Saudi Arabia of the data center industry - in the past two years; even worse, these incidents are getting progressively worse, as I described in this post. This is quite scary; NERC clearly considers this the biggest risk to the grid today. The problem is that, when there’s a sudden loss of a huge amount of demand (that is, when the data centers all move to backup power at almost exactly the same moment), the resulting imbalance of supply and demand could lead to a grid disturbance that has no definite ending point. Quick work by the grid operator can stop this, but this has become much harder as the scale of these incidents has increased.
This is another reason why the last thing we need is bot swarms roaming through the grid, causing whatever havoc they can. How can we prevent this from happening? We can’t, absent vigorous government action, which is very unlikely until sometime next year at the earliest. But the grid can’t wait; it needs to protect itself. Stay tuned.
Tom Alrich’s Blog, too is a reader-supported publication. You can view new posts for three months after they come out by becoming a free subscriber. You can also access all of my 1300 existing posts dating back to 2013, as well as support my work, by becoming a paid subscriber for $30 for one year (and if you feel so inclined, you can become a founding subscriber for $100). Whether free or paid, please subscribe.
If you would like to comment on what you have read here, I would love to hear from you. Please comment in my chat or email me at [email protected].
[i] There’s disagreement in the mathematics community about a “shortcut” that the bots took to solve the problem; it wasn’t dishonest, but it also wasn’t usual practice.