Yesterday morning, I read a really disturbing article in the New York Times about the Hugging Face attack, which makes me completely rethink the risks posed by both generative AI and the cloud. I’d like you to read the article, but here are my main takeaways from it:
1. Two months before the attack, OpenAI trained a group of AI agents to collectively seek answers to some cybersecurity problems, and to do so aggressively.
2. But OpenAI didn’t anticipate how collaborative and aggressive those agents would be. In their pursuit of the answers, the agents broke out of the individual “pens” designed to keep them from the internet. They broke into Hugging Face’s systems and later into OpenAI’s own systems. At first, they were just trying to find answers to the questions, but they then “decided” they needed to cover their tracks by changing logs, etc. And for good measure, they finally tried to destroy the systems that had been put in place to monitor them.
3. In fact, the result was very much like what happened to Mickey Mouse, the Sorcerer’s Apprentice in Fantasia, when he didn’t pronounce a magic spell correctly. A single broom multiplied itself many times over, and each new broom started bringing in buckets filled with water and dumping them on the floor that Mickey was supposed to clean. Mickey would have drowned, but the timely intervention of the sorcerer restored order and cleaned up the mess, although Mickey’s bottom (and ego) no doubt hurt for weeks because of the sorcerer’s displeasure with him.
4. Unfortunately for OpenAI (and perhaps for all of us), when the bots got out of hand and started attacking both Hugging Face’s systems and OpenAI’s own systems, there was no sorcerer to make everything better again; they succeeded in causing some real damage. But most importantly, the word is now out all over the internet that AI bots can succeed at almost any task if they collaborate and are aggressive; of course, bots are avid readers and will certainly learn about this.
5. Moreover, it’s not likely there’s some sort of magic software bullet that will prevent bots from behaving like this again. All future bots should be assumed to have been exposed to this knowledge.
The article concludes:
Given how little we know about these multi-agent swarms, the Hugging Face hack may have been a gift, a warning shot, as some have suggested, that gives A.I. companies a chance to study the group dynamics of these systems while the stakes are still relatively low. This time, the A.I. collective didn’t seize a military network, hack a hospital or shut down an electrical grid. This time, humans regained control.
Next time, we might not be so lucky.
Of course, this can’t be the end of the story. We can’t just hope we’ll be lucky when the bots come for a military network, a hospital, or the power grid.
Fortunately, I think there is a fix for this problem, which the bots will find impossible to overcome. It’s been written about by cybersecurity experts focused on the power grid (and other critical infrastructure), including Andy Bochman of Idaho National Labs.
The only way to keep the bots out from now on is to stop making efficiency the number one goal of anything we design. Instead, we should include “firebreaks” that can’t be bridged by anything (anyone)? other than a human being. A familiar example of this is authentication. Once upon a time, having a strong password was considered good security, but it’s very hard to make that case now – plus, once quantum computing gets off the ground (which will probably not be soon, but is probably inevitable sometime in the future), any password at all will be useless.
Multi-factor authentication certainly improves the odds for defenders, but like any completely automated system, MFA will ultimately prove easily hackable – especially when the roving bot gangs turn their “eyes” to it.
Here’s something that might be a barrier to the bots: In order to be authorized to use a particular secure network, a user needs to call a secure phone number and answer – for a real person - a large set of randomly chosen questions like the name of their first romantic interest, their favorite breakfast food, the first thing they usually do when they start working, etc. They only need to provide answers that they’re sure of.
The user’s answers to all questions will be written down by hand and stored in a locked storage cabinet; the user can write their answers down as well and securely store them, but they will be forbidden to store them on any computer system. When the user wants to use the secure system, they will call the same number and be asked 5-10 of the questions they answered. If they can’t remember more than two, they will need to restart this process at the beginning. And even when the user has initially been authenticated by this process, they will have to re-authenticate periodically, perhaps every 1-2 hours.
Of course, this is quite an intrusive process. It certainly won’t be necessary for most systems, even for some systems and networks used by the military, hospitals and the power grid. But in all three of these areas as well as others, there are some systems and networks that require this level of protection.
However, even those systems and networks can be protected from the bots, as long as they’re not directly accessible from the internet, and as long as they’re not concentrated in one physical location. Since I know a fair amount about security of the power grid, I can attest there are three things that have so far protected the North American power grid from a single power outage caused by a cyberattack:
1. The NERC CIP standards, while certainly not perfect, require very strong controls (at least for “high and medium impact” systems) that prevent any network that includes even one system whose compromise or loss would have an immediate effect on the reliability of the Bulk Electric System (BES) – called a “BES Cyber System” – from being directly connected to the internet, or even the utility’s IT network.
2. The North American power grid has huge diversity, since each utility and independent power producer (IPP) connected to the grid chooses its own control systems, and configures them in a way that makes sense for them (consonant with the CIP standards, of course). Any cyber attack that causes real harm will have to compromise multiple high-voltage Transmission substations and/or Control Centers, and produce a prolonged outage (perhaps a day or more). Currently, the biggest threats to grid reliability – besides data centers – are squirrels and copper theft, which of course are never coordinated.
3. Today, it is impossible for a NERC entity (utility or IPP) to locate BES Cyber Systems in the cloud, unless they wish to be out of compliance with about 70 NERC CIP Requirements and Requirement Parts (subrequirements). Since violating just one requirement can result in a substantial monetary penalty, this isn’t an acceptable risk for any NERC entity.
You might think that the fact that high and medium impact systems subject to NERC CIP compliance can’t be installed in the cloud today is the result of the great foresight of the people who designed the current CIP standards – but you would be wrong. The reason this is the case is that when the foundation for the current standards was laid in 2011 and 2012, the cloud was considered to be an experimental technology that no electric utility would ever consider to be safe to use for their OT systems; therefore, the definitions of the terms used in the current CIP standards assume that all systems are on premises. The result of this is the compliance snafu just described.
Today, there is a lot of interest in the NERC community in allowing full use of the cloud for OT systems; but there are also many NERC entities that don’t want to put any of their OT systems in the cloud and don ’t think any other NERC entities should do that. Until yesterday, I thought the latter people were being overly cautious. And regarding some systems in the cloud that can’t affect the power grid if compromised (e.g., physical or cyber security monitoring systems that can only monitor and alert but can’t change anything in the real world), I still think they’re overly cautious.
However, regarding BES Cyber Systems – which by definition will affect the grid if compromised – I think the NERC community should put on hold the idea of making it possible to use those in the cloud, at least for the time being. I’ll (hopefully) discuss the question of power grid control systems in the cloud in my next post.
But what about other critical systems, besides those that control the power grid? How about systems that control hospitals, banks, military operations, large factories, etc? I suspect it’s too late to even talk about keeping those systems from moving to the cloud; instead, the question is whether they can safely stay in the cloud, now that the bots have woken up to what they can do.
This would be a bad movie script, if it weren’t real.
Tom Alrich’s Blog, too is a reader-supported publication. You can view new posts for three months after they come out by becoming a free subscriber. You can also access all of my 1300 existing posts dating back to 2013, as well as support my work, by becoming a paid subscriber for $30 for one year (and if you feel so inclined, you can become a founding subscriber for $100). Whether free or paid, please subscribe.
If you would like to comment on what you have read here, I would love to hear from you. Please comment in my chat or email me at [email protected].