Who will audit the CSPs?

 

One question that the current NERC “Risk Management for Third-Party Cloud Services” drafting team is discussing a lot is: What actions need to be taken to verify the security posture of the CSPs? By CSP I mean the two or three major “platform” cloud service providers, not other providers, like providers of SaaS and security services that are usually deployed on one of the major CSP platforms.

This question came up before, when CIP-013, the NERC supply chain cybersecurity standard, was developed. In that case, the answer to the question was quite clear: Like all NERC Reliability Standards, the CIP standards only apply to entities that own and/or operate facilities that are part of the Bulk Electric System (BES). Third parties that supply hardware, software and services that go into medium and high impact BES Cyber Systems do not fall under that designation. They do not have to comply with any of the CIP standards directly, but they do have to cooperate with their NERC entity customers, in some cases by providing them with evidence they need to present at audit.

Will the same consideration apply in the case of platform CSPs once the “Cloud CIP” standard(s) are in place (even though those standards are still years from being enforced)? After all, the platform CSPs will furnish not only the hardware and software on which BES Cyber Systems (and other systems like EACMS and PACS) operate, but they will also furnish the entire environment, including full IT staff, in which the hardware and software operate. Will that consideration “push them over the line” into being regulated as NERC entities?

No, it won’t. It’s qsafe to say that the CSPs will never be regulated as NERC entities, even if a significant amount of generation capacity (e.g., DERs) is deployed on their platforms. Platform CSPs also host systems that control pharmaceutical production. Does that mean they should be regulated by the FDA? Or, since farmers’ tractors now depend on cloud data to determine exactly where to drop seeds, should the CSPs be regulated by the Department of Agriculture?

On the other hand, there’s no question that the electric power industry needs to conduct some oversight of the CSPs. How should this be done, if they aren’t NERC entities? The answer in CIP-013 was for the NERC entities themselves to be responsible for vetting the suppliers. The easiest (and by far the most widely used) method for a customer organization to assess a CSP’s security is simply to ask the CSP about their certifications. If the customer likes the names they hear – FedRAMP, ISO 27001, etc. – those will often be all they want to hear.

Of course, the problem with this method is that it hides the details of the certification audit. The audit may have uncovered an issue with, for example, the CSP’s employee remote access procedures, but their customer won’t learn about this unless they ask to see the audit report. A better assessment method is to ask the CSP for their certification audit report and query them about any issues noted in the report.

However, there’s a problem with this second method as well: I’ve heard employees of two huge platform CSPs that are widely used by the electric power industry state unequivocally that although their employers will be pleased to provide access to FedRAMP, SOC 2 Type 2 or ISO 27001 compliance assessment materials, they can’t provide individualized responses to security queries by NERC entities. In other words, the NERC entity will be able to learn about issues that came up during the compliance audit, but they won’t be able to ask about them.

If certification audits always asked every question that’s relevant to a CSP’s level of security, that would be fine, since the audit report would provide a complete picture of the CSP’s security posture. But is this really the case? In short, the answer is no. Consider the following Tale of Two CSPs:

CSP 1

In 2019, one of Platform CSP 1’s customers suffered a devastating breach in which over 100 million customer records were accessed. The attacker was a CSP 1 employee who had been laid off and was out to get revenge. The attacker took advantage of a misconfigured web application firewall and later bragged online that lots of CSP 1’s customers had the same misconfiguration. In fact, the attacker asserted they had been able to penetrate the cloud environments of over thirty CSP 1 customers by exploiting that misconfiguration.

Of course, the platform CSP can’t be blamed for every customer misconfiguration. However, the fact that at least thirty customers of that one CSP all had the same misconfiguration points to a clear problem on the CSP’s part: They need to offer free training to their customers about avoiding this misconfiguration, as well as any other common misconfigurations (in fact, I ended up meeting with CSP 1 later, after I had made this suggestion in one of my posts. They were receptive to what I said, and I imagine they followed up as well, if they hadn’t done so already).

CSP 2

CSP 2’s problem was that they utilized third party access brokers to sell access to a popular hosted service on their platform, but they hadn’t adequately vetted their security. At least one of these access brokers was compromised, leading to the compromise of about 40 of their customers; all the compromises were on CSP 2’s platform.

But there’s more: This may have been a multi-level supply chain attack. Not only did the compromise of the access broker lead to the successful attacks on 40 customers of the access broker, but at least one of those 40 customers later was the victim of a hugely consequential attack. Have you heard of SolarWinds? I thought so. They were one of the access broker’s customers that was compromised, although I don’t think it’s been proven that the attackers got into SolarWinds through the broker.

There are three important lessons to be learned from these two attacks:

1. Both CSPs were up to date on their security certifications.
2. Because the failings of both CSPs should have been discovered in their compliance audits if the right questions had been asked, it’s a safe bet that none of the major security certifications for CSPs addresses either of these failings. Of course, there have been a number of other cloud breaches in recent years whose root causes are also probably not addressed by the major certifications. 
3. Therefore, it would be a mistake to rely on one or more certification audit reports to provide a good picture of the state of security of a platform CSP.

This leaves us with a problem: Since a certification audit report won’t give us a complete picture of a platform CSP’s security posture, something more should be required by the new “Cloud CIP” standard(s). That is, there will need to be a custom assessment of the CSP that will go beyond the set of questions that the certifications ask. However, the NERC entities can’t be required to ask these questions, since I’ve already pointed out that the platform CSPs are not going to answer questions from individual NERC entities. How do we square this circle?

The only viable path out of this problem that I can see is for NERC itself, or a third party engaged by NERC, to conduct the assessment. Specifically:

1. A NERC committee (perhaps the current “Cloud SDT”, although it might be a separately constituted committee) will review records of cloud breaches (like the two above), as well as other vulnerabilities and attacks that are only likely to affect the cloud (and thus are not likely to be included in audits based on standards like ISO27001. Based on this review, they will compile a list of perhaps 10-15 questions to pose to a platform CSP. For example, “How do you vet the security of cloud access brokers that utilize your platform?”
2. Every year, the committee will conduct a new review and update the list of questions.
3. Every year, NERC or a designated third party will get oral and/or written answers from the platform CSPs to the questions on the current list.
4. NERC will summarize the responses received from each CSP and share the summaries with NERC entities that utilize the CSP for OT applications.
5. NERC will not draw from the questionnaire responses any conclusion regarding the suitability of the CSP for use by NERC entities. Each entity will need to decide for itself whether to continue using the services of the platform CSP, based on their questionnaire responses, certifications, news articles, or any other evidence they wish to consider.

There’s only one fly in the above ointment, although it’s a big one: The current NERC Rules of Procedure would never allow what I’ve just described to happen. I would hope (but I can’t be sure) that there will be broad support for making this change to the RoP, but that change will probably need to be made through a separate process; that will take time (probably at least a year). This is why I still doubt that the new Cloud CIP standards will be approved and in effect before 2031 or 2032 – unless some action is taken to accelerate this whole process.[i]

To produce this blog, I rely on support from people like you. If you appreciate my posts, please make that known by donating here. Any amount is welcome, but I would greatly appreciate it if regular readers can donate $20 per year – consider that my “subscription fee”. Thanks!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at [email protected].