Note from Tom:
I have moved to Substack as my primary blog platform. If you want to see all my new posts, as well as my 1200+ legacy posts dating from 2013, please support me by becoming a paid subscriber to my Substack blog. The cost is $30 a year. Thanks!
On Monday September 29, NERC presented a webinar titled "BCSI in the Cloud”. BCSI stands for BES Cyber System Information. The definition of BCSI is quite long, but it is essentially information about a BES Cyber System (BCS) – the fundamental asset type in the current NERC CIP standards - that could be used to locate and compromise it. This is an important subject, because the goal of the CIP standards is to protect BCS. Protecting BCSI is an important component of achieving that goal.
Here's some background information on why a good webinar on this subject was needed:
1. Before about 2018, the general feeling among NERC entities, as well as NERC and FERC staff members, was that it was too dangerous to entrust anything related to the Bulk Electric System to the cloud, including BCSI. However, by 2018 a trend started to develop that has only accelerated today: Increasing numbers of software providers are giving notice to their customers that they were going to move entirely to the cloud, or at least that new and improved features will only be available in the cloud. As a result, NERC entities have increasingly been forced to accept inferior and expensive on-premises software offerings, rather than risk a multitude of CIP violations by deploying systems subject to CIP compliance in the cloud.
2. Another problem, which had been in place since CIP version 5 came into effect in 2016, is that many important new security monitoring services were only available in the cloud; therefore, such services were completely off limits to NERC entities with high or medium impact BES environments. One auditor told me in 2017 that he had to require an entity with high impact Control Centers to stop using one such service – which is very well-reviewed and has a huge worldwide customer base - and go with completely on-premises software. He said it broke his heart to have to do that, since he knew the entity would be less secure because of that change. Of course, NERC entities like that one still can’t use cloud-based security monitoring services.
3. In 2018, NERC decided there were two components of the “Cloud CIP problem”: a) NERC entities with high and medium impact environments not being able to deploy BES Cyber Systems, EACMS and PACS in the cloud, and b) the same entities not being able to store or use BCSI in the cloud. NERC knew the first problem would be much harder to deal with than the second one, so they decided to tackle the second one first. A drafting team worked through the pandemic and came up with a very elegant solution to the BCSI-in-the-cloud problem. Their solution was embodied in a) changes in the wording of two or three Requirement Parts in the then-current CIP-004-6, b) a new CIP-004-7 Requirement R6, c) a revised CIP-011-3 Requirement R1, and d) an unchanged CIP-011-3 Requirement R2.
4. The two revised standards, CIP-004-7 and CIP-011-3, came into effect on January 1, 2024. The purpose of the changes was to allow “storage and use” of BCSI in the cloud, but by far the most important practical change was making it “legal” for NERC entities with high and medium impact BES environments to utilize SaaS (software-as-a-service, i.e. cloud-based software), if the SaaS product needed to use information that meets the definition of BCSI. Two important categories of SaaS that were affected by the CIP issue at the time were cloud-based configuration management and mobile authentication services.
5. Unfortunately, after the two revised standards were implemented, NERC provided almost no guidance (or guidelines, if you will) to NERC entities on how to comply with them, except for endorsing a document as “Implementation Guidance” in December 2023. That document was developed by the NERC Reliability and Security Technical Committee (RSTC) – which has no involvement in drafting new or revised standards - six months previously. This was a curious choice, since the document hadn’t been developed to be official guidance and therefore was not too careful in its use of terms or subject matter. It was a good guidelines document, but not a good guidance document. It remains the only official NERC “guidance document on BCSI” in the cloud today.
Even after the two revised standards came into effect, NERC entities mostly avoided SaaS that required BCSI access altogether – and that situation hasn’t changed today[i]. This is mainly due to confusion among NERC entities over how they should comply with the new and revised requirements, and especially what compliance evidence they need from their SaaS providers. In an August 2024 webinar sponsored by the informal NERC Cloud Technical Advisory Group and SANS, Peter Brown of Invenergy stated that point very well. I described what he said in a post two weeks later:
Peter attributes this to the lack of good compliance guidance. The only guidance on CIP-004-7 and CIP-011-3 at all is the already-existing document that was unexpectedly approved by NERC as “implementation guidance” at the end of 2023. I’ve heard there are already calls to create something better than that; I agree with them. In fact, there needs to be a whole education program on BCSI in the cloud; this needs to be combined with education on SaaS that utilizes BCSI. The lack of SaaS/BCSI guidance has meant that, other than one SaaS configuration management product that was already in use by many NERC entities six years ago, I know of no other use of SaaS with BCSI today.
The webinar, which had over 400 attendees, started with a good discussion of BCSI in general[ii]. However, when it got to the topic of BCSI in the cloud (the reason everyone was on the call, I’m sure), the cracks began to show. The main problem I had with it was that the presentations were clearly prepared in advance and approved by higher-ups at NERC; the presenters clearly weren’t allowed to deviate significantly from the language that was approved.
I don’t mind scripted presentations; in fact, before I do a webinar, I usually write out beforehand what I want to say, rather than try to ad lib my remarks (which a few times hasn’t gone well). However, I always encourage people to put questions in the chat, so I can answer them at the end of the webinar. I always try to leave as much time as possible for Q&A, since my general feeling is that attendees pay much more attention to the Q&A than to the presentation itself (in fact, I try not to do presentation-type webinars at all. I much prefer conversations, because I think people get much more out of that format, but even then, I try to type up the answers to the questions I anticipate beforehand).
Of course, to take questions, you need to leave time to answer them (without going over the time allocated, one hour in this case). The presenters finished with about 20 minutes left to go in the hour. Since there were about ten questions or comments in the chat, I thought for sure they would be answered in the remaining time.
However, I was quite surprised that, as soon as the last presenter had finished speaking, the moderator immediately shut down the webinar. The moderator said the presenters would reply by email to each question, but of course that doesn’t help the 399 people who didn’t ask that question and will never learn the answer.
It wasn’t a huge surprise that there was no Q&A period, since NERC (and Regional Entity) employees are forbidden to provide “compliance guidance” on a NERC standard. However, I copied the chat log and later reviewed the ten questions that were asked (a couple were formulated as comments but were clearly questions). One or two questions didn’t make sense, but most of the rest could have been easily answered without providing compliance guidance. Even the one or two questions that bordered on a request for compliance guidance could have been answered by redirection, e.g. “I can’t answer your question directly, but I can tell you that…” It’s disappointing there wasn’t any attempt to answer questions at all.
However, the presenters did bring up the document mentioned above, which NERC has endorsed as “Implementation Guidance”. Of course, NERC’s endorsement made everything in this document implementation guidance, meaning that auditors are bound to consider the document whenever the three BCSI requirements are included in the audit.
To provide an example of the problems with this document, one set of terms that is used frequently in it, and which the presenters discussed in the webinar, is “overlay/underlay”. Here’s the definition of underlay from page 6 of the document: “Infrastructure implemented by the CSP that runs all services offered by the CSP. This infrastructure could comprise the hardware, software, networking, and facilities that run cloud services. The security controls associated with this infrastructure are likely verified through certifications or other internal/external activities such as penetration testing.”
Here’s the definition of overlay: “The portion of the cloud service/product that sits on top of the underlay and is developed by the customer or has been developed for the customer’s use. This is how the Responsible Entity generally accesses their BCSI.” Are these definitions understandable by most people who might need to use them (generally speaking, IT and OT professionals)? I think they are.
However, if these terms are going to be used in a future CIP standard (which they may be), they need to clear a much higher bar: Auditors and Responsible Entity staff members will need to have the same understanding of what the terms mean. If not, there may be hell to pay. For example, I remember the huge controversies that arose over the meaning of terms like “programmable” (used in the definition of “Cyber Asset”), during the almost three year period when CIP version 5 was being implemented[iii] (as you may know, CIP version 5 was a complete rewrite of all previous CIP standards. It introduced the terms used in the CIP standards today: Cyber Asset, BES Cyber Asset, BES Cyber System, EACMS, PACS, ESP, PSP, External Routable Connectivity, etc). I’d hate to see that experience repeated with the BCSI requirements.
However, if NERC were to start using the terms overlay and underlay and approve use of these two definitions, audits where those terms come into play will be a huge mess. There will be questions like:
1. How do you draw the line between what’s underlay and what’s overlay? It isn’t a physical line, but it also isn’t a logical line (e.g., based on IP addresses).
2. What does “on top of” mean? It definitely doesn’t mean that the overlay is physically on top of the underlay. And since there’s no way to say that one IP address is on top of another, it’s not a logical relationship, either (like in a network diagram).
3. When you talk about someone having access to the underlay or the overlay, are you talking about access to all the hundreds of thousands (or perhaps millions) of physical and virtual devices that constitute the CSP’s underlay or overlay? And since a system deployed in the cloud can be spread across physical and virtual servers housed in multiple data centers and can move between them constantly, how can you even say where the system is located at any particular time?
4. Will the cloud service provider need to provide each NERC entity with all the certifications and pen test results that apply to either the overlay or underlay, or the “certifications or other internal/external activities such as penetration testing” that prove the security of the overlay and the underlay?
Of course, it’s not surprising that these two terms are so hard to define in auditable terms. But it is surprising that a document that makes heavy use of these vague terms has been endorsed as implementation guidance by NERC.
When I wrote a post announcing the upcoming NERC BCSI webinar, I expressed the hope that the webinar would provide enough clarity on possible ways to comply with the two revised standards that NERC entities with high and medium impact BES environments would finally feel comfortable using SaaS that utilizes BCSI. I also hoped that the webinar would discuss types of compliance documentation that SaaS providers need to provide to their CIP customers (without “blessing” one type of documentation). Neither of these happened.
What is most unfortunate about this is that compliance with the BCSI requirements is straightforward. It wouldn’t have been hard for the presenters to give the audience members at least some suggestions about how they could comply with those requirements. However, I’m sure the consulting/training industry will step up to fill this gap, so NERC entities can finally start using SaaS on the OT side of the house.[iv]
There’s an important lesson to be learned from this webinar, especially for the team developing what I call the “Cloud CIP” standards. Those standards (which will likely include the three BCSI requirements unchanged) are sure to introduce many new concepts. Whenever the standards are rolled out, they will need to be accompanied by a lot of training and education. Absent that, NERC entities are unlikely to be the first on their block to start using the cloud. If you (just) build it, they (probably) will (not) come.
If you would like to comment on what you have read here, I would love to hear from you. Please email me at [email protected] or comment on this blog’s Substack community chat.
I’m now in the training business! See this post for more information. Note that one of the web training sessions I’m offering is on BCSI.
[i] There is one cloud-based configuration management service that is used by many NERC entities for CIP-010 R1 compliance today, and has been for close to a decade.
[ii] I don’t believe NERC has released the webinar recording yet, so my discussion is based on my memory and the limited notes that I took. I will probably write a follow-up to this post when the recording becomes available (note to NERC: I usually receive the link to the recording for a meeting I’ve run in Zoom within an hour of the end of the meeting).
[iii] For example, in 2014 or 2015, NERC put out a document stating that a device that can only be “programmed” by setting DIP switches, was therefore programmable, and thus needed to be considered as a possible BES Cyber Asset. This had huge implications for large coal plants – which could have tens of thousands of devices that would potentially be designated as BCAs. Shortly after NERC published that document, there was a meeting where, from what I heard, it was a miracle that no fights broke out – emotions were running so high.
[iv] I will be providing a webinar introducing this topic on November 11. See this post for more information.