EnergyIoT Article 6 – Energy Services (DevOps) Domain - The Heart of the Ecosystem
Image by Stuart McCafferty - No copyright
- May 13, 2019
- 836 views
EnergyIoT Article 6 – Energy Services (DevOps) Domain - The Heart of the Ecosystem
By Stuart McCafferty, Eamonn McCormick, and David Forfia
The viewpoints in this article and others in the series are the personal views of the authors and in no way are meant to imply or represent those of the companies they work for.
Figure 1- EnergyIoT Conceptual Model
(developed for this series with the Gridwise Architecture Council)
This is the sixth in a series of articles introducing an event-driven, data-centric EnergyIoT Conceptual Model. This article describes the virtual Energy Services (DevOps) Cloud Domain highlighted in the image above by the thick red outlined cloud in the upper right. The Energy Systems Cloud Domain we envision:
- “Abstracts” the complexity and brittleness of communications between systems and OT assets using digital twin agents
- Provides modern “microservices” to support virtualization, containerization, and orchestration
- Leverages messaging bus technologies such as pub/sub
- Uses rich semantic information model standards to drive interoperability
- Enables intelligent grid-edge devices to self-report “by exception” or on a timed basis and support an event-driven architecture
- Simplifies data usage with secure data-retrieval services and publish/subscribe (pub/sub) message bus standards that enables easier, less expensive system integration, and faster command and control response capabilities
- Enables a variety of modern data storage systems to support structured, unstructured, smart contract/digital ledger formats
- Exposes energy-specific services for DevOps teams including source code management, software development and IoT orchestration
- Provides the most advanced, modern cyber security technologies and tools across the entire ecosystem with the added capability of adapting to new cyber security challenges universally as new/evolved threat management tools become available
- Reduces time and costs for systems integration by providing services and standardized data definitions
- Accommodates opportunities to all stakeholders for innovation, new services and capabilities
- Includes analytic tools and capabilities to continuously improve and optimize operational and market activities
Figure 2 - Energy Services Cloud
This article describes the architectural components within the above Energy Services Cloud pattern, which also includes a “DevOps” environment designed to support the energy software development community. The image below was briefly introduced in Article 3 - EnergyIoT Domain Building Blocks. The middle layer surrounded by the red-oval is another view of the subject of this article.
Figure 3 - EnergyIoT architecture "stack" view
In current electric power business applications, vendors have their non-scalable proprietary pipes that connect business apps. The green cloud layer in the conceptual architecture is the energy services “bridge” that abstracts away the hardware from the software. The abstractions are represented by purple boxes in the image above.
The upper communications layer are basically “business semantic” layer SOA services. The lower layer are “hardware oriented” communications services. The digital twins will connect to the “drivers” in the OT communications layer to talk to particular types of devices similar to an operating system has drivers for printers and other peripherals.
Conceptually separating the Energy Services Cloud from the Energy Systems Cloud Domain directly addresses the limitations of siloed systems. This is the big “aha” within the EnergyIoT architecture. The “green cloud” abstraction layer exists to enable seamless communication between physical systems, Energy business systems, and data. Conceptually, this approach allows assets to directly self-report status data when an event occurs rather than use polling systems - or really, any system. Remember, these devices in the field are intelligent and their capabilities are expanding every day. The architecture should leverage that intelligence rather than forcing status reporting through some remote system. This approach enables “grid intelligence”, taking advantage of smart “edge” devices connected to the grid. There are obvious positive implications to this type of approach:
- Data is reported only when threshold value changes occur that trigger an event report, ensuring that only the most valuable information is:
- Reduced communication traffic.
- Reduced storage costs.
- Reduced data to analyze.
- Distributed publishing ensures that data is made available directly to authorized “subscribers” and common data stores.
As a simple example, consider electric power meters. In today’s architecture, centralized systems poll each smart meter, requesting status information every hour or at another interval. What if, instead, that intelligent meter knew what interval to report at automatically? What if a homeowner could bid 2 KWh of demand response from 5PM to 6PM into a newly created distribution market? The meter could recognize or be notified of the event, then automatically send a meter read at the moment the bid obligation began and another when it concluded. There would be historical data to see what the readings were before the event and the meter reads at the beginning and end to determine whether the contractual obligation was met to make settlement fairly easy. And, what if that data resides in a data store that other authorized systems could easily gain access to and use for whatever analysis makes sense? Imagine the economic and operational efficiency to be achieved. These are the fundamentals of an IoT, event-driven, data-centric architecture.
The separation of the Energy Services DevOps Cloud unlocks the ability to create a common set of secure services to access data and use in other systems and analysis tools. It also enables a paradigm of “distributed intelligence” that leverages smart devices on the grid, in businesses, and in homes. This in turn empowers a democratized ecosystem that embraces innovation and creates an “Uber-like” environment that provides economic opportunities for new and numerous stakeholders with the net effect of more efficient, clean, resilient, and personalized delivery of power.
Fundamentally, the EnergyIoT architecture is “data-centric”. The importance of data cannot be underemphasized. In fact, “it’s all about the data”! Being event-driven is also extremely important, but the events generate data, and that is the most fundamental element of the entire IoT ecosystem. The EnergyIoT Services Cloud includes a rich set of secure data communication (message bus) and storage/retrieval services.
Designing the correct data structures and services arguably is the first and most important step in designing the Services Cloud. It is also going to be difficult and likely take time to design it intelligently. Luckily, in the electric power industry, we have rich semantic information models developed by the International Electrotechnical Committee (IEC) such as 61968/70 Common Information Model (CIM), 61850 (originally a substation automation standard that modeled traditional assets and that now includes DER), IEEE’s 2030.5 (Home Area Network - which is harmonized with IEC 61850), OpenFMB (also harmonized with 61850), and other rich semantic information models that are mostly harmonized with one or the other IEC standards mentioned. This will be hard work and will require experts from these standards organizations as well as technology experts with experience in designing DevOps systems, data repositories, and messaging payloads.
Smart Contracts, Digital Ledger Technology (DLT)
The benefits of distributed digital ledgers is currently untapped in the energy industry. DLT often gets mischaracterized as “BlockChain” or cryptocurrency technology. Although BlockChain and cryptocurrency are forms of DLT, there are other ways of implementing DLT besides BlockChain and applications besides cryptocurrency where DLT is a legitimate and compelling potential technology for energy applications.
The green cloud energy services layer must be founded on data that can be trusted. DLT technologies were built to be trusted. They include the following foundational principles that translate well into an EnergyIoT data-centric architecture:
- Transparent smart contract/business rules
- Verifiable processing
- Trusted orchestrated processing of events
- Distributed control of processes
- Standards based processes based upon open scientific and agreed standards
- Encrypted and quantum safe
- Consensus based
Structured data is highly organized information that is stored in fixed fields within relational databases. It uses a Structured Query Language (SQL) or other standardized method to perform searches, add, modify, or remove fields. A common IoT best practice for today’s relational databases is to create methods or “microservices” that encapsulate standard SQL searches and data-manipulation commands. This abstraction allows for a highly controlled additional level of access to relational data and simplifies data processing for authorized systems. It also prevents an old “worst” practice of sharing database passwords with developers and jeopardizing the integrity of the table structures and the information stored.
Not all information is easily decomposed into relational fields and tables. This type of information may be human-generated or machine-generated, and it may be text or non-text data. This includes video, documents, contractual information, pictures, logs, and any other type of data that doesn’t fit neatly into a highly structured format.
Cloud providers have mastered how unstructured data is managed and stored. Solutions like Hadoop, Apache Hive, MongoDB, and Cassandra have excellent performance and offer existing tools and services to support highly scalable capabilities.
Digital Twin Agent
The architecture includes an abstraction layer that includes Digital Twin Agents which simplify the communication between Energy Systems and grid assets (adapters) or can be used in simulation environments to emulate the behavior of assets. This special form of a microservice is a critical element when scaling grid networks and Energy Systems to support thousands or even millions of DER. Consider Digital Twin Agent adapters as containers that spin up when you talk to a physical asset and go away when you are done – meters and switches are examples of Digital Twin Agent microservices that could live in the cloud and come and go only when you are talking to the asset. Some Digital Twin Agents may remain resident in memory all the time if they are real-time mission critical pieces of the overall system and are likely to be physically located within or near the asset they virtualize. One major advantage to this approach is the ability to manage Digital Twin Agent container upgrades that provide additional functionality or bug fixes. The upgrade changes can be performed one time and propagate immediately out to all assets of that model and make.
The architecture envisions Agents that announce the asset, describe the asset’s capabilities, provision the asset, and commission the asset. The Digital Twin Agent ensures there is redundancy/fail-over capabilities built in and there is distributed intelligence that guarantees when communications are lost, the Agents continue to operate independently in the last command set they were provided from a parent or authority Agent. This is currently a technology gap since there is no energy standard for digital twins.
There are Probably at Least Two Types of Digital Twins
The figure below is the authors’ current thinking on Digital Twin Agents, but it may not be exactly right. More work is required to establish agreement and common interfaces, features, and containerization/orchestration methodologies. Docker and Kubernetes have many of the capabilities that support the functionality envisioned for Digital Twin Agents in the architecture.
Figure 4 - Author's Digital Twin Agent concept
One important point of distinction is that many people jump to the conclusion that a Digital Twin is a virtual “mirror” or an emulator of the physical hardware or entity being represented that accurately mimics the operational behavior of the asset. After much thought and discussion, some of the authors believe that is not necessarily true - and it is why the authors use the term “agent” to make the distinction that this concept may not align with some implementations of Digital Twins.
In some IoT implementations, Digital Twin emulators are used to support simulation - this type of Digital Twin may be used in power flow modeling or planning types of exercises. In this case, the Digital Twin is quite sophisticated and requires an operating environment that has significant horsepower, such as a server. It is a “mirror” of the physical asset that includes an emulation engine, mirroring the full behavior of the asset.
The second type of Digital Twin is a communication abstraction, providing services and systems a “bridge” to an asset that standardizes communication to assets of the same class through “adapters” that translate semantic message payloads to whatever protocol the hardware speaks (e.g. ModBus, OPC, DNP, etc.). In fact, some IoT developers would simply call these adapters, rather than a form of Digital Twin. No matter what you call it, the adapter form of a Digital Twin Agent within the EnergyIoT architecture not only performs the adaptive communication, but it also includes event handlers, an archivist, properties, and methods - so it is more than just an adapter.
Digital Twin Agent adapters are compact, have relatively low processing power requirements, and can be physically located in the cloud or on the OT physical system near the asset or as part of the asset. For the purposes of the EnergyIoT architecture, adapter functionality is a minimum requirement of a Digital Twin Agent, while having the additional emulator capabilities would be helpful when performing simulation and optimization processes.
Security and Identity Management
In a data-centric ecosystem, there is the ability to inspect every bit of information as it travels through the system. The data can be filtered to check that it is within appropriate operational limits, searching for potential “spoofing” of an authenticated data publisher. Intelligent analytics can be trained to search for “bad actors”, tampering, theft, and intrusions. Any anomalous system behavior can be detected, flagged, quarantined, and/or have a human dispatched to inspect in person. The EnergyIoT ecosystem described in these articles “has security designed-in” that allows us to apply the most sophisticated analytics the industry has to offer, which will continuously improve, “learn”, and adapt from larger and larger data sets.
Figure 5 - Example of how built-in security services could support the EnergyIoT architecture
The image above provides an example of how security is “built-in” in an event-driven, data-centric architecture. The orange-colored objects provide opportunities for security microservices to:
- Ensure that data entering the system is from an authorized and authenticated source
- Ensure data is within normal operating limits
- Ensure that business systems and other actors are authorized and authenticated to view the data
- Provide mitigation services when any of the above 3 conditions have not been met, which could escalate in scale from flagging erroneous data and its source, quarantining data publishing sources, or dispatching a human to perform a visual inspection
Software Development and Source Code Management
The Energy Services Cloud supports the development community through its DevOps environment. Source code management is a key component of any development environment. All of the major cloud vendors provide built-in services for software developers. The development environments support multiple languages, operating systems, and target deployments (cloud, on premise servers, devices). The software development services include:
- Collaboration and reuse
- Version management
For the purposes of this orchestration discussion, the authors are not describing the orchestration of containers (e.g. Kubernetes), but rather a simple “What You See is What You Get” (WYSIWYG) development toolset. This may or may not be part of the Software Development toolkit described in the previous section. Instead, it is a visual open source development environment called “Node-Red” or other yet to be developed toolset The “Node-Red” IoT software development tool currently has a large existing user community and hundreds of developed software objects to support rapid development by “wiring” different objects together and “orchestrating” the processes similar to National Instruments’ Labview. This tool was designed with IoT in mind.
Figure 6 - Node-Red IoT Development Tools (source: https://node-red.org)
Node-Red is an event-driven architecture built on top of node.js and can run natively on many low-cost single board hardware platforms, including Raspberry Pi, BeagleBone, and Android. As an orchestrator, it provides publication/subscribe message buses like MQTT simply and seamlessly. The executable runs on all operating systems within a browser environment. At the time of this writing, there were 3,228 open source contributions, and is growing all the time.
The Energy Services Cloud would ideally have its own library of software objects and orchestration products that could be contributed by vendors to support connecting to their Digital Twin Agents and by software development companies to create new capabilities and expose innovative services to other developers. Some independent validation and rules will be necessary, but the key point is that tools like the Node-Red environment allow for rapid development and reuse of node objects that will evolve and grow in number over time. Applications developed in these tools can be deployed using containers and associated orchestration services, which will be discussed in the following section.
Cloud Microservices and Container Technologies
Microservices are an Application Programming Interface (API) for Service-Oriented Architectures (SOA) that decompose software methods into the smallest functionality as practical with the intended purpose of maximizing its reusability by other services or applications. Microservices and container technologies are modern software techniques that have compelling implications for the energy industry, offering opportunities to “virtualize” the Operational Technology (OT) physical assets and create a much more interoperable electric power grid and its systems. It is the “fabric” of the Energy Services Cloud abstraction capabilities.
The Open Group defines a service as supporting four foundational principles:
- It logically represents a business activity with a specified outcome.
- It is self-contained.
- It is a black box for its consumers.
- It may consist of other underlying services.
There are good and bad things associated with microservices. The good is that the use of microservices can dramatically reduce the amount of development and testing requirements, resulting in more rapid software development. The bad is that the use of an individual vendor’s microservices can “lock in” software development teams who become dependent on proprietary microservices that will not easily port to another vendor’s solution.
Containers are the minimum set of Operating System (OS) components and software dependencies packaged together to allow an application to run as a Virtual Machine (VM). In other words, virtualization is performed through the use of containers. Docker is the preeminent tool for creating containers that can be deployed to a variety of hardware devices including servers, personal computers, industrial computers, and single-board computers like Arduino and Raspberry Pi.
The real “magic” of containers is in the orchestration services. The most popular orchestration system, Kubernetes (pronounced Koo-ber-net-eez) provides automated deployment, scaling, redundancy, and management of containers. Kubernetes monitors and manages the container deployments and can quickly spin-up new or redundant containers if something goes wrong, managing the overall health and operations of the container.
The authors’ Digital Twin Agent concept presented previously is a form of a microservice that may use container technology for virtualization. It would be deployed using an orchestration product like Kubernetes.
Artificial Intelligence and Optimization
Perhaps the most exciting opportunities for the electric power industry and the transition to an EnergyIoT architecture is the ability to apply today’s (and tomorrow’s) most advanced analytic systems. One analytic opportunity was showcased earlier in the Security and Identity Management section to identify bad actors, spoofing, theft, tampering, and intrusions. The same types of Artificial Intelligence (AI) and deep learning techniques can be applied to all types of operational, market, planning, forecasting, settlement, and billing efforts. Analytics are especially effective in an event-driven, data-centric ecosystem where the data is meaningful and available to authorized systems through reusable microservices. System vendors will rapidly adopt these analytic tools to enhance their capabilities, predict and solve problems before they occur, and to optimize their customer’s business functions that their systems support. The number and breadth of opportunities are boundless, enabling a Renaissance era for the electric power industry as these tools to learn and the operation of businesses, systems and assets become “faster, better, and cheaper”.
Service Oriented Architectures (SOA), Message Buses, and Message Payloads
SOA, message buses, and message payloads have been around for decades. The basic premise of SOA is the idea of “loosely coupled” services that provide some business value and can be reused by other services and applications. It is a rather simple concept that has its roots dating back to the early 90’s with the advent of Visual Basic eXecutables (VBX) and ActiveX reusable components. These early software components allowed other software developers to embed highly complex capabilities into their own applications using the components’ Events, Properties, and Methods to abstract the inner logic as a “black box”. Conceptually, this is the same loosely coupled foundation that SOAs provide.
Message buses, sometimes referred to as message-oriented middleware, orchestrate communications between different services and applications. Some message buses include a message queue and “broker” that perform routing logic in a structured way. Some brokers may even provide “quality of service” and guaranteed message delivery services for some routing situations.
Some forms of message buses route messages in “broadcast” or “multicast” modes to allow any approved actor on the bus to see the message. This is not the optimal paradigm for security and also creates additional processing needs of each actor on the bus to determine whether the message was intended for them or not. Instead, the publication/subscribe (pub/sub) message bus technique adds an extra layer of protection through a concept called “topics”. Authorized Publishers can publish topics to authorized Subscribers, providing highly-granular and structured management of data, the actors that can send it, and the actors that can receive it. Data is routed in a peer-to-peer fashion and can be encrypted for an additional level of security.
Message payloads can be schedules, variables, or any type of structured or unstructured data. Using today’s compression and binary capabilities, payloads can be extremely small, allowing low latency communication. Agreement on common messaging standards is critical to ensure interoperability. Use case and information modeling methodologies can be used to define common payloads for different grid operation, market, and business functions. Much of this work has already been accomplished through groups like the IEEE and IEC (to name a few of the prominent information model authors) and can be quickly leveraged to support standardized messaging payloads.
SOA, message buses, and standardized message payloads are a foundational piece of the EnergyIoT architecture that promotes rapid integration and interoperability of new grid assets and systems.
The Energy Services (DevOps) Cloud domain (Green Cloud) is the heart of the overall EnergyIoT ecosystem. This domain is the abstraction layer that dramatically simplifies integration of new assets and systems. The microservices provide common “black boxes” to access data, apply analytics and deep learning techniques, interoperate with other services and applications, connect to and aggregate grid assets through Digital Twin Agents, and perform virtualization, containerization, and orchestration. The most advanced identity management and security tools are available as part of the native environment. The domain is scalable, flexible, adaptable, and extensible. With the inclusion of a DevOps environment the time and costs for systems integration are substantially reduced using common services and standardized data definitions. The architecture is data-centric. The architecture is event-driven, so grid assets can report state information only when something changes or on a timed basis, reducing the amount of communication traffic, non-actionable message payload processing, as well as the amount of data to store and analyze.
The company that develops the “green cloud” portion of the EnergyIoT architecture will not only greatly benefit financially, but will also achieve the “social innovation” kudos from the rest of the world for providing the mechanism to modernize the electric industry’s technology, provide a method to directly address the reduction of greenhouse gases (GHG), and enable the rapid integration of more economical generation sources such as grid scale PV. This is a big deal – a moonshot opportunity for mankind. But, it will take great vision, dedication, and commitment from one or more technology companies with both the financial and human resources as well as a strong understanding of the specific needs of the electric power industry. If you are one of those companies, feel free to contact any of the authors at Energy Central or on LinkedIn.
This is the sixth in a series of EnergyIoT articles proposing a fundamentally different architecture to solve the problems of today, propelling the electric power industry into the 21st century and beyond. The seventh and final article, “EnergyIoT Article 7 – The Roadmap and Next Steps”, will be published next week on Energy Central and LinkedIn.
The rest of the article series can be found here:
About the Authors
Stuart McCafferty, IoT Architect, Black & Veatch
Stuart McCafferty is an accomplished Smart Grid technical executive with an innovative history, strong relationships in the utility and vendor communities, business and partner development, platform and solution design, go to market planning and execution, and practical application of existing and emerging/disruptive technologies. Prior to B&V, he was VP of EnergyIoT for Hitachi America, where he led the architectural design of a distribution system platform supporting microgrid and Distributed Energy Resource (DER) related businesses. At B&V, Stuart supports the utility, technology, and vendor communities in strategy and pragmatic application of DER that combines IoT best practices and technologies with energy standards and protocols.
Thought leader in the Internet of Things (IoT), Big Data, Cloud Computing, Artificial Intelligence (AI), Machine Learning, and connected home with practical application within the Smart Grid ecosystem. Expert in utility IT/OT and the application of DER and microgrids for resilience, economics, and reliability.
Stuart is a US military veteran, Air Force Academy graduate, an Energy Fellow for community resilience at the National Institute of Standards and Technology (NIST), an Energy “Expert” for Energy Central, and Vice Chair of the Open Field Message Bus (OpenFMB) user group.
David Forfia, Gridwise Architecture Council Chair
David is the Chair of the GridWise Architecture Council since 2015 and has been a council member since 2013.
The GridWise Architecture Council (GWAC) is a team of industry leaders who are shaping the guiding principles of a highly intelligent and interactive electric system. The Council is neither a design team, nor a standards making body. Its role is to help identify areas for standardization that allow significant levels of interoperation between system components. More about the Council can be found at www.gridwiseac.org
Eamonn McCormick, Chief Technology Officer, Utilicast
Eamonn McCormick is the CTO at Utilicast, a leading energy industry consultancy. Eamonn is a passionate believer in the bright future of the energy industry and the importance of collaboration as the foundation for solving for our current industry challenges. He is a results driven technology leader with a track record of success. He has implemented strategic technology change at several large energy companies over the last twenty years in the areas of wholesale markets, transmission and energy distribution primarily. In addition Eamonn is currently chief architect of the Energy Block Chain consortium.