Intelligent Utility Network Community

The smart grid, smart meters and increasing quantities of intelligent devices have put an unprecedented amount of data into the laps of utilities. Our community is for professionals tasked with making sense of that data, integrating it with other data, and increasing customer and utility value by unique analysis and use of that data, while protecting customer privacy. 

17,682 Subscribers

Article Post

Metamorphosis of Grid into Distributed Model Will Drive Need for Better Grid Analytics Technologies

by Michael Bauer
President & Founder — Sentient Energy, Inc.

As the technology and economics of distributed PV solar continue to improve, strong growth in U.S. capacity is expected to continue. It is projected that by the year 2020, between four and five million homes will have rooftop PV solar with combined power capacity that exceeds the Three Gorges Damn in China, the world’s largest power plant.  The growth will not be limited to the areas of the country with the most sustained sunshine, like California. Instead, this phenomenon will be nationwide, as payback for rooftop PV is now estimated to be around 10 years or less in the majority of U.S. states.

As the U.S. grid undergoes this metamorphosis into a distributed model with a two-way power flow, the information required from the grid will expand from traditional safety and reliability data to include economic and technical data for siting feasibility and future innovation. This will not only require a proliferation of new sensors to record grid activity and events, it will demand a new generation of intelligent sensors equipped with high-resolution oscillography technology and waveform analytics in the sensor unit itself, as well as advanced analytics software on the head end for more detailed analysis.

The large scale deployment of intelligent sensors represents a significant paradigm shift in the way utilities exploit information from the distribution grid. Over the past 120 years, the utility industry has never placed major emphasis on distribution grid analytics. Relatively few sensors have been deployed here, and even today, many utilities do not collect and analyze a great deal of information from this part of the grid. As always, the number one objective of gathering information on power delivery will remain public safety. Next on the list is reliability, as ever more of our lives and commerce depend on uninterrupted electrical service.  

Defining the Top Power Delivery Objectives

The upper left bullets in Figure 1 represent the typical objectives of distribution grid management over the last several decades. However, as Distributed Energy Resources (DER) continue to gain momentum, the three bullets in the lower left of figure 1 (Safely enable two-way grid, DG siting data, and continued innovation) will become increasingly important.

Whether or not the grid is one directional or two, safety will still remain the number one priority. Furthermore, as solar continues to become more attractive, utilities will be challenged to provide data on where this additional generation should be located. This is particularly important as more C&I customers adopt DER with the potential to become major generators. Solar demand will put pressure on utilities to provide new innovation, not only in the technology to safely operate the infrastructure, but also innovation in the packaging and pricing of this distributed power generation.  The quickest and most efficient way to meet these objectives is through better grid analytics.

Figure 1: Grid analytics has potential to transform power delivery operations

Grid analytics are becoming a vital component of the Utility Industry’s public safety initiatives. They can help detect, pinpoint, and expedite the removal of hazards such as energized power lines on the ground.  Although utilities have managed to continually improve restoration speeds, they’ve made less progress on understanding and predicting the underlying causes of events that cause outages. Grid analytics have the capability to help utilities fix the causes of outages, not just the symptoms, which leads to significantly improved reliability.

Grid analytics not only monitor grid events, they can also monitor the condition of grid assets, such as wires, switches, and transformers. This contributes to better lifecycle management of the grid. Grid analytics can help to significantly reduce Operations & Maintenance (O&M) costs by replacing costly and inconvenient emergency maintenance with planned maintenance that can be performed more cost effectively, and results in considerably higher customer satisfaction due to less power disruptions. 

In order to safely operate a two-way grid, analytics are needed to determine the direction of the current, presence of harmonics, and other power quality issues. Without a grid analytics system, it is hard to track where power is actually flowing. This makes it very difficult to determine the best places to site additional distributed generation. If the flow of power isn’t tracked by feeder and by lateral, or perhaps even by span, then there is no data to provide solar project developers that want to build more generation in a given location, because utilities don't know what the real flows are, and whether or not additional generation actually creates additional value. If a feeder has no distributed generation whatsoever, and significant power is injected into that feeder, most of the time the utility will be able to pay the provider of that power a fair amount of money. On the other hand, if a feeder that is already producing more power than the feeder itself needs, then it becomes more of a headache to the utility, as it now needs to put in new protective equipment to manage the additional power that may have no additional value.

Lastly, without good data from analytics, it is hard to determine what innovation will have the most impact. For example, if a new type of battery is put on a feeder, good monitoring of power flows and good resolution on that feeder is required, or else the battery’s effects cannot be measured.

Prioritizing where Analytics Create the Most Value

Effective Grid analytics require three things: grid data which is provided by devices such as intelligent sensors; access to the data which is a function of data communications technology; and analytics technology, which comes in the form of software applications. While this may be obvious, the following analysis will show that these three key elements cannot be taken for granted.

Figure 2: Grid data framework and analytics hotspots

Figure 2 presents a framework chart for identifying the grid analytics “hotspots” where analytics bring the most value. Looking at the horizontal axis (X axis), on the left-hand side there is simple data and small data tables. On the right-hand side of the axis are both simple and complex data. This includes not only the simple data, but also complex data such as waveforms that contain complex data such as events waveforms, load profiles, times series, and other complex data sets. On the vertical Y axis, there are small amounts of data on the bottom and large amounts of data on the top. In the upper right hand quadrant of this 2 X 2 framework there are lots of waveforms and complex data as well as lots of the simple data. Clearly this is the place where utilities want to have the lion’s share of their analytics because it is where the richest data is available and where the biggest return on investment is to be had.             

Figure 3: Grid data acquisition and aggregation

Using this same 2 X 2 Grid (Figure 3), it can be determined where currently grid data is acquired and aggregated. The electric smart meter is in the lower left-hand corner because it just collects and communicates simple data points rather than complex data sets like waveforms, time series, etc. The meter also doesn’t collect a lot of data outside of its revenue focused function.

Meter data is aggregated by the meter data management system (MDMS) which appears in the upper left-hand corner. The MDMS aggregates all the data from the meters, so it gathers lots of simple data, but it doesn't have any complex data because the meters don't provide any.

Switches and other grid devices in the field can, if they are equipped with telemetry, provide some complex data because they may have additional functionality added to them to allow the capture of waveforms or other complex data, even though this is typically not their main function. Line sensors, on the other hand, are built to collect both simple and complex data, and are found towards the of the center of the chart.

The data that gets aggregated from devices such as switches and sensors goes into the Distribution Management System (DMS), and from there into a Historian database. These systems are in the upper left-hand corner because they typically don't understand complex data and can only handle simple data.

Figure 4: High value analytics targets are not addressed by available analytics offerings

As can be seen in Figure 4, available analytics, such as Meter Data Analytics or Distribution Automation Analytics, are dedicated to the analysis of large amounts of simple data because that’s the type of data that is readily available to them. However, the most interesting area in the chart is the hotspot in the upper right hand corner where large amounts of complex data reside; this area is completely neglected by today’s analytics systems. Yet here is where data is found on the location and cause of faults and non-fault events, high-impedance faults, DER integration, incipient feeder outages, and many other grid events – all high-value targets unaddressed by today’s systems.

Networking and data system architecture

Figure 5 below shows a schematic and simplified IT and networking architecture of the systems that forward looking utilities have available today for the purpose of controlling and managing the distribution grid. These same systems are also used for grid analytics.

Field devices connect via data networks to backend systems at the utility data center: Smart meters most often connect via a typically a mesh AMI network to an AMI headend, which makes the data available to the Meter Data Management System for processing before it is analyzed by Meter Analytics software. For Distribution Automation, switches and IEDs with telemetry and line sensors are connected via a DA network, often also mesh or cellular, to the SCADA and DMS systems. Data collected by these control systems is stored in the Historian database, which is in turn the main source of data for Distribution Analytics software.

The above systems and data centers can be connected to analytics systems hosted in the cloud, as shown in the top part of the diagram. However, many utilities view their grid and user data as proprietary and shy away from interconnecting their critical operations data to cloud services.

Figure 5: Schematic DA and AMI network and IT architecture

Architecture bottlenecks impeding Grid Analytics

Figure 6 illustrates the key bottlenecks inherent to the prevailing IT and networking architecture that limit the potential of grid analytics.

DMS and SCADA are control systems. They are optimized for control, with analytics as an afterthought. Presently, these systems are only equipped to handle simple data (e.g. “DNP points”). This creates a bottleneck that immediately limits what data gets to the distribution analytics (Figure 6 - on the top of the stack on the left-hand side). In addition, distribution analytics currently are in a very nascent development stage and not nearly as mature as meter analytics. This is a consequence of the fact that there is much more meter data available, and analytics vendors have concentrated on products that can utilize this data. Although many vendors have plans to develop analytics for the distribution grid that utilizes data available from SCADA and DMS, in practice, most follow available funding which is presently focused on the meter side.

The next, and probably the most important bottleneck, are the networks connecting the utility data center to the field. The DA network on the left hand side of Figure 6 is a control network focused on reliable and effective control of grid devices; it hasn’t been designed with the objective to maximize rich data throughput and access. The AMI network on the right-hand side is a revenue network focusing on collecting and communicating the revenue data related to power usage. Neither of these networks is focused on analytics.

There are also limitations from a communications technology perspective. If the utility is using mesh technology, that mesh technology is designed to make sure that every last meter can be reached securely and assuredly. These networks are therefore built for reliable connection and reach, but not for the transfer of large amounts of data.

If cellular communications technology is used, bandwidth is available and there is no issue with data volume, but reach is often limited because cellular coverage is centered on locations where many people are present, whereas utility power lines and assets often connect these locations but find themselves in areas where there is no cellular service. Cellular can also be unattractive to utilities that prefer capital costs over O&M costs, or the variability of cellular charges that can be caused by temporary data overages.

Figure 6: Architecture bottlenecks that impede analytics

As a result, it is obvious that the existing IT and networking architectures require enhancements to maximally exploit the benefits of grid analytics.

Requirements for a Grid Analytics System

Utilities still have serious bottlenecks to resolve as DMS aren’t ready for complex data, distribution analytics are immature, and the communications networks are really built for other purposes.

So what are the requirements for a grid analytics system?

First and foremost, it needs to provide analytic capabilities that match the data types and structures that are generated from the distribution grid.  The analytics capabilities need to be applicable not just for simple data, but also for complex data.

Given the network limitations, these analytics capabilities need to be deployed locally out in the field, as well as centrally at the distribution control center. Secondly, the analytics system needs to enable data access over the network that is being utilized, as building a separate network for analytics purposes is generally too expensive.

Lastly, the grid system needs to interface with all devices collecting grid data, and have the ability to import all the data sets being collected in their entirety, rather than being limited to the subset of data used by SCADA and DMS for control purposes.

Building an Effective Grid Analytics System

The starting point for building a grid analytics system is the deployment of sensors in the field. As previously stated, Utilities need analytics capabilities that are local as well as global. This means that the sensors they deploy must have intelligence onboard to facilitate local analytics capabilities and to process the collected data into manageable data sets that can be communicated over the constrained communications networks that are in place.

Figure 7: Building blocks of an intelligent sensor and their function

An intelligent sensor is an oscilloscope – the trusted tool used by an electrical engineers - combined with a Linux computer that handles all local processing and analytics. The communications module establishes connectivity to the network and manages it to meet the utility’s operations preferences in terms of cost, bandwidth utilization, and other parameters. Most importantly, this module compressed the rich data form the oscilloscope in the sensor for transfer of the existing network. Bandwidth need is further reduced as the onboard computer performs local analytics to interpret measurements locally and communicate results rather than raw data.

An easy calculation shows that both local analytics and data compression are critical capabilities for the intelligent senor. Un-processed and un-compressed, one triplet of sensors (three sensors) collects as much information as one million AMI meters transfer over the network. A company like Florida Power & Light (FPL), for example, has more than 4.5 million meters, and are presently adding about 24,000 intelligent sensors or about 8,000 triplets for grid management and analytics. If all data collected by these sensors was transmitted back to the utility control center, the overall data volume would be the same as if FPL had added 8 billion additional smart meters to their existing 4.5 million. .

Obviously, present grid communications systems can’t possibly transmit that amount of data. So it is necessary to use local processing in the sensor to time-stamp, aggregate, and organizes the data, detect and analyze events, and filter out the noise.

Figure 8: Required building blocks of an effective grid analytics system

Communications pre-processing in the intelligent sensor must be paired with its counterpart in the utility data center for the end-to-end data transfer to work reliably and effectively across available AMI and DA networks. These communications processing capabilities are the second building block of a grid analytics system.

The next component needed is real time and historic analytics that, unlike currently available DMS and SCADA systems, can take advantage of the rich and complex data coming from the sensors and other field devices. These analytics systems are not limited to the subset of data that SCADA and DMS focus on, and can therefore provide much deeper insights and real-time information than any systems that are located “north” of the Historian.

Finally, a grid analytics system needs the ability to share its data and analyses with other systems. These interfaces can open up even richer analytics of the grid overall, as higher level systems can now combine rich distribution grid data with other information without any of the discussed data path bottlenecks and restrictions.

These four components: Intelligent sensors, communications processing, real-time/historic analytics, and data integration with other systems, together form an effective grid analytics system that works within the constraints of the current Utility architecture.

Grid Analytics System Maturity Today

Intelligent grid analytics is no longer a theoretical concept. Intelligent sensors with local analytics are currently available, and have been deployed by the thousands at leading utilities, such as FPL. They have been proven to perform as designed with high reliability. Their event detection capabilities continue to mature, and the analytics capabilities are developing at a steady pace.

For more information, Michael Bauer can be reached at, or visit

Explore Related Topics:


Very informative post. Thanks.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Marc, Really informative.  Thanks for posting.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.