Is this big data? Or is it complex data?

Posted on February 13, 2014
Posted By: David Pope
Topic: Metering

Yes, we have all heard about the 3 V's of big data by now. Or the 4 V's or possibly 5 V's, depending on who you ask. However, I would argue the value of data isn't necessarily tied to the "volume" part of big data (unless of course you are a storage vendor, and then the value is all about volume and storage).

Everyone else should be focused on how best to monetize complex data. Obviously, there can be a relationship between complex data and big data but let's define complex data so we can talk about solving business problems. Complex data is found in all industries including oil and gas as well as electric utilities. Whether you are talking about drilling for gas or monitoring the electric grid or "smart grid" the data coming off the sensors for one rig or one section of a grid may not meet the "volume" requirements of big data. However, it does meet the criteria of variety: multivariant, multivariate, multidimensional, and stochastic (MMMS) - it really doesn't get more complex than this type of data. It also meets the velocity criteria since it needs to be analyzed in near real-time in order to provide the operators with feedback to help drill an optimal well or to detect problems in the grid to better avoid potential power outages.

The data, and the use of the data, is complex because you need to continuously provide input into a variety of different predictive models (all of which interact or impact each other) in order to provide the best possible well to be drilled from both a quality and speed perspective or to ensure power remains on, safe, and as reliable as possible.

This type of analysis was really not possible until recently, when new technologies in both hardware, software, and the ability to add more sensors on all parts of a drilling rig or electric grid and to collect this data efficiently. We now have the capability of building the necessary system to allow for near real-time feedback to drilling operators.

As a matter of fact, I am currently working with others to develop a system combining event stream processing and a variety of advanced analytic capabilities to do just what I described.

Authored By:
David Pope has over 25 years of business experience in advanced analytics working in R&D, IT, Marketing and Sales. He developed expertise in big data analytics and enterprise architecture across several industries including Finance, Communication, Healthcare, Government, and Education prior to focusing on the Energy industry. He graduated Magne Cum Laude from North Carolina State University with a BS in Industrial Engineering and a Certificate of Computer Programming. He has presented

Other Posts by: David Pope

On meter data and analytics - August 26, 2016

Related Posts

The power sector goes digital By Sebastien Dolige


February, 14 2014

Azana Baksh says

David, nice article. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at

Add your comments:

Please log in to leave a comment!
back to top

Receive Energy Central eNews & Updates

Sponsored Content