Market Research: How is it Shifting with the Market while Retaining Data Integrity and Quality?Posted for AESP
ID 119704564 © Forkosmos | Dreamstime.com
- January 30, 2019
- 510 views
By Jeannette LeZaks
As consumers, we are constantly bombarded with requests for our opinions; whether it’s at the checkout line at the supermarket, when we are at home when a telemarketer calls, or through emails asking us for “just two minutes” of our precious time. It seems that these days everyone wants to know what we all think. And when it comes to the energy efficiency sector, it is not very different. We want to know what customers think and how people use energy, what decisions they make when they are about to purchase an appliance, and how they will react to an incentive program. This primary data is not only critical to help us confirm that energy efficiency programs are doing their job, but they also help us explore how we might augment programs and policies.
We set out to understand how methods of effective primary data collection are shifting in this highly connected and mobile world. How has the research landscape changed over the past five years? To do this, we reached out to experts in the energy efficiency field who deal with primary data all the time: evaluation consultants, utility company program managers, and survey implementation houses. We asked them about their experiences with primary data collection: how it’s changing, what challenges they are facing, and how they expect things will change in the future.
From our respondents’ perspective, the majority of surveys are still being implemented through telephone but this is rapidly changing toward web-based surveys. Snail-mail surveys – while worth considering for some types of primary data collection e.g. hard-to-reach customers – are not often used. In-person focus groups comprise a small percentage of our respondents’ data collection, but the data can’t be extrapolated to a larger population so it has its limitations.
One of the biggest challenges that we heard was the oversaturation of surveys, which can lead to difficulties in meeting survey targets and survey fatigue. As mentioned above, many sectors of the economy are requesting consumers’ opinions. Energy-related surveys can get lost in the myriad of other requests for information. Additionally, a household that participates in an energy efficiency programs may get called a number of times about their experiences in those programs. The increasing number of surveys can lead to more respondents ignoring phone calls from unknown numbers, declining survey participation when they do answer, or cutting the survey short if it gets too long or detailed.
A related challenge is the difficulty of creating a succinct yet effective survey that can capture the information needed but does not become overly complex or convoluted. Experts in survey implementation houses felt challenged by clients who want to add more questions to an already long survey. Long surveys can lead to survey fatigue; respondents may just
give answers to get the interviewer off the phone without providing accurate responses. They may ultimately hang up before the survey is complete, which ends up wasting both the responder’s time as well as the implementer’s.
Sometimes the biggest challenges lie in reaching the right person. For utility-targeted surveys, getting accurate contact information can be difficult. The contact information may be out of date which leads to disconnected and wrong numbers. Having too many wrong numbers out of a sample can lead to a bias in results. For nonresidential customers, the contact information provided by the utility may not reach the person that may be best able to answer the questions required. Finding the right person to talk to adds time and money.
The list of challenges is long. And while we know we will still need primary data, it is good to recognize these challenges exist and not only work to overcome them but to try to understand the impact on data quality.
Here are a few tips on ways to overcome challenges. Some of these tips appear basic, but are still worth noting based on their importance.
• Make the extra effort to ensure questions are clear. Take the time to review questions and test them to make sure you are getting the right data. The upfront effort will make the analysis and conclusions that much stronger.
• Limit the number of times that a customer is included in the sample. This may allay any customer dissatisfaction and reduce survey fatigue.
• Use a list-assisted sample rather than just random digit dial (RDD). Use lists with cell-phones in addition to land-lines.
• Supplement an existing list with purchased data to supplement a client’s population. Of course, this would increase overall budget which may not be feasible.
• Use a mixed mode approach, either with phone and web or potentially web-only. This allows for respondents to work on their own time.
• Offering incentives for completion of surveys may provide an additional motivating factor that increases likelihood of survey completion. Our collective studies have shown incentives increase responses considerably.
The bottom line is that how we collect primary data is changing, from gathering accurate contact info to finding ways to entice potential respondents to participate. We, as researchers, need to be cognizant of how these changes may alter the strength of our conclusions. We can’t assume that just meeting our sample target automatically means we successfully achieved our research goals – or that the data is valid. Now, more than ever, we need to make surveys that are capturing accurate information without disengaging the respondent and that the responding group is representative of the population of interest. Using a mixed-mode approach might be the way of the future, but we should also be rigorous in ensuring that these approaches meet statistical standards and reduce potential biases so that we can feel confident in our results.
The author would like to acknowledge the contributions of Lisa Obear of EMI Consulting and Laura Schauer of Illume Advising in the development of this article.
Jeannette LeZaks is a researcher at the Energy Center of Wisconsin where she develops and manages residential, commercial and industrial market research projects.