Survey Analysis of Open-Ended Responses? Its easier than you think!

If a company is seeking feedback about itself, surveys are the most used silver bullet tool. One approach for doing this is to go for “structured surveys” — where respondents are given predefined options to choose from. This is a sub-optimal solution and as the options restrict the scope of the survey and also end up influencing the response from the respondents. However, the pitfall with verbatim feedback is that the survey analysis part becomes a mammoth task in itself.

With the advent of AI-based text analysis, this roadblock is steadily melting away. In this article, we demonstrate how survey analysis for open-ended responses can be carried out at scale using our in-house tools. The reader can apply the insights to any another context where open-ended text needs to be dealt with (such as interview transcripts, movie subtitles etc). Survey analysis is just one popular use case that we are discussing in this post to exemplify the power of AI in real-world scenarios.

Survey Analysis for McDonald’s: Conceptual Case Study

Analyzing reasons for sales decline at McDonald’s

McDonald’s is arguably the greatest fast food joint in the entire world. They have grown to global levels and generate billions of dollars in revenue. It is natural to assume that a lot of people give their opinion or feedback regarding McDonald’s services and quality. Carrying out a survey analysis of this magnitude is a task that is neither cost nor time intensive especially when we are looking at open-ended responses.

In this case study, we present a conceptual overview of how our AI-powered survey analysis tool works. Suppose McDonald’s observes that its overall sales have declined over the past few months. They hire a market research agency to figure out what is the underlying reason for the sales decline.

The agency starts its investigation and notes that the home delivery sales have remained on the expected growth trajectory while the decline in sales has primarily occurred in the restaurants. Given that the problem is only with sales in the restaurants, the agency rules out Price, Offerings and brand image as possible causes and instead decides to do further analysis on Ambience and Service at the restaurants.

The agency decides to undertake a survey of 10,000 customers across the USA who had reduced their frequency of visits to McDonald’s over the last 6 months. The agency decided to do an open-ended text survey as shown below:

A sample survey response

Analyzing the open ended-text survey responses

Karna-AI’s deep learning powered text analysis algorithms allow you to identify key information from the responses.

A sample response and key information extraction from one of the survey questions: What did not meet your expectations?

We can do this for thousands of responses and then use sentiment analysis to dig out key aspects where McDonald’s failed to meet customers’ expectations. In the example below, we find that among many themes related to McDonald’s Ambience and Service, people were most unhappy about the “smell” at their restaurants.

Breaking down the distribution of sentiment for multiple aspects

Now we have identified the ‘what’ (smell) is possibly causing poor customer satisfaction, now we dig deeper to find out the ‘why’.

Digging Deeper — Examining Key Responses

On Karna-AI’s survey tool, the researchers can quickly check “smell” theme related responses with the most negative sentiment. The below graphic shows how this works. The algorithm can identify all the responses where the key theme is “negative sentiment towards smell”, even when the word smell itself is not mentioned.

Analyzing ‘smell’ related responses

Smart Word Cloud

If you have thousands of responses with negative tone towards smell, then reading all of them is a challenge. To read at scale, we generate a smart word cloud that picks up all the key topics and phrases mentioned in responses and displays them in one visualization. You also have the option to focus on very context specific word clouds. For example, you can generate one word cloud that only mentions all the adjectives used to describe smell (“foul”, “funny”, “smokey”) and another one that shows the sentiment associated with key food-products (“McChicken Burger”, “French Fries”, “Burnt Garlic Wraps”) when mentioning smell.

Keyword cloud for ‘smell’ related responses

Digging Deeper into the smart word clouds

One can dig deeper to actually read all the responses associated with a particular keyword. The idea is to use smart world clouds to read at scale and then use this to narrow down to key responses that require the attention of the researcher.

Key messages related to smell where the word ‘burnt’ is mentioned.

Finally — Unraveling the insights as a result of Survey Analysis

By deploying AI to read at scale and zeroing in on the key themes, a researcher can figure out the underlying reason for the decline in customer satisfaction. In this conceptual case study, the researchers were able to identify that one of their new products called Flame Smoked Chicken Wings was creating a slight burning smell in the restaurants which was not appreciated by customers.

With the advent of AI, analyzing open text responses on a large scale is now a reality. The idea is to let AI do the first level of reading for you and then you as a researcher can read only the specific key responses that are most likely to contain the answers. For more details on a researcher should think about analyzing open-ended text, we recommend reading Ray Poynter’s take on this topic.

When it comes to images, AI systems have become smart enough to analyze even subtle diagnosis signals in medical x-ray radiographs. But when it comes to analyzing text, the challenges are much bigger. The AI research community is step-by-step making strong progress in this area. At Karna-AI and ParallelDots, we are contributing towards this shift.

Karna AI is the Market Research AI solutions division of ParallelDots, a premier applied AI research group. At Karna, we believe AI will be at the core of successful market research undertakings of the future. Our vision is to help drive this shift.

Want to get accurate analysis? Click here to schedule a free demo.