Mailchimp: How not to do customer discovery 🙈

TLDR: WATCH THE VIDEO (click above)

When it comes to user feedback, talk is cheap. And today we’ll find out why— by taking Mailchimp’s ‘15-minute’ customer feedback survey (spoiler: it felt like 50 minutes).

On one level, this is a teardown of a feedback survey. But it’s also a map for how to change your classic ‘traditional’ research questions into more predictive and informative product insights. And the key here is not letting people do cheap talk.

What is cheap talk? It’s asking ‘Do you like this?’ Even if the answer is ‘yes’, it doesn’t matter. Because there’s no skin in the game.

This is well known in the pricing literature. Under hypothetical settings in willingness-to-pay studies, people state a higher valuation than their actual valuations. We call this phenomenon the hypothetical bias, a.k.a. cheap talk. (Harrison and Rutström 2008; Murphy et al. 2005; Wertenbroch and Skiera 2002). Basically, people report a higher value than they’re willing to pay in reality. To overcome this bias, ‘incentive-compatible methods’ have been developed to reveal people’s true willingness to pay. This means if you say you’ll buy something in a survey, there’s a real chance you will actually have to buy it!

In this vein, to get real customer insights about their preferences, we need folks to make tradeoffs or have some cost to their decision. Instead, Mailchimp’s survey fished for compliments— or tried to validate features in a way that seemed more aimed at crafting user personas than meeting my needs as a customer.

This isn’t trivial. Asking the wrong questions can lead to misguided conclusions, which can influence product development in unproductive directions.

So today, we’ll give feedback on the feedback survey. We’ll talk about time expectations, incentive design, and of course, question design—and why I’m nervous for Mailchimp’s product development if they don’t change their approach.

This isn’t just about pointing fingers at Mailchimp. It's also a lesson in how to do product or customer discovery that actually works).

Things I cover in this teardown:

📝 Why someone who takes a 15-minute survey is NOT your average user (& how to solve for self-selection)

📝 How to design financial incentives in your survey that actually work

📝 One of the hardest questions in survey design and how to get it right

‘How likely are you to consider returning next week for another teardown?’ Don’t answer, just subscribe 😆👇

See you next week 👋

Have a friend who would enjoy these teardowns? Click the button below to refer them (& earn some great rewards)👇

Questions about your product? Email kristen@irrationallabs.com.