TLDR: WATCH THE VIDEO (click above)
We run a lot of surveys at Irrational Labs, so I was excited to do a teardown of one of the most popular survey apps – Qualtrics, a.k.a. “SurveyMonkey on steroids.” But how good is its onboarding at showing me all its capabilities?
3 things you’ll learn from watching this:
Where in the flow Qualtrics uses “forced choice” – & why it works
What Qualtrics does for advanced users, but not new users
Why you should use onboarding to set the mental model of everything your product can offer
See you next week with an all-new teardown 🎬 Which one? Let’s just say it “Looms large” on any list of screen-recording apps 😉
👉 This is a NEW series of product teardowns. Subscribe below to get future ones. Hit “subscribe.”
Have a friend who would enjoy these teardowns? Click the button below to refer them (& earn some great rewards)👇
👂 Transcript:
Okay, we are here at Qualtrics. Qualtrics is basically SurveyMonkey on steroids. It's what most academics use for all their studies. It's what Irrational Labs uses. And we’re going to pretend that we are a new user. We are not. We already have an account, but we will pretend we're a user, sign in, sign up, and see what the experience is like.
So first we're going to say “Free account.” They're really going hard with the free, but they don't love this button “Request demo.” It's like so much is going on the screen. As we mentioned before, contrast is the main thing that drives clicks. So you want your button to be big and high contrast.
There's not a specific color except just to say make it stand out. This is obviously really hard to see in this psychedelic thing that's going on behind it. But we're going to say free account. So up here, I'm sure that they have other CTAs, not really, really pushing towards the demo.
We'll go with the free account. Okay. So they're going to ask me how I’m using it. This is what they care about because they do discriminate pricing. And we'll say “Personal” and then we're going to sign up because I've already signed up with so many emails we're going to have to use this one. Okay, great. It's going to send me an email. Hopefully it's quick. Wonderful. I'd already reset my password. Uh oh. It's not going. Great. Verify email. Okay. So it's pre-filled with things that it thinks it knows about me. That's a little annoying, but, you know, these, these types of fields are just easy to answer.
Don't really get a lot of drop off usually. So I'm not too concerned about this type of annoyance. Really, the things that have drop off are the things that are really hard to answer. Take a long time. You have to go up to your computer and get something and come back. So we're setting up the free account.
It's interesting. Doesn't really save the correct one. Add to LastPass. Interesting. Okay, so we are here. At their onboarding. This is the first thing I see. Interesting. So basically this wants me to do three things. Discover, start a blank survey, and upgrade account. So first, I like that there's forced choice.
I can't enter the product without actually choosing a way to enter the product. And I do like that they're pushing the compromise effect to start a blank survey. Hopefully if I can create a survey, my time to value will be very quick. Discover Qualtrics. I bet that they could sell this a little bit more.
You know, what are three things that new users like to see, or what are the mistakes that old users make that you can avoid? You know, you can really kind of sell me on discovering Qualtrics. Right now it seems a little bit lame. And then they're going hard on the “Upgrade account.”
I just got in here. We just met. Are we really going to take me out on a second date when we just had we ordered our drinks? Probably not. So this seems to be the obvious and only answer. Start a blank survey. Oh, and I have to click continue. So this is how their surveys work, where sometimes you have to click continue, but it does feel a little weird from a UX perspective.
Okay. So there, this is like just way too much text. So when folks are going here, you know, do you really want to take a tour? To take a tour, you have to acknowledge that there's so much to learn and you want to do it all right now. I'm sure a percentage of people who do this but the majority of folks are going to say “Get me in there.”
Especially when you can see it right behind you. All you have to do is close the screen. Get out of my way. Okay, wonderful. So I'm writing in survey creation mode. This is wonderful. Okay, here we go. So I know Qualtrics a little bit, but if I didn't, this could be a little bit overwhelming.
So basically, right to have a question. I understand that I could choose different types of questions. The reality is Qualtrics is so powerful. Qualtrics is one of the most powerful survey platforms ever. And so yes, they have to simplify it for me as a new user, but it can do so much and so it’s curious that I've gotten to this point without them asking me why or what survey I want to create.
Contrast that with our logged in paid version. If I click “New project,” it's going to land me on this really fun template screen where I can just be like, “Oh wow. I should do manager feedback, team event feedback.”
I don't have to create that. How great. Right. And so these are really interesting ways to anchor me in different ideas. We have, by the way, another Qualtrics account, and they do even more. Maybe they're testing different homepages, but you know, for one, we vote and rank.
So it's funny that they're doing this for advanced users, but they're not yet doing it for a new user to really guide me to the point where I'm comfortable to write one of these, to write one of these surveys. So we'll go back here. We'll basically say, how was your tear down with Kristen… Sometimes it pre-fills it so it says “loved it” or “I liked it a lot.”
See if they do it. Oh no. Sometimes they'll pre-fill it. Seems like they’ve got some AI going on here. Now here we can say published. So curious what happens here. Obviously this is kind of just like a sandbox for a new user. Do you kind of expect your new user to just like test something out really quickly before diving fully in and so that's what I'm doing.
I'm playing like a new user publish. And it's been, it's been activated. So at this point I haven't used any of the upgrade or features. I also haven't been exposed to them. So if I'm Qualtrics and I want to expose me to something, let's go back here. We'll go back to the homepage.
Okay, so it's Untitled project. I still wonder if I create projects again. Will I be, oh, here we go. Okay, here we go. They're really pushing the tour here. Now is where we get to some of the ideas of the template. So they didn't have that on first use, but it seems like they're introducing it here. Very nice.
Now, if I were Qualtrics, I would probably gray out some of the premium features here. So it seems like in my premium account, I may have more templates in this account. I have less templates, but you could imagine, I don't know what I don't know. I don't know that there's even more templates available. And so how would you show the user that? You'd probably show that to them and gray it out and say that this is a premium feature. Wonderful. And so here we have, we have more templates, more templates. So in this case, basically Qualtrics knows zero about me, except that I'm using them for personal use.
By the way, personal use - probably everyone knows that if you say business use, it may charge more. So I would be very suspicious of that data, Qualtrics. But at this point I would probably see if during the onboarding, if you could ask somebody more questions that you can then help them get exposed to the premium features, get exposed to, even just use the product more. Imagine you knew that I was doing a big study and I wanted to find a thousand people to give me product feedback, right? You could obviously just show me the product feedback template right away if you'd ask that question up front.
Instead, they kind of just drop you in. And so this is the general thing to weigh with onboarding is that you want to set the mental model of everything that the product can offer. You also want to decrease friction and get people the time to value really quickly. I would say probably in this case, Qualtrics went a little bit overboard with getting me time to value without exposing all of the product features that it does.
We really don't want people to think this is a glorified SurveyMonkey. It has so many more features than SurveyMonkey has. No offense, SurveyMonkey. Survey Monkey is great for a certain amount of things but yet as a free user, I have no idea.
Questions about your product? Email kristen@irrationallabs.com.
How Qualtrics Uses Onboarding to Set Mental Models: Product Teardown