Experience Strategy Podcast: How Conjoint and Max Diff Can Help CX Leaders Make Better Decisions With Craig Lutz

To listen, click HERE

Experience strategists are navigating increasingly complex decisions. Customer journeys are increasingly complex, channels are proliferating, expectations continue to rise, and data is abundant. How do we help our companies make the best decisions about how and where to spend money?

Craig Lutz is uniquely well-positioned to help us answer this question. Craig has been a data scientist and researcher at Qualtrics since 2007 where he has conducted thousands of conjoint analysis projects with Qualtrics. He is also the designer of Qualtrics' conjoint analysis DIY technology, the author of Exploring Conjoint Analysis, and a prominent thought leader on innovative quantitative research methods. We are thrilled to announce that he also recently joined our team at Stone Mantel as Conjoint Advisor!

In this episode, Dave and Aransas talk to Craig about how he helps teams like Google, Goldman Sachs, Uber, and others make the decisions that really matter for customers, based on powerful analytics and what shifts when companies understand that time really is money.

Voiceover: [00:00:00] Welcome to the Experience Strategy Podcast, where we talk to customers and experts about how to create products and services that feel like time well spent. And now here are your hosts, experience nerds, strategists, Dave Norton and Aransas Savas.

Aransas: Welcome to the Experience Strategy Podcast. I'm Aransas Savas.

Dave: And I'm Dave Norton.

Aransas: Dave, as experience strategists are navigating increasingly complex, customer journeys are more complex than ever, channels are proliferating. Customer expectations just continue to rise, it seems like exponentially every week, and there's more and more data available.

I think so many experience strategists feel a little bit lost, frankly, and overwhelmed about how to prioritize their investments and their customers and [00:01:00] how to look at the data that really matters to inform their decisions. So the big questions we want to ask today are how we can help our companies make the very best decisions about how and where to spend their money.

To answer those questions, we are joined by Craig Lutz, who is uniquely well positioned to help us answer the question. So Craig has been a data scientist and a researcher at Qualtrics, which probably all of you out there have heard about since 2007, and he's conducted literally thousands of conjoint analysis projects with Qualtrics.

He's also the designer of a tool that I personally used in my last organization, Qualtrics' conjoint analysis, DIY tag. Craig is the author of Exploring Conjoint Analysis and truly a prominent thought leader on innovative quantitative research methods. And we're thrilled to announce that he also recently joined our team at Stone [00:02:00] Mantel as a conjoint advisor.

So today we're gonna talk to Craig about how he helps teams like Google and Goldman Sachs and Uber and so many others make the decisions that really matter for their customers based on powerful analytics and where he sees the future of quantitative research heading. Craig, welcome. Thanks for being here.

Craig: Thank you for having me.

Aransas: Absolutely. And we're so excited to have you join the team.

Craig: Yeah I'm excited as well.

Aransas: I know Dave knows you pretty well, but I'm gonna get to know you a little bit better over the course of the day along with our audience who may be meeting you for the first time.

Before we talk about your career history, tell me a little bit about how you got into this field. What excited you about it?

Craig: Yeah, so I've always really liked math. I grew up loving math. My dad was a mathematician [00:03:00] and so I was studying that at school and then randomly got a job at Qualtrics while I was still in school.

And I was kinda the only math guy at Qualtrics. And so there's a need to run some, a little more math heavy type projects. Specifically conjoint analysis and probably just because a lack of a better choice, they asked me to help and so had to learn on the fly early on.

But I just really fell in love with this idea of this union of math and marketing using statistics and mathematics to help understand people's decision making and prioritization. So that's kinda how things got started. I was the best choice available of probably a bunch of bad choices in terms of the small number of employees we had.

Dave: I don't believe that for a second, my guess is that Ryan Smith knew exactly what he was doing when he hired you. You were like employee number 10 or something like that, weren't you so early?

Craig: I was in early, early days, basement days. I was, like I said, I was actually still at Brigham Young University and got hired while I was still in school. So I lucked out that way, but It's been a fun ride.[00:04:00]

Aransas: Wow. What a ride indeed. So now roughly 15 years into this work, what do you believe is the power of marrying the art and science of experience?

Craig: I think conjoint analysis and other types of quantitative research are just like a great opportunity to open the eyes and really try to put use survey data and leverage survey data in a unique way where we're not just asking like blatant choices.

Like on a scale of one to 10, what would you give this product? We're putting respondents in kind of real world experiences. Having them make choices as if they were purchasing or were experiencing something, and then using those decisions, those discrete choices that are made to help enlighten us and really inform these really important, powerful decisions on what we should go to market with, or what we should price something, or how we should package something. I think more and more it's starting to catch on this idea that we can really have these really well-informed decisions [00:05:00] using methodologies that really model real world experiences.

Aransas: Yeah, I'd love to hear an example of how that's helped a brand make a better decision that they felt more confident in.

Craig: Yes. So we've ran thousands of these but groups like Uber, like trying to understand, we recently, or maybe been a couple years now ran a conjoint analysis in partnership with them to try to understand, okay, how can we get more loyalty from our riders?

What are some things in our loyalty programs to get people to be loyal to our brand, to ride more, to have a better experience? And so we tested different things like ride benefits and eat benefits and timing and convenience and drivers and all these different variables to understand, okay, what are some things that we can offer to our customers to improve and optimize their experience so that they wanted to keep using Uber, maybe not some of the competitors. And so we found things like eats, like improving the eats benefits actually drove and improved the experience, maybe some of the ride benefits that you may [00:06:00] might expect with but offering more points towards restaurants and things like that.

And that in turn, even helped their business more because people were using Uber to go to those restaurants and those breweries and those bars. And there's just some really enlightening insights that came from these studies what are some things that they can include in the loyalty programs to make people excited about riding with Uber and having a great experience.

Dave: That's fantastic. I've heard you tell this story a little bit before, but it just dawned on me as you were describing it that their loyalty program, the Uber loyalty program started with how can we get people to prefer us when it comes to riding, and you found this really powerful thing with Uber Eats, where they wanted that as a benefit component. What that reminds me of Craig, is Clayton Christensen's work on jobs to be done because one of the things he said is that the companies that are most successful are the [00:07:00] companies that identify multiple jobs that their customers can get done. So I no longer just use Uber just to get me from one place to another. I now use Uber to help me get food to me. And those two things work together synergistically. They create greater loyalty. I hadn't thought about that particular piece and your analysis brought it to life for them. I think that's amazing.

Craig: Yeah, absolutely.

Aransas: I think it can be really overwhelming because this is such a great example of how companies can strategically plan their roadmaps based on data that they feel confident in and one of the things I think that's really exciting about the way you do this is in all instances, you're putting people into real life situations and as somebody who has a deep background in behavior change and customer experience, that's really interesting to me because I think about the difference between what people say they want and what they actually want. So how does this type of [00:08:00] analysis get closer to understanding their true actions and behaviors or intentions?

Craig: Yeah, great question and nothing's ever gonna replace having a customer go to a store and actually experience it and purchase it and view their decisions there. That's always gonna be the best source of truth. But sometimes we have to learn some things and have some insights before we can even get to that place. Or sometimes that's prohibitive because of costs or convenience. And so that's where there survey data can come in is we have to do some things maybe before we could offer that kind of experience or before it could even go to market with something. There's different ways to gain these insights. Traditionally, it's maybe been not done the best way in that we've asked respondents, okay, on a scale of 1 to 10, how much do you like this food option? And they would give it a rating. But the problem with that is what does my seven mean versus your six versus Dave's eight?

What does that actually mean? And where conjoint analysis is great is we put actual [00:09:00] packages, we formulate different packages and simply have a respondent select is the package A better or package B? And it might have, if we're doing like a dinner option, it might be like package A might be chicken and french fries and a soda.

And package B might be steak, a salad, and water. And then they both would have a price as well. And which one of those two would I prefer more? So I would choose package B or package A or package B. And then we'd run through a number of scenarios of that. And so why that's a much better approach. Okay, on a scale of zero to 10, how much do you like steak, is we're trying to pull out some of the biases and trying to represent and reflect, what would happen. When you go to a restaurant, they're never gonna ask you on a scale of one to 10, how much do you like steak?

They're gonna have some different options and different menus. And then you're going to make your selection. The real power comes from this is that for the respondents through some real world experiences where they're making selections, and then from that we can extract, run some multi-variate regression to really extract, okay, what are the key drivers? What's really moving the needle? What [00:10:00] are people willing to give up in order to get, are they really price sensitive? Do they not care about price as long as steak is included? Is salad so much better than fries, any plate with salad. So we can learn all these different things of what really moves the needle.

And we can do that for every individual and so we can aggregate that. The whole population or can aggregate that for males versus females or people in this area versus this area, people with kids, without. So we can do all these different type of things because we can understand preferences of individuals as they go through these different scenarios in the survey.

It'll never replace the power of hey, we had customers come in and this much steak was bought and this much chicken was bought. That's always obviously going to be the best data, but this is probably the best way with survey data that we can really understand the decisions and trade offs that customers would make.

Aransas: That's very cool. I love too this idea of partnering this with other types of data. So I think what you're saying is this isn't data that's meant to be used in isolation. Do companies make big decisions [00:11:00] based on this type of data or is it usually used in concert with observational data and qual and other types of analysis?

Craig: Yeah. Yeah. Great question. I would never recommend that this is the only source of truth that you ever use. I'm probably a strong as believer as anyone in conjoint analysis and the power of it. I like to think about it like there's this truth inside a house, but if I only look at it through one window, it might be dis distorted, I might not see the dimensions of it, but if I can look at it from a window on the other side I can triangulate it and I can see dimensions, I can see the full shape.

And so I'd always recommend using as much as available and within budgets and things like that. I'd always recommend backing this up with some qual. This is the package we think is best. I'm gonna go take this to 10 potential customers and do a in-depth interview and say, okay, is this really what you prefer?

Do you like this more than this other option we've been thinking of, would you actually purchase this? So I would absolutely agree with you in multiple views. [00:12:00] And at the truth, what will give us the best answers and the best insights.

Dave: Projects that I am most proud of in my history of working are just exactly those projects where it was all about triangulation.

We at Stone Mantel traditionally have been known for our great qualitative, our really good observational research. But when we have partnered with people who do strong conjoint and we come to similar conclusions. Even if those initiatives are going on at the same time, oftentimes you think that it has to be quant, qual, quant, or qual, quant qual in order to be successful.

That's not necessarily true. You can actually have both going on from a discovery standpoint and have a very rich response that helps you to further refine what it is that you're trying to accomplish.

Craig: Yeah, absolutely. The fun times are when you run some qual and then you run some quant or vice versa, and you [00:13:00] get totally different answers, and then you gotta try to figure out, okay, did we ask the right question both times? Why did this happen? Sometimes you gotta go all the way back to the drawing board and start over. I've definitely seen that happen, but for the most part a high majority of the times we've been fortunate enough to see the qual and the quant kind of marry up and provide a pretty clear picture on what we should do.

Aransas: I know you're working with a lot of companies that have really huge budgets like Google and can ask these questions from lots of different angles, can look through every single window five times, because frankly, not only do they have the resources too, but they also, they have the responsibility too, right? Given what they are able to invest in this learning and what they have at stake for that matter. So what can we learn from them for companies that may not be Google and Amazon sized?

Craig: Yeah so I would always say that some data's better than no data. And so there's always probably some responsibility to go out and try to prove [00:14:00] hypotheses and try to glean some kind of insight that can help better formulate a decision.

Yeah, not everyone's Amazon and Google and has rather large budgets. But everyone has the capability to go out and try to get insights, whether it's just doing some simple qual interviews with people on the street or family members.

There's always a way for us to get some information, and I would always argue that some data is better than no data, even if it is limited. And there's creative ways, whether it's crowdsourcing. We ran a project, just a personal fund project where I just crowdsourced on LinkedIn.

I put it on there and got some people in my network to respond and was able to get some pretty fun insights for just a side project by just putting it there. There's usually probably some creative ways that you can go and collect the data. I think it's in our best interest and I've never regretted getting data even if it's limited, even if I don't have the budget. I've never regretted having more insights and more data to help inform decisions.

Aransas: Yeah. And I think part of what I'm hearing and what you're saying is that actually [00:15:00] capturing the data isn't that costly or difficult, right?

You can create some real world scenarios. Ask people to pick and choose. Will it be as sophisticated as what you're doing? Maybe not, and will the analysis be as deep, but if you have a small data set, it is something that at a very basic level, even the smallest coffee shop owner could do.

Craig: Yep. Absolutely. Couldn't agree with you more.

Dave: Absolutely. Craig, one of the things that we're excited about at Stone Mantel to be working with you on, is this idea of situational analytics. And we've talked about it a little bit on the podcast in the past. It stems from this idea that companies have spent a lot of time trying to understand who their customer is, and there's only so far they can go by trying to understand customer preference to accomplish what it is that they want to accomplish.

[00:16:00] But if they were to change their focus and instead of spending so much time on the who, Instead, focus on the situation or the what is going on, that they would find some very surprising, very important, and also very intimate things that are occurring that they haven't been paying attention to, and that are closer to this idea of getting the job done for the customer.

Can you talk a little bit about this dynamic between situational analytics versus preference analytics or who versus what. What are your thoughts on that?

Craig: Yeah so I think a lot of it has been the who. It's been trying to understand what's the how can we get the most bang for the buck in terms, okay, if we offer this and this, who would buy it and what would they pay for?

But I think a really interesting component that conjont analysis could be a great mechanism to better understand [00:17:00] is the what. Is trying to understand what they're trying to accomplish, what's trying to what jobs are they doing and is the time devoted to that time well spent.

I know that's a big thing that you talk about, Dave, is time well spent. And so traditionally the conjoint analysis, a term we use a lot is like where can we get the most bang for the buck spent? So bang would be like the most preference, the most interest for the dollars that we're going to invest.

I think a really interesting, really untapped area of work in conjoint analysis could be utilized is how can we get the most bang for the time spent, whether that's the job getting done or whether that's minutes or whatever it might be. I think there's a really interesting component that you've kind opened my eyes to of like, how can we better understand the time that needs to be devoted and how can we change the optimization algorithm to the most bang for the time devoted rather than the bang for the buck that we've been focused on for years and decades.

I think there's some really interesting opportunity there but I don't think a lot of that's, I've never done anything [00:18:00] like that and I think there's all kinds of avenues where that could be a really cool thing to do.

Aransas: It's funny too that you use that language because we've talked to many thousands of people about their perceptions of time and in qual I can't think of a single conversation we've had where when we say talk to me about your thoughts around time, that within the first maybe three to five sentences, somebody in the room hasn't said time is money. And so that buck connection has been made in our minds in a very deep way. It is as entrenched in a value system as the almighty dollar.

Craig: I think customers think that way, I think, but I don't think that research, at least with conjoint analysis, we haven't thought that way. We've always thought about the buck being invested and a dollar of the dollar.

We haven't really thought about people's time and what it takes to invest in terms of effort and energy and time to accomplish something. Yeah, I think it's a really interesting idea and a lot of [00:19:00] opportunity there.

Dave: Yeah, we're just at the beginnings, I think of something that's really big for the industry because I do think that everyone that we've talked to, all of the companies that we have shared out with about time well spent, they've all said, yes, we've gotta do this.

But they haven't thought through the fact that you really can start to look at the situation that people find themselves in through conjoint. Through Max Diff. You can do that type of work. You can have the same kinds of benefit statements, but replace maybe money with time and you're in a totally different ballpark. That's one of the things I'm most excited about that we're going to be doing together.

Craig: There's a lot of times we do this like willingness to pay calculation with conjoint analysis where price is one of our [00:20:00] variables and we have other attributes and the price that someone pay for.

I think just as interesting, it'd be like, rather than price, it would be the time necessary to complete that task or the time necessary to have that experience and then we can get like this willingness to commit the time variable rather than this willingness to pay. I think there's some really cool applications there.

Dave: Yeah. I love it. I love it.

Aransas: It's really exciting, . So when do we get started ? What are the next steps, guys? What are we doing?

Dave: Yes, not just being funny, but we're already well down the path with the work that we've been doing over the last five to seven years, just on time well spent. The frameworks are all there and there have been a few companies that have started to work on this work of situational analytics. Not the least of whom is Aransas' former employer. They've done some great work. They're at the forefront and part of that due to Aransas' [00:21:00] leadership while she was there.

So, I think there's a lot of opportunity, and I do think that as companies struggle to figure out, where they go next, they're at a plateau. They've hit a point where there's only so much that they can extract out of the data that they have. By just changing the lens a little bit, using the same tools, the great technologies that are out there like Qualtrics. They can begin to get to a next level. They can go to a different place, and that's what we want to do together.

Craig: Yep. Absolutely.

Aransas: So I think the next steps really are that we're going to go out and test this with a bunch of our partner companies, and gather more data to understand this even more deeply and I think while we've been hustling away at it with our unique skills and tools. The beauty of having you join the team is that we'll be able to look at even bigger data sets and be able to fully [00:22:00] put more sophisticated mathematics into our understanding of customer needs.

Dave: Yeah. Love that.

I have got to hear the story from you about Goldman Sachs and their employee experience. Cause while we're talking a lot about customers and consumers, a lot of the principles that we're describing also apply to employees. Could you tell us a little bit about your work with Goldman Sachs?

Craig: Yeah so I actually think Qualtrics has thought about experience more than just the customer experience, where we also focus on employee experience, brand experience, and product experience. And I think there's a lot of applications in all four of those of where conjoint analysis and max diff.

Which is kind a variation of conjoint analysis can be really utilized, to understand brand and product and customer and employee experience. And so with Goldman Sachs we ran a really large conjoint analysis exercise with them on their employee benefits. So they had of their core benefits, that they had in place.

But they were [00:23:00] trying to figure out, and this was right before Covid, they were trying to figure out what are some additional benefits that we can offer that would really move the needle. We have again, we're going back to budget and dollars, but we have budget allocated to try to retain our employees.

And we want to invest quite a bit into improving our benefits so that we can keep employees around and improve the culture, improve the experience, and make them more excited to come to work and be a part of the Goldman Sachs family. And we ran a conjoint analysis exercise to and we threw all these different benefits some wild, some more traditional, doggy daycare was one of the things that we tested, but then some more traditional ones like adoption and fertility and elderly care and some more traditional ones. So we tested all these different things and we then asked the employees, we ran them through some different exercises.

If this was the benefit package that included X, Y, and Z, is that one better than this benefit package that included A, B and C? And we ran them through a number of exercises like that where they simply made a discreet choice on which one they preferred. And then [00:24:00] after we collect all that data we ran it through our statistical models and we understood. Okay, these are the things that matter most to the employees in these different offices. Because they had offices all over the world and some people in Singapore were very different than some people in San Francisco and had different needs. And so we segmented the data by office and then we say, okay, this is the budget, like per employee budget that we have for each of these different offices.

And so then we went and ran some optimization algorithms and said, okay, in this office if we included these two we're gonna increase preference more than if we included these other ones that we were thinking of. And so what we did is for each office. We found the optimal bundle of usually two to five things that we could add within our budget constraints to improve the employee benefits program. That was a really fun project. I feel like we did some cool things. The winner, or excuse me the person that I worked with on that, on the Goldman Sachs side ended up winning a big award, a Qualtrics award or X4 Summit for being a breakthrough artist because of the great work they did and [00:25:00] the accolades he got both internally and then in the HR space. Really cool project but it just shows that what conjoint analysis can do in terms of helping understand where can we get the most bang for the buck? Where can we improve experiences and really try to understand preferences of people and provide them the things that really move the needle. So yeah, that was a really fun project to work on.

Aransas: I had not thought until you shared that story either, what a powerful change management tool this is so that you're able to play back to customers certainly, but especially employees, how they have informed this through their own preferences and to quantify some of these decisions that frankly, in organizations can often feel a little arbitrary or confusing from an individual perspective.

So I'm curious though. That aside, what were the big preference surprises that came out of that? Was there anything that you were just like, what, that's what people care about?

Craig: Yeah. There was definitely some ones that like weren't thought, I mean, every office we actually did it for 14 different offices [00:26:00] and so every office is a little bit different, but I don't think the doggy daycare ever won out. But there was like some, one office really liked the free yoga, and so they had yoga programs that people that came to the office and ran yoga multiple times of the day was I think something that had really high preference and was something that was relatively inexpensive compared to some of the other ones.

So there are some things like that, that rose to the top, but the employees voiced that they really felt like they were being empowered to be a part of the decisions being made. They felt like Goldman Sachs the head of benefits wasn't just, oh, this is what I wanna do, so this is what we're gonna do.

They included the employees in those decisions. They empowered them to provide the insights and then they went forward with the things that really mattered most to them. It was kinda a win all around, like the employees were happy and that drove the employee engagement and the experience that the employees were having and that's a better all around experience for everyone at Goldman Sachs.

Aransas: So cool. So cool. So Dave, I'm afraid it's that time where we have to start to wrap this up. So I'm going to ask you to go first. What are you taking from this? What do [00:27:00] you hope all experience strategists will take from this?.

Dave: When I think back on my career and the organizations that I've worked with over the years who have done conjoint, there was this history way back when conjoint was used to upfront determine whether or not a product was a good idea.

And there's still room for that. But now because of all of the wonderful tools that are out there that have been developed, conjoint can be used in very different stages of experience design and delivery, and it can be used in ways that are new and go beyond kind of the traditional marketing approach.

That's what I'm excited about. That's why every time I talk to Craig, I'm like, oh, I just love tapping into your mind, man. You are amazing. Because there's something really exciting about this highly [00:28:00] understood, mathematically clear methodology being applied to very different situations.

Aransas: I love it. I love it. And how about you, Craig? What do you hope everyone will carry from this?

Craig: I've always obviously am passionate about conjoint analysis. It's been a part of my career and I've really enjoyed it. So much so that we've even used Max diff to to help name a child one of our kids, in our family.

Dave: Do you have a kid named Max?

Craig: No. So we actually put a bunch of names into a max diff exercise and me and my wife took the exercise a bunch of times, so we were both respondents, but we did it a bunch of times and there was some randomization at play there, and so we're not seeing the same survey every time.

And we selected the name that we liked most and the name that we liked least. I think we probably could have got to the same answer just because we liked the name already, but then I wouldn't have the story. It's fair to say I'm really passionate about these things and [00:29:00] I hope that other people can find just the incredible applications that conjoint and max diff can be applied to, and they're really rich insights. And really like quantifiable insights that can help with predictions and likelihoods. There's these really clear numbers that can come out of it that can really provide some confidence and some backing that you don't feel like you have to just go with your gut alone.

You can use these insights to help inform your decision. I hope that other people that maybe hear this can see it can be a little more interested in conjoint analysis or have some more questions and find all the good that conjoint analysis can do for the decision making that we have.

Dave: Is your wife a math geek?

Craig: No. No. She's not.

Dave: She's a very patient woman. She's a very patient woman.

Aransas: I'm gonna use this argument tomorrow though with my seventh grader who asks me every day why she has to take these advanced math classes. I'll tell her that it's so that she might be able to make better decisions later in life.

Craig: Yep. Yeah, absolutely. Yeah, absolutely.

Aransas: Good stuff. I think that's certainly true for experience strategists, like you said, [00:30:00] whether it's for customers or employees, people have preferences. They care how they spend their time and money.

And as organizations, if we are going to add value instead of detracting from the value of our company, or our experience, we have to know what they choose and what they care about. And I am so excited for us to collectively build this next chapter of collective understanding together. It feels so ripe and full of, use the word breakthrough, massive breakthrough potential for all of us.

And it really does feel like we were at the beginning of something that could change everything for the long time. So I am excited to welcome you onto the team. Excited to work with you and Experience Strategy podcast listeners, I hope you'll keep listening closely. This is not the last you'll hear from Craig, and it is certainly not the [00:31:00] last you will hear of his ideas.

So please reach out to us if you have questions about how to use these ideas or check out the Experience Strategy Trend Report. It is on our site, stone mantel.co trends report and on there you can learn some ways to apply the power of context to any experience. Thank you as always for listening.

Craig, thank you for being here and for all you're going to continue bringing to the table.

Voiceover: Thank you for listening to the Experience Strategy podcast. If you're having fun, nerding out with us, please follow and share wherever you listen to your favorite podcasts. Find more episodes and continue the conversation with us at experiencestrategypodcast.com.