The free iPad Question

One of the most challenging parts of being a product manager is being tasked with finding out what customers really value. Yet unless we ask the right questions in the right way, we may be getting answers that are lead us to wrong conclusion. Here is how to avoid that from happening.

Steve Blank rightfully told us all to “get out of the building”, meaning that we need to actually talk to customers and not only that, but observe them ethnographically (that is to say, in their natural habitat, doing the things they are doing).

But, in reality, it’s more complicated than getting face time with a customer because:

• People don’t want to hurt your feelings and will be polite (i.e. lie to you. albeit with the best of intentions).
• People do not like confrontation, so may often be personalities that do not challenge what you say but instead viscerally accept it even though it’s not what they think (meaning they lie to you).
• People have different aspirations from reality. We all think we’re going to go to the gym three times a week next week (meaning they lie first to themselves, then lie to you)
• People are concerned about being judged, so will not want to admit true behaviour they feel guilty about in a room full of people (meaning they lie to whoever is present then lie to you).

If you are aware of the common sources and conditions of bias, then of course there is plenty you can do to avoid it, such as asking questions like “when was the last time that..” and “last year, how much did you..” which ask for factual details of previous behaviour and asking for time, money or recommendations (basically, three forms of currency) instead of meaningless questions like “would you in the future buy…” or “might you be interested in…”, which are variations of the going-to-the-gym problem we spoke of above.

But here is the biggest thing not to do when interviewing customers. Don’t ask them if they would like a new feature. Instead ask them what they would choose if they could only have one of A or B. Then ask why that choice was more valuable. Then ask what difference it would mean to have it. Then ask what would change on a day to day level if they had it. Get to the root problem or benefit.

What you don’t want to do is say “hey, would you like a new feature that lets you chat directly with clients in real time (say)”. This is equivalent to asking someone “do you want a free iPad”. If someone says yes, you have no idea if they put any value in it, just that they will take something if you are giving it.

I spoke with someone recently who had user generated content, where users did some actions on the site without being paid. When asked if they would like to be paid for doing these actions, 90% said ‘yes’. This is a free iPad question.

Of course, a better question would be to ask those users which of the following would make you do more of this desired action: A, B or C (where A could be unlocking some features, B could be greater visibility to other users and C could be money)? And then drill into how much more etc..

I know this seems a little silly, but I see this fallacy all the time. As Voltaire said:

Judge a man not by the answers he gives, but the questions that he asks

If you work as a product manager, it has never been more true.