CultureProduct ManagementProduct MarketingResearchSaeed

Your customers are not lab rats!

by Saeed Khan

mazeEverywhere you turn these days, someone is talking about running experiments and tests to understand your customers or target market.

They say things like:

  • use experiments to validate your ideas
  • A/B test responses to interface designs
  • build minimal solutions that can be built and released quickly to get ongoing feedback.

It seems that many product people can no longer make a decision without first testing everything out on users.  Some would say:

 “Of course, why wouldn’t you do things that way?”

but I’m going to take a counter point.

Just because you can run lots of experiments or  A/B test almost everything, doesn’t mean you should or that it’s the right thing to do.

In a recent discussion about improving sales effectiveness, someone actually suggested running an A/B test with sales teams; training one group one way, the other, another way, and then seeing which group did better.

Clearly this person didn’t understand that A/B testing requires ALL other variables to be held constant which would be impossible for different sales teams, with different territories, customers, objectives etc.

Additionally, let’s be honest…with exceptions such as A/B testing specific web page designs/layout over MANY impressions, most experimentation and user validation is done empirically with small numbers of customers. And the results themselves may be skewed or contain significant margins of error.

  • Did you ask the right questions?
  • Did you ask the right people?
  • Did you use the right tests?

Is anyone measuring that when the feedback is incorporated into product decisions?

Don’t get me wrong; I completely support making INFORMED decisions, but the mantra of experimentation is getting out of control.  Just because you didn’t ask someone some specific questions(s) about some issue in the past week, it doesn’t mean you can’t make a decisions about that issue.

Between Wild Ass Guesses and Calculations

Take a look at the the diagram below.


On the left, we have the (infamous) Wild Ass Guess. Sometimes called “gut feel” — though IMHO gut feel is a bit to the right of WAG —  this is a decision based on little or no data and has huge uncertainty.

On the right, we have what I call Solid Calculation. This is a decision made with a complete set of high confidence data and is clearly understood as fact.

In the middle is what I call the Decision Zone. It’s skewed to the right of center, and there is a reason for that.  Here, there is some or a  lot of data, but it’s never 100% of the data. Thus it’s a decision and not a calculation. It’s skewed to the right because you want to have a SOME level of information (explicit or implicit), to base your decision on, and as you get further to the right it starts turning more into a calculation.

We are paid to make good decisions

In his book Blink, Malcolm Gladwell writes:

“The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.” 

Experiments and  particularly A/B testing give us data and some knowledge, but where do understanding and insights come from? You can’t A/B test your way to deep insights. These come from experience, hindsight, domain knowledge and the like. In life, we use these traits everyday to make both big and small decisions, so what about work?

In fact, one of the main jobs of good managers, including Product Managers, is to make good decisions.

VERY Successful products and businesses can and have been been built without excessive amounts of testing and constant experimentation. Apple is the obvious company that comes to mind, and you can read short post here about how Apple views customer experimentation and iterative feedback.

So, let’s continue to make informed decisions, but also, let’s decide that not every decision requires A/B testing or customer experimentation before it can be made. Let’s trust in ourselves and in our coworkers enough so that we don’t waste time (ours and our customers) in unneeded data collection activities, simply to tell us what we should already know.


 Tweet this: Your customers are not lab rats! http://wp.me/pXBON-4aD #prodmgmt #prodmktg #ux

About the Author

Saeed Khan is a founder and Managing Editor of On Product Management, and has worked for the last 20 years in high-technology companies building and managing market leading products. He also speaks regularly at events on the topic of product management and product leadership. You can contact him via Twitter @saeedwkhan or via the Contact Us page on this blog.

  1. Heather Searl


    I read this with trepidation. You know I like testing concepts and designs with users. 🙂 The good news is I agree with you.

    I’ve never been in the situation where the team wanted to test too much — money and time always taken precedence.

    I would imagine that when this situation occurs the person suggesting the testing has a hidden agenda or is trying to avoid making a decision and taking responsibility.

    The key I think is to have a user insight plan for a project. The plan should start by determining where the team needs customer input the most. This could be features that the team doesn’t know much about (in terms of usage patterns), things that the company has a history of doing poorly with, a completely new idea etc.

    Many projects don’t need any new customer insight. There is no need to test things that are easy the team is comfortable with, or things that can be inferred with reasonable accuracy.

    I think direct customer insight should be done when there is a big gap in knowledge (not the same as opinion) about customer pain points, work flows etc. The point of the testing is to get you on the right path — not necessarily to validate every decision.

Leave a Reply

Your email address will not be published. Required fields are marked *