by Saeed Khan
Everywhere you turn these days, someone is talking about running experiments and tests to understand your customers or target market.
They say things like:
- use experiments to validate your ideas
- A/B test responses to interface designs
- build minimal solutions that can be built and released quickly to get ongoing feedback.
It seems that many product people can no longer make a decision without first testing everything out on users. Some would say:
“Of course, why wouldn’t you do things that way?”
but I’m going to take a counter point.
Just because you can run lots of experiments or A/B test almost everything, doesn’t mean you should or that it’s the right thing to do.
In a recent discussion about improving sales effectiveness, someone actually suggested running an A/B test with sales teams; training one group one way, the other, another way, and then seeing which group did better.
Clearly this person didn’t understand that A/B testing requires ALL other variables to be held constant which would be impossible for different sales teams, with different territories, customers, objectives etc.
Additionally, let’s be honest…with exceptions such as A/B testing specific web page designs/layout over MANY impressions, most experimentation and user validation is done empirically with small numbers of customers. And the results themselves may be skewed or contain significant margins of error.
- Did you ask the right questions?
- Did you ask the right people?
- Did you use the right tests?
Is anyone measuring that when the feedback is incorporated into product decisions?
Don’t get me wrong; I completely support making INFORMED decisions, but the mantra of experimentation is getting out of control. Just because you didn’t ask someone some specific questions(s) about some issue in the past week, it doesn’t mean you can’t make a decisions about that issue.
Between Wild Ass Guesses and Calculations
Take a look at the the diagram below.
On the left, we have the (infamous) Wild Ass Guess. Sometimes called “gut feel” — though IMHO gut feel is a bit to the right of WAG — this is a decision based on little or no data and has huge uncertainty.
On the right, we have what I call Solid Calculation. This is a decision made with a complete set of high confidence data and is clearly understood as fact.
In the middle is what I call the Decision Zone. It’s skewed to the right of center, and there is a reason for that. Here, there is some or a lot of data, but it’s never 100% of the data. Thus it’s a decision and not a calculation. It’s skewed to the right because you want to have a SOME level of information (explicit or implicit), to base your decision on, and as you get further to the right it starts turning more into a calculation.
We are paid to make good decisions
In his book Blink, Malcolm Gladwell writes:
“The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.”
Experiments and particularly A/B testing give us data and some knowledge, but where do understanding and insights come from? You can’t A/B test your way to deep insights. These come from experience, hindsight, domain knowledge and the like. In life, we use these traits everyday to make both big and small decisions, so what about work?
In fact, one of the main jobs of good managers, including Product Managers, is to make good decisions.
VERY Successful products and businesses can and have been been built without excessive amounts of testing and constant experimentation. Apple is the obvious company that comes to mind, and you can read short post here about how Apple views customer experimentation and iterative feedback.
So, let’s continue to make informed decisions, but also, let’s decide that not every decision requires A/B testing or customer experimentation before it can be made. Let’s trust in ourselves and in our coworkers enough so that we don’t waste time (ours and our customers) in unneeded data collection activities, simply to tell us what we should already know.
Saeed
Tweet this: Your customers are not lab rats! http://wp.me/pXBON-4aD #prodmgmt #prodmktg #ux
About the Author
Saeed Khan is a founder and Managing Editor of On Product Management, and has worked for the last 20 years in high-technology companies building and managing market leading products. He also speaks regularly at events on the topic of product management and product leadership. You can contact him via Twitter @saeedwkhan or via the Contact Us page on this blog.