by Saeed Khan
If you’re a Product Manager, you’ve probably conducted a beta program of some kind. It may not have been called a beta program; maybe it was called “early access” or “release preview” or something else.
Regardless of the name, the process and goal were probably the same: before releasing to the broader market, get focused feedback on key functionality from a smaller group of users or customers in order to ensure there are no major issues that have been missed as part of the development and QA process. Does that sound right?
One of the first public articles I ever wrote was on beta testing entitled Building a Beta Beta; originally published by Pragmatic Marketing in 2005. It’s quite a long and detailed article. You can download the full PDF at the link below.
| Related Download – Building a Better Beta A detailed how-to guide on running a successful beta program |
Beta programs can be difficult to run. I wrote the following in that article:
Think about it. A beta program involves having people spend their valuable time using your unreleased, and in many cases, buggy software. And on top of that, you usually have a fairly short period of time to gain valuable insights on what issues need to be addressed with the product before it is released.
Seeing it from the other side
But recently, I was able to see a beta program from the other side — as a beta tester myself — and it really opened my eyes to the challenges beta testers have in participating in beta programs. Additionally, I gained a couple of insights that I will share with you into how to get better results from a beta program.
The software is a web-based tool. It’s aimed at “business users” — a Product Manager is definitely considered a business user — and I had a specific use case that it was intended to address. The tool could save me several hours per month by automating some rather tedious and quite-honestly, brain-dead analysis I do on the products I manage. I don’t want to say more about the product as it’s not yet released, but I do have to say that it’s a cool tool. 🙂
The beta process was ongoing — i.e. there was not a fixed beta period, but my participation involved about 12-15 hours of my time over a period of many weeks.
Here’s a list of challenges I faced as a beta customer, and you should think about when launching a beta program. There may be other challenges depending on the type of product being tested, but whatever they are, eliminating or minimizing them should be part of any beta plan.
Challenge 1 – My time was very precious
As much as I wanted to spend time on the beta, it was always a struggle to focus on it given that I had lots of higher priorities on my list. e.g. like my day job!
It shouldn’t have been tough. The beta was simply a few browser clicks away — no software to install, no VM to launch, no servers to set up, no databases to start etc — and the software could benefit me directly, yet, I had to make conscious efforts to find time to use it.
Challenge 2 – The learning curve
Although the product was relatively simple and meant to be used by “business users” — there definitely was a learning curve at the beginning of the beta program. It wasn’t insurmountable by any means, but it was enough to act as a small hurdle in initially testing the product.
Challenge 3 – The context switch
Working with any new product or tool requires a context switch. I had to change the way I normally performed my tasks and think through how to do them with the beta product. This was part of the learning curve, but it was also an issue on it’s own. Even after I became familiar with the product, there were things I had to do that seemed unintuitive. Why couldn’t the product work exactly the way I wanted it to work? 🙂
Challenge 4 – The expectation of failure
When I first started the beta, there were a number of key limitations in the product that prevented me from achieving the goals I had. Each time they updated the software, I had to try my scenarios again, expecting yet another limitation or bug to prevent me from achieving my goals. After several such cycles, I did grow a bit frustrated. I could see they were improving the product, but it still didn’t meet my needs. Was it worth my time to continue with the beta?
Challenge 5 – The tyranny of repetition
I never realized it before, but beta testing is really boring. When QA teams test software, they typically have automation tools to help with repetitive tasks. That was not the case with most beta testers. I don’t know about anyone else, but I get bored very quickly when I have to repeatedly do the same things over and over again.
While none of these — with the exception of product limitations — were absolute showstoppers, each hurdle was a barrier to performing the beta testing at the level I wanted. The testing took longer than I wanted and it wasn’t always as easy as I would have expected. I stuck with it because I had committed to finish the beta, but if other work priorities had come up, I probably would have focused on those instead of the beta.
| Related article – How to know when you are ready for Beta |
Advice for future beta programs
The hurdles I faced gave me empathy for what beta testers themselves face. Even those with the best of intentions have to deal with time issues and balancing against higher priority issues. Beta testing is not in anyone’s job description. It’s an optional task they do IF they see enough value in it.
We as Product Managers have to keep this in mind. Part of planning a successful beta should include ways to eliminate (or minimize) every potential blocking issue and increase the value of the beta to the testers.
1. Eliminate the need for a product install
With a hosted (cloud) application there is no install, but if you have on-premise software, provide a pre-installed, pre-configured hosted environment. In previous betas I’ve run, this hurdle knocked out several eager beta users. They either couldn’t procure the necessary hardware, or configuration, environmental or security issues stopped them. e.g. they could install the software, but couldn’t get permission from their DBAs to create new databases etc. needed by the software.
IF testing the installation process is part of your beta program, then set up some hardware, either at your company and invite customers in, OR use a hosted service — such as Amazon or Skytap — to make the hardware resources and necessary software available. Then the user can install the product without any DBA or other limitations.
The other benefits of a hosted environment is that you can log in and see what they’ve done at any time. Also, if they run into a problem, it’s very easy to start a webex or screenshare and have them show you where the problems are. Given that most of these services are pay-per-use models, they are very cost-effective as well.
2. Have both live and recorded training available where possible
Different people learn in different ways. It’s important to understand and accommodate that. I ran a small beta last year. There were only a few customers testing specific functionality in a new release. I had 1:1 sessions with each customer to walk them through the new functionality so they understood how it worked.
We also had a detailed Word document that outlined the functionality as well so they could also reference it if needed. If the Beta had more participants or a broader scope, I would definitely have recorded a few videos to help people overcome the learning curve in a way that suited them.
3. If the customer faces any blocking issues, address them ASAP
In the beta I participated in recently, the product team was very responsive in helping me get past any product issues. They had several releases during the beta where new functionality was added. At the beginning of the beta, I think I could address about 30% of my key use case.
By the end, it was closer to about 80%. Had they not been that responsive, I probably would have abandoned the beta. The fact that they moved forward to help me move forward was a great reassurance that they were listening to me and were serious about my success. That really helped keep me engaged.
Final thoughts
There’s no magic bullet when it comes to beta programs. They are, in my opinion, one of the most difficult product development tasks to conduct successfully; primarily because so much depends on people who are external to your company and have other higher priority tasks to complete.
But a well planned and executed beta program can have significant impact on the product and the company’s top line because it can identify key issues missed in the regular development process. I’ve run many beta programs over the years and in every one of them we found product issues that would not have been identified internally. These include new key capabilities, 3rd party compatibility issues and even new use cases for some features.
The value of the beta programs was definitely worth the significant effort.
Saeed
Tweet this: Lessons learned from being a beta-tester http://wp.me/pXBON-47h #prodmgmt #beta #release
Saeed Khan is a founder and Managing Editor of On Product Management, and has worked for the last 20 years in high-technology companies building and managing market leading products. He also speaks regularly at events on the topic of product management and product leadership. You can contact him via Twitter @saeedwkhan or via the Contact Us page on this blog.