Organizations are adopting experiments at a higher rate than ever. Thanks to new experiment tools and a general experimentation culture shift based on successes from experiment-centric tech giants, organizations are realizing that they, too, can leverage the power of experimentation.
But don’t experiment just because others are doing it.
Experiment to learn so you can build better products and make better business decisions.
So how might experimentation benefit you? Read on, data truth seekers!
What is experimentation in a general sense?
To experiment is to try something new. From a very young age, humans are constantly — and naturally — experimenting. For example, we learn what we like to eat by trying different foods. Live, experiment, and learn. Informally, experimenting helps us learn about the world around us.
In grade school, we learn about the scientific method. We are taught to identify a hypothesis, test it, review the outcome, and modify and/or develop a new hypothesis to learn more. Rinse and repeat. Our informal process of learning about the world through trial and error has now been formalized into a scientific process.
For many, this was the end of the road for formal experimentation unless one went on to become a doctor or researcher. Thanks to the explosion of new data, new tools, and a desire to make better products, tech has been added to this list. But in tech, we’re just getting started…
Always be experimenting
If you’ve ever worked at a large tech company, odds are you’ve seen some form of experimentation. From updating button colors to launching new products to evaluating spend decisions, experimentation has become the gold standard within organizations. Why?
You experiment for the same reason you did so as a kid and continue to every day: to learn. The beauty of experimentation is that if you design and execute an experiment correctly, you’re guaranteed to learn. When you have limited resources, few things are as magical.
If you’re running a business, you experiment to be able to measure the impact of the work your team is doing. This can help justify more (or fewer) resources, prioritize a roadmap, and help predict and model future improvements. This can also help you consistently move in the right direction when iterating, specifically with initiatives that require a lot of human intuition.
Identify subjective opinions that you want to turn into objective facts
Everyone has opinions, from CEOs to interns to customers. Many use their opinions to craft narratives to understand how things work. Without verifiable proof of your narrative, though, it’s possible to hold onto a false narrative for quite some time, and this is likely to result in misallocated resources and hinder your long-term progress.
If you have opinions you’d like to validate, you can run experiments to see whether these opinions are verifiable facts and update your narrative accordingly to reflect your new knowledge.
Focus on what does and does not work
Experimentation benefits do not stop at simply validating or invalidating your hypotheses. After running an experiment, you’ll want to answer the following questions to understand your results:
- Did your change outperform the status quo according to your success metric?
- Is the benefit of the change (e.g., product engagement, revenue gain, or cost savings) greater than the cost of the change (e.g., resources to build the product or incentive given)?
The results of your experiment generally fall into one of the “next steps” categories below:
An experiment is a win-win in all cases as every outcome has an associated action you should take!
Further, it’s sometimes more important to identify whether an idea does not work, as this will force you to dig into the data to understand why or for whom it doesn’t work. Better understanding what’s not working will help you divert your resources away from projects that won’t drive business value and invest in those that might.
Data is the gift that keeps on giving
The value of experimentation does not end here. In addition to understanding how your idea might move your primary metric, there are a variety of additional metrics and segments you can dig into that will tell you far more about your impact.
It’s possible that your idea looks great from the perspective of the primary metric, but one or more secondary metrics moves in an undesired direction. For example, you might see an increase in engagement, but a decrease in overall revenue. Though the engagement outcome is positive, you’ll likely want to understand and solve the revenue issue prior to launching.
Regarding segments, a deep dive might help you understand which segments benefit from your idea the most and which segments are negatively impacted by your idea. It’s possible that your net impact is a mix of good and bad, so truly understanding where the impact is — and where it is not — can help you better target your customers and potentially elicit more hypotheses to test.
Getting buy-in for experimentation
Hopefully you now understand why you should run experiments. However, getting buy-in from those in your organization is a hurdle many face. The bottom line is that if you don’t run an experiment, you may not be able to determine whether the impact of an initiative was net positive or negative or how to iterate on or improve your initiative. Unfortunately, this may not satisfy everyone.
Tips for getting buy-in
Below are the most common concerns folks have regarding experimentation and how you may be able to address them.
| “Experiments add too much friction” or “We don’t have time to experiment” |
| It is definitely true that experimenting can add some friction to an initiative. However, it is positive friction that forces you to really think through the impact your initiative will have and how you might measure it. You’ll also waste far more time launching a negative initiative than confirming it has a positive impact in the first place! Further, you can create templates, leverage tools, and build processes that make experimentation more accessible for your organization. |
| “We know this is going to be successful, so we need to get it in front of everyone as soon as possible’” or “How can this be worse than what we have?” | |
| Most folks who have run experiments have experienced surprising results at some point. Sometimes a change can make things worse, benefit only a subset of users, or have mixed results. Experimenting can help you (1) determine whether the change has the desired effect and (2) understand why you’re seeing an impact. Human intuition is not right 100% of the time, and customers don’t always respond as you anticipate. |
| “Most experiments fail” or “I have to show a win” | |
| It’s very exciting to run an initiative that produces a desired result. However, you will likely try many initiatives that don’t perform as well as you’d like or perform worse than you anticipated. You learn from all experiments, whether they have a statistically significant result or not. Said another way, every experiment helps you learn so you can improve your chances of success in the future. |
| “We don’t have a large enough sample size to measure results” | |
| This is a common response among smaller start-ups that have few customers or prospective customers. Though sample size is one of the key considerations for experimentation, there are a few strategies you can explore that might help make experimentation possible. For example, you can move the primary metric of your experiment closer to the change you’re making (e.g., beginning a sale vs completing a sale), or change the duration of your experiment to run for a longer time period. |
| “Our customers tell us they want it” | |
| This can be a tough hurdle to overcome, especially if you’re launching a commonly requested feature. However, customers don’t always know the spectrum of possibilities, and oftentimes what customers say is different from what they do. Further, if customers like a particular feature, it doesn’t mean it will change their engagement or behavior with the product. It’s also possible that you’re hearing from a vocal subset of customers that don’t represent your broader customer base. Experimenting will enable you to determine what the impact is and for whom, and this can help you confidently prioritize future product development. |
Next steps
Ready to build your experimentation practice? Learn more here regarding how to do so.
Happy experimenting!
