Experimentation is now the gold standard within data-driven tech companies. We know why we need to experiment, but the next question is how to enable an organization to experiment with as little friction as possible. This will require a thoughtful mixture of people, tools, and process, but once in place, watch your experimentation culture blossom!
Enabling an experimentation practice
Before diving into experimentation, you need to put a few critical pieces in place.
Experimentation owners
The first critical piece is to appoint your fearless experimentation leader, who will own your experimentation practice. This person will be responsible for setting up, maintaining, and improving your practice, including making sure folks adhere to the practice’s standards. This person or group is generally skilled in experimentation and will relentlessly advocate for experimentation within the business.
In addition to your experimentation leader, you may also need advocates, e.g.:
- Engineering advocate: Experiments typically involve some engineering and require involvement from someone who can set up and manage a platform and process
- Leadership advocate: Help your experimentation leader amplify the purpose and value of experimentation to ensure the whole organization is bought in
Platform and tools
The next piece is identifying an experimentation platform on which you can run your experiments. There are a variety of experimentation platforms available that enable different kinds of experimentation:
| Experiment type | Generally used by | Used for |
|---|---|---|
| Multivariate testing and personalization | Marketers | Website marketing optimization that enables you to optimize variants across different audiences |
| Website optimization | Marketers | Website marketing optimization via A/B tests with a friendly user interface (i.e., drag and drop) |
| Email and mobile messaging campaigns | Lifecycle marketers | A/B tests of messaging campaigns |
| Feature flags and phased rollouts | Engineers | Launching and monitoring phased product rollouts |
| Product and feature launches | Product teams | Analyzing the impact of new features or products |
It’s important to delineate what your organization needs regarding features and functionality in order to find the platform that most closely aligns with your needs at an appropriate cost.
In addition to a platform, you will also likely need experimentation resources, such as those that enable you to estimate sample size and measure results. The experimentation platform you select may have these resources, and there are also a variety of resources online that you can use.
Design and results template
The last piece to set up is your design and results template to enable your team to easily document experiments. It can be tempting to brainstorm ideas quickly and launch experiments without going through a formal documentation process. However, there are multiple reasons that you’ll want to clearly document your experiment design and results:
- Detail exactly what you did so others can replicate it
- Share your design with others to get feedback and ensure you’re set up for success
- Document and share your results so others can easily find your learnings
Generally, the core components of your experiment design and results document include:
| High-level Info | Point of contact Team Goal Timeframe Context |
| Design | Hypothesis Independent variable Target users Experiment groups Experiment split Sample size Primary metrics Secondary metrics Hold steady metrics |
| Execution & Shortcomings | Execution details Technical issues Execution issues |
| Analysis & Results | Metric results Additional findings |
| Interpretation & Next Steps | Interpretation of results Opportunities Next steps |
In an effort to ensure your team is focused on the experiment idea versus the technical setup, and because this design and analysis template can be rather technical, it can be helpful to have an intake form in which your stakeholders can describe their goals. Your experiment experts can then help them transform their idea into a technical plan.
If you’re wondering what an intake form or experiment design template might look like, contact us at hi [at] datacrt.com!
Running an experiment practice
Once you have the right people, tools, and documentation in place, it’s time to run a standard experiment process to maximize the success of your experimentation practice.
Experiment review
After you’ve designed your experiment, a great way to share your experiment with stakeholders is to hold a live review. This review will enable the experimenter to review the experiment design and details with others and receive feedback to improve the experiment.
We recommend requiring folks from different teams in each review so they are aware of what’s going on and can surface potential issues and prepare for launch. Though the appropriate teams to include can vary by organization, or even by experiment, you’ll generally want:
| …a representative from… | …to help you… |
|---|---|
| Finance | manage financial and business implications. |
| Operations / sales | identify and plan for the impact on business operations. |
| Customer support | respond to customer inquiries or questions related to the experiment groups. |
| Product | determine what might be considered for the product roadmap. |
Adding additional folks as optional will enable them to review experiments in advance to determine whether or not they are needed. Optional folks who might be beneficial for your reviews include:
- Other teams that are running experiments on the same users so that they can identify potential interferences
- A Community Support representative who trains agents to handle questions or concerns about changes to the customer experience
- Folks who can poke on the metrics or design to ensure the team is set up to learn
- Folks who can surface similar experiments that have been run or data that can help answer what the team is trying to learn in an effort to not waste resources
We also recommend holding a review after the experiment concludes and the analysis has been completed to surface results, learnings, and next steps.
The frequency and consistency of these reviews really depends on how much experimentation is happening. If you are running one experiment per month, you may only need to hold time for folks on an as-needed basis. If you are running experiments regularly, a consistent meeting might be best. If there are no experiments ready for review, you can always repurpose this time to brainstorm and improve future experiment ideas.
Last, another best practice is to have a company-wide messaging channel dedicated to experimentation where all folks in the organization can find and track information about experiments. You can even include a link for folks to invite themselves to the reviews.
Knowledge repository
Now that you’re documenting your experiment design and analysis and ensuring your learnings are shared with others via a review, you need a centralized repository in which you can store all the experiments you’ve run so folks can access the knowledge you’ve gained at any time. This repository is generally a single location, such as a shared folder or Confluence page, that houses all design documents, which include all information about your experiments as well as links to code and dashboards that may be helpful for others to review or use.
As organizations grow and more folks begin to run experiments across different teams, it can be easy to lose track of all of the experiments and learnings over time. Further, without a forcing function to share experiment designs and knowledge, many experiments that don’t result in a clear winner might never be shared! A repository can help you avoid these situations and minimize wasted time and resources as well as encourage those in your organization to review past experiments and learnings.
Experimentation pitfalls
In theory, this all sounds great. In reality, there are some limitations.
Solid experiment design is key
Being thoughtful and diligent about experiment design is a must as the ability to learn from experiments hinges entirely on your ability to design experiments appropriately. Fortunately there are tools and resources to help with this, though designating an experimentation owner with experimentation expertise on your team is generally the best practice. Reach out to us at hi [at] datacrt.com to learn about our favorite online resources!
The common design issues we see include primary metric selection, clarity around the sample size needed, and random assignment. Below are a few generic examples we’ve seen over the years:
| Design issue | Generic (bad) example |
|---|---|
| Primary metric selection | Selecting a metric that you may want to move, but for which it is difficult or impossible to detect an effect |
| Insufficient sample size | Claiming a significant finding when altering a few data points would result in a different outcome |
| Random assignment | Measuring the impact of those who opt in to an initiative versus those who don’t |
Having a clear process around each of these and reviewing each experiment design prior to launch can go a long way!
Experiments need to run their full course
Once you have an experiment process in place and the team is fully on board the experiment train, you must remain diligent! People can get excited about testing their ideas and showing the impact their teams are having, and it’s easy to get carried away with early results that look great when the reality is it’s too early to draw any conclusions. Therefore, it’s important to have patience and run experiments for a predetermined length of time to get a true read of the results.
Below are a few experimental effects of which you want to be cognizant:
| Effect | Description |
|---|---|
| Novelty effect | Your data may look great at first, but it eventually reverts to its original trends. If you’re looking for short-term engagement, this may be a win. However, this may not be a long-term win, and you will need to evaluate why and tweak your idea prior to rolling out. |
| Pull-forward effect | The idea increases the velocity of a desired action, but it does not necessarily increase the overall adoption or conversion. Depending on your goal, this could be a win, but it could also require you to tweak your idea. |
In both of these cases, running the experiment for the appropriate length of time you determined as part of your experiment design will ensure you have enough information to make the best decision.
Experimentation is not always possible
There are many cases in which A/B experimentation is difficult or impossible. In these cases, your first step should be to identify the reason you believe you cannot experiment and explore alternative solutions:
| Bucket | Example | Alternatives |
|---|---|---|
| Small sample | Not having enough users to generate meaningful results | Explore alternative success metrics, or run your experiments for a longer period of time |
| Infeasible | Showing a billboard or airing a radio ad to only half the city | Try an alternative experimentation technique, such as a synthetic control |
| Internally complex | Difficulty rallying the troops internally to execute an experiment, such as requiring folks to follow a playbook | Get buy in from those who are responsible for executing the experiment |
| Network effects | The treatment impacts more than one experiment group | Try an alternative experimentation technique, such as a switchback |
If you’ve exhausted your options and don’t see a path to experimentation, this is where you might need to get comfortable with ambiguity and rely on your beliefs, past data, and experiences to make decisions. Just because you can’t experiment doesn’t mean your idea is not good!
After all, if you only move forward with ideas that you can test, you may be missing out on opportunities. However, if you can test an idea to understand the impact, you’ll have more information with which you can make better decisions in the future.
Next steps
Want to chat more about experimentation? Hit us up at hi [at] datacrt.com!
