How to Set Up an Experimentation Practice

Experimentation is now the gold standard within data-driven tech companies. We know why we need to experiment, but the next question is how to enable an organization to experiment with as little friction as possible. This will require a thoughtful mixture of people, tools, and process, but once in place, watch your experimentation culture blossom!

Enabling an experimentation practice

Before diving into experimentation, you need to put a few critical pieces in place.

Experimentation owners

The first critical piece is to appoint your fearless experimentation leader, who will own your experimentation practice. This person will be responsible for setting up, maintaining, and improving your practice, including making sure folks adhere to the practice’s standards. This person or group is generally skilled in experimentation and will relentlessly advocate for experimentation within the business. 

In addition to your experimentation leader, you may also need advocates, e.g.:

  • Engineering advocate: Experiments typically involve some engineering and require involvement from someone who can set up and manage a platform and process
  • Leadership advocate: Help your experimentation leader amplify the purpose and value of experimentation to ensure the whole organization is bought in

Platform and tools

The next piece is identifying an experimentation platform on which you can run your experiments. There are a variety of experimentation platforms available that enable different kinds of experimentation:

Experiment typeGenerally used byUsed for
Multivariate testing and personalizationMarketersWebsite marketing optimization that enables you to optimize variants across different audiences
Website optimizationMarketers Website marketing optimization via A/B tests with a friendly user interface (i.e., drag and drop)
Email and mobile messaging campaignsLifecycle marketersA/B tests of messaging campaigns
Feature flags and phased rolloutsEngineersLaunching and monitoring phased product rollouts
Product and feature launchesProduct teamsAnalyzing the impact of new features or products

It’s important to delineate what your organization needs regarding features and functionality in order to find the platform that most closely aligns with your needs at an appropriate cost.

In addition to a platform, you will also likely need experimentation resources, such as those that enable you to estimate sample size and measure results. The experimentation platform you select may have these resources, and there are also a variety of resources online that you can use.

Design and results template

The last piece to set up is your design and results template to enable your team to easily document experiments. It can be tempting to brainstorm ideas quickly and launch experiments without going through a formal documentation process. However, there are multiple reasons that you’ll want to clearly document your experiment design and results:

  1. Detail exactly what you did so others can replicate it
  2. Share your design with others to get feedback and ensure you’re set up for success
  3. Document and share your results so others can easily find your learnings

Generally, the core components of your experiment design and results document include:

High-level InfoPoint of contact
Independent variable
Target users
Experiment groups
Experiment split
Sample size
Primary metrics
Secondary metrics
Hold steady metrics
Execution & ShortcomingsExecution details
Technical issues
Execution issues
Analysis & ResultsMetric results
Additional findings
Interpretation & Next StepsInterpretation of results
Next steps

In an effort to ensure your team is focused on the experiment idea versus the technical setup, and because this design and analysis template can be rather technical, it can be helpful to have an intake form in which your stakeholders can describe their goals. Your experiment experts can then help them transform their idea into a technical plan. 

If you’re wondering what an intake form or experiment design template might look like, contact us at hi [at]!

Running an experiment practice

Once you have the right people, tools, and documentation in place, it’s time to run a standard experiment process to maximize the success of your experimentation practice.

Experiment review

After you’ve designed your experiment, a great way to share your experiment with stakeholders is to hold a live review. This review will enable the experimenter to review the experiment design and details with others and receive feedback to improve the experiment. 

We recommend requiring folks from different teams in each review so they are aware of what’s going on and can surface potential issues and prepare for launch. Though the appropriate teams to include can vary by organization, or even by experiment, you’ll generally want:

…a representative from……to help you…
Financemanage financial and business implications.
Operations / salesidentify and plan for the impact on business operations.
Customer supportrespond to customer inquiries or questions related to the experiment groups.
Productdetermine what might be considered for the product roadmap.

Adding additional folks as optional will enable them to review experiments in advance to determine whether or not they are needed. Optional folks who might be beneficial for your reviews include:

  1. Other teams that are running experiments on the same users so that they can identify potential interferences
  2. A Community Support representative who trains agents to handle questions or concerns about changes to the customer experience
  3. Folks who can poke on the metrics or design to ensure the team is set up to learn
  4. Folks who can surface similar experiments that have been run or data that can help answer what the team is trying to learn in an effort to not waste resources

We also recommend holding a review after the experiment concludes and the analysis has been completed to surface results, learnings, and next steps. 

The frequency and consistency of these reviews really depends on how much experimentation is happening. If you are running one experiment per month, you may only need to hold time for folks on an as-needed basis. If you are running experiments regularly, a consistent meeting might be best. If there are no experiments ready for review, you can always repurpose this time to brainstorm and improve future experiment ideas.

Last, another best practice is to have a company-wide messaging channel dedicated to experimentation where all folks in the organization can find and track information about experiments. You can even include a link for folks to invite themselves to the reviews.

Knowledge repository

Now that you’re documenting your experiment design and analysis and ensuring your learnings are shared with others via a review, you need a centralized repository in which you can store all the experiments you’ve run so folks can access the knowledge you’ve gained at any time. This repository is generally a single location, such as a shared folder or Confluence page, that houses all design documents, which include all information about your experiments as well as links to code and dashboards that may be helpful for others to review or use.

As organizations grow and more folks begin to run experiments across different teams, it can be easy to lose track of all of the experiments and learnings over time. Further, without a forcing function to share experiment designs and knowledge, many experiments that don’t result in a clear winner might never be shared! A repository can help you avoid these situations and minimize wasted time and resources as well as encourage those in your organization to review past experiments and learnings.

Experimentation pitfalls

In theory, this all sounds great. In reality, there are some limitations.

Solid experiment design is key

Being thoughtful and diligent about experiment design is a must as the ability to learn from experiments hinges entirely on your ability to design experiments appropriately. Fortunately there are tools and resources to help with this, though designating an experimentation owner with experimentation expertise on your team is generally the best practice. Reach out to us at hi [at] to learn about our favorite online resources!

The common design issues we see include primary metric selection, clarity around the sample size needed, and random assignment. Below are a few generic examples we’ve seen over the years:

Design issueGeneric (bad) example
Primary metric selectionSelecting a metric that you may want to move, but for which it is difficult or impossible to detect an effect
Insufficient sample sizeClaiming a significant finding when altering a few data points would result in a different outcome
Random assignmentMeasuring the impact of those who opt in to an initiative versus those who don’t

Having a clear process around each of these and reviewing each experiment design prior to launch can go a long way!

Experiments need to run their full course

Once you have an experiment process in place and the team is fully on board the experiment train, you must remain diligent! People can get excited about testing their ideas and showing the impact their teams are having, and it’s easy to get carried away with early results that look great when the reality is it’s too early to draw any conclusions. Therefore, it’s important to have patience and run experiments for a predetermined length of time to get a true read of the results.

Below are a few experimental effects of which you want to be cognizant:

Novelty effectYour data may look great at first, but it eventually reverts to its original trends. If you’re looking for short-term engagement, this may be a win. However, this may not be a long-term win, and you will need to evaluate why and tweak your idea prior to rolling out.
Pull-forward effectThe idea increases the velocity of a desired action, but it does not necessarily increase the overall adoption or conversion. Depending on your goal, this could be a win, but it could also require you to tweak your idea.

In both of these cases, running the experiment for the appropriate length of time you determined as part of your experiment design will ensure you have enough information to make the best decision.

Experimentation is not always possible

There are many cases in which A/B experimentation is difficult or impossible. In these cases, your first step should be to identify the reason you believe you cannot experiment and explore alternative solutions:

Small sampleNot having enough users to generate meaningful resultsExplore alternative success metrics, or run your experiments for a longer period of time
InfeasibleShowing a billboard or airing a radio ad to only half the cityTry an alternative experimentation technique, such as a synthetic control
Internally complexDifficulty rallying the troops internally to execute an experiment, such as requiring folks to follow a playbookGet buy in from those who are responsible for executing the experiment
Network effectsThe treatment impacts more than one experiment groupTry an alternative experimentation technique, such as a switchback

If you’ve exhausted your options and don’t see a path to experimentation, this is where you might need to get comfortable with ambiguity and rely on your beliefs, past data, and experiences to make decisions. Just because you can’t experiment doesn’t mean your idea is not good! 

After all, if you only move forward with ideas that you can test, you may be missing out on opportunities. However, if you can test an idea to understand the impact, you’ll have more information with which you can make better decisions in the future.

Next steps

Want to chat more about experimentation? Hit us up at hi [at]!