Page 3 of 43
As product managers, designers and marketers, we love the clarity that comes from A/B testing. We have an idea or design of what we want to test, a theory on why and we let the users behavior determine the outcome from a controlled experiment.
Be aware of A/B testing has limitations and variabilities possibly outside of your control.
- Assumes you have tested and have the right JTBD or Workflow.
- Limit of two options will save time and reduce cost but minimizes insights.
- Temporal effects can be seen from running the test to long or short.
- Benefits may be short lived or cyclical.
- Mode effect from different devices or screen sizes may skew results.
- Interaction effects can amplify results.
- The more tests you run, the higher the chance of error.
- It may not account for all intermittent or unknown variables.
- It’s not multivariate, mixed method or qualitative research.
In today’s Product Management and Product Marketing world data is an increasingly important element. Understanding it, using it, processing it, protecting it or even selling it are all potentially important. Data can be used to make decisions about your product’s strategy, design and marketing as well as within your product to provide more insights for your customers and end users.