As product managers, designers and marketers, we love the clarity that comes from A/B testing. We have an idea or design of what we want to test, a theory on why and we let the users behavior determine the outcome from a controlled experiment.

Be aware of A/B testing has limitations and variabilities possibly outside of your control.

  1. Assumes you have tested and have the right JTBD or Workflow.
  2. Limit of two options will save time and reduce cost but minimizes insights.
  3. Temporal effects can be seen from running the test to long or short.
  4. Benefits may be short lived or cyclical.
  5. Mode effect from different devices or screen sizes may skew results.
  6. Interaction effects can amplify results.
  7. The more tests you run, the higher the chance of error.
  8. It may not account for all intermittent or unknown variables.
  9. It’s not multivariate, mixed method or qualitative research.