Data Analytics: Foundations to Practice · A/B Testing and Experimentation
Common A/B Testing Pitfalls
Beyond the design fundamentals covered in the previous chapter, several specific, recurring pitfalls trip up even well-designed tests during execution and interpretation. This chapter names them.
A novelty effect occurs when a change performs unusually well or unusually poorly simply because it's new and different, an effect that tends to fade as users become accustomed to the change over time; a test run for too short a period can mistake this genuinely temporary novelty effect for a real, durable, lasting effect that will actually persist once the initial novelty has fully worn off.
Key Takeaways
- A novelty effect can make a change appear to perform well or poorly simply because it's new, fading as users become accustomed to it over time.
- Running a test during an unrepresentative period (holidays, external events) can produce a result that doesn't generalize to typical, ordinary conditions.
- A sample ratio mismatch, where the actual group split differs from the intended one, is a serious warning sign of a technical problem that should be resolved first.
- Running multiple simultaneous tests on overlapping populations can produce interaction effects, distorting results compared to running each test in isolation.