MAC MAC A/B Testing & Experimentation 1 — Questions and Answers
Question 1: In a marketing A/B test, what does the 'control' represent?
- The new variation being tested
- The original version against which variations are compared (Correct answer)
- The statistical significance threshold
- The holdout group excluded from all variants
Correct answer: The original version against which variations are compared
The control is the original, unchanged version of a page or element that serves as the baseline for comparing the performance of new variations.
Question 2: What does statistical significance indicate in the context of an A/B test?
- The test has reached the required sample size
- The observed difference between variants is unlikely to be due to chance (Correct answer)
- The winning variant has been deployed to all users
- The test has run for a minimum of 30 days
Correct answer: The observed difference between variants is unlikely to be due to chance
Statistical significance indicates there is a sufficiently low probability that the measured difference between variants occurred by random chance.
Question 3: Why is it important to define a primary metric before launching an A/B test?
- To limit the number of variants that can be tested
- To prevent 'p-hacking' by changing the success metric after seeing results (Correct answer)
- To ensure the test runs for exactly two weeks
- To automatically allocate traffic between variants
Correct answer: To prevent 'p-hacking' by changing the success metric after seeing results
Pre-defining the primary metric prevents cherry-picking results after the test, which inflates the false positive rate and leads to unreliable conclusions.
Question 4: What is the minimum recommended statistical confidence level commonly used for marketing A/B tests?
- 75%
- 80%
- 95% (Correct answer)
- 99.9%
Correct answer: 95%
A 95% confidence level (p-value ≤ 0.05) is the widely accepted standard for declaring a marketing test result statistically significant.
Question 5: What is 'sample size' in the context of A/B testing, and why does it matter?
- The number of variants being tested; more variants means faster results
- The number of users in each group; too few users leads to unreliable results (Correct answer)
- The length of time a test runs; longer tests are always more accurate
- The percentage of traffic excluded from the test
Correct answer: The number of users in each group; too few users leads to unreliable results
Sample size refers to the number of users in each test group; insufficient sample sizes produce unreliable results because small differences can appear significant by chance.
Question 6: What is 'novelty effect' and how can it distort A/B test results?
- Users clicking on a new variant simply because it is different, inflating its early performance (Correct answer)
- A bug that causes the new variant to load faster than the control
- Statistical bias introduced by running tests during holiday periods
- The tendency for returning users to prefer the control version
Correct answer: Users clicking on a new variant simply because it is different, inflating its early performance
Novelty effect occurs when users engage more with a new variant simply because it is unfamiliar, causing temporary inflated performance that fades over time.
MAC MAC A/B Testing & Experimentation 1
In a marketing A/B test, what does the 'control' represent?