A/B testing is the backbone of data-driven decision-making at most technology companies. The core idea is simple: when you want to know whether a change improves an outcome, you run a controlled experiment where some users get the change and others don't, then measure whether the difference in outcomes is real or just noise. Let's walk through every piece of that process carefully.

The Purpose of A/B Testing

The fundamental problem A/B testing solves is causal inference. Without an experiment, we can observe correlations but can't cleanly say "our change caused this outcome." Maybe users who see the new checkout button are also more likely to be mobile users, or were already more purchase-intent. An experiment breaks that confounding by randomly assigning who gets the treatment, so any systematic difference in outcomes can be attributed to the change itself.

The business motivation is straightforward: we want to ship changes that help users and avoid ones that hurt them. A/B tests give us a principled way to tell the difference before committing a change to everyone.

Step 1: Hypothesis Formulation

Before touching any data, we need a crisp hypothesis. A good hypothesis has three parts:

The change: what we're doing differently (e.g., making the "Add to Cart" button green instead of grey)
The expected effect: what we think will happen and why (e.g., more contrast makes the button more visible, increasing clicks)
The direction: are we expecting an increase, a decrease, or just any change?

This step forces intellectual honesty. If we don't commit to what we're testing before running the experiment, it becomes easy to p-hack by retroactively choosing metrics that happened to move. In formal terms, we write down a null hypothesis $H_0$ (the change has no effect) and an alternative $H_a$ (the change has an effect, usually in a specific direction). For example:

$$H_0: \mu_{\text{variant}} = \mu_{\text{control}}$$

$$H_a: \mu_{\text{variant}} > \mu_{\text{control}}$$

Step 2: Random Assignment to Control and Variant

This is the step that makes everything else work. We split users into groups, typically with a 50/50 split for maximum statistical power, though other splits (e.g., 90/10) are used when risk is a concern.

The assignment should be:

Random: each user has an equal probability of landing in either group
Stable: the same user should get the same experience on every visit (usually achieved by hashing user ID)
Mutually exclusive: a user is in exactly one group, not both

A common implementation is to hash a combination of the user ID and experiment ID, then use the result modulo 100 to assign a bucket. Something like:

import hashlib

def assign_variant(user_id: str, experiment_id: str) -> str:
    key = f"{experiment_id}:{user_id}"
    hash_val = int(hashlib.md5(key.encode()).hexdigest(), 16)
    bucket = hash_val % 100
    return "variant" if bucket < 50 else "control"

One critical sanity check here is the Sample Ratio Mismatch (SRM). If you expect a 50/50 split and observe 48/52, something has gone wrong in the assignment or logging pipeline. A chi-squared test can detect this. Never analyze results from a broken randomization.

Step 3: Metric Selection

Choosing the right metric is often harder than people expect. We want metrics that are:

Sensitive: they move when something real happens
Specific: they reflect the thing we actually care about, not a noisy proxy
Timely: they're observable within the experiment window

In practice, we usually track a hierarchy of metrics. There's the primary metric (the one we're optimizing, say, checkout conversion rate), a few secondary metrics (that give us a fuller picture, like revenue per user and session length), and guardrail metrics (things we must not break, like page load time or user complaints).

The reason to choose conversion rate over raw conversions, for example, is that conversion rate normalizes for the number of users, making the groups comparable even if they end up slightly different in size. For revenue, using revenue per user rather than total revenue has the same logic.

Step 4: Running the Experiment

Once live, we let the experiment run until we've collected enough data. The sample size needed is determined before the experiment starts using a power calculation. The key inputs are:

$\alpha$: the significance level, typically 0.05 (our tolerance for false positives)
$\beta$: the type II error rate, often 0.2, giving us 80% power (our tolerance for false negatives)
$\delta$: the minimum detectable effect (MDE), the smallest lift we'd care about
$\sigma^2$: the variance of the metric

The standard formula for the required sample size per group in a two-sample test is:

$$n = \frac{2 \sigma^2 (z_{1-\alpha/2} + z_{1-\beta})^2}{\delta^2}$$

where $z_{1-\alpha/2} \approx 1.96$ and $z_{1-\beta} \approx 0.84$ for the typical settings. Notice that the required sample size grows with variance and shrinks with a larger MDE. This creates a real business tradeoff: if you want to detect a 0.1% lift, you'll need a very long experiment.

We should not peek at results and stop early when significance is reached. That practice inflates the false positive rate dramatically. If early stopping is needed, use proper sequential testing methods like the sequential probability ratio test (SPRT).

Step 5: Analysis and Statistical Significance

Once the experiment has run its course, we compute the test statistic. For a proportion metric like conversion rate, we use a two-proportion z-test. Let $\hat{p}_c$ and $\hat{p}_v$ be the conversion rates in control and variant, and $n_c$, $n_v$ the group sizes. The test statistic is:

$$z = \frac{\hat{p}_v - \hat{p}_c}{\sqrt{\hat{p}(1 - \hat{p})\left(\frac{1}{n_c} + \frac{1}{n_v}\right)}}$$

where $\hat{p}$ is the pooled proportion across both groups. We then compute a p-value: the probability of observing a difference this large (or larger) if the null hypothesis were true. If $p < \alpha$, we reject $H_0$.

But statistical significance alone isn't enough to ship. We also ask:

Is the effect size practically significant? A 0.01% lift might be statistically real but not worth the engineering complexity.
Did our guardrail metrics hold? If conversion went up but page load time spiked, that's not a win.
Are there heterogeneous effects across user segments? Maybe the change helps desktop users but hurts mobile users.

Common Pitfalls to Watch For

A few failure modes come up repeatedly in practice:

Novelty effects: users engage more with anything new, so short experiments can show inflated lifts that decay over time. Running experiments long enough to see past the novelty period helps.
Network effects: if users interact with each other (social networks, marketplaces), a control user's behavior can be affected by variant users. Standard A/B testing assumptions break down here, and you may need cluster-level randomization.
Multiple testing: if you test 20 metrics at $\alpha = 0.05$, you'd expect one false positive just by chance. Use corrections like Bonferroni or Benjamini-Hochberg when looking at many metrics simultaneously.
Sample Ratio Mismatch (SRM): always verify the split before interpreting results. A logging bug or selection effect in assignment can silently corrupt the experiment.

Making the Decision

Statistical results inform the decision but don't make it automatically. The final call weighs the magnitude of the effect, confidence intervals around the estimate, potential long-term effects, implementation cost, and strategic alignment. A good experiment culture treats a well-run null result as just as valuable as a positive one: knowing what doesn't work is real information.

The overall framework, from hypothesis to random assignment to metric selection to rigorous analysis, exists to give us the best possible answer to a causal question while being honest about uncertainty. That honesty is what makes A/B testing genuinely useful rather than just a ritual.

1879. A/B Testing Basics

Similar Questions