What it is
A/B testing is a controlled experiment on your own product. You split visitors randomly into two groups: the control (A) sees the current version, the variant (B) sees a change - a different headline, button, layout, or price. You measure a chosen metric for each group, usually a conversion rate, and keep the version that wins. It is the scientific method applied to product decisions.
It closes the loop of the marketing analytics story: you get traffic, you measure what it does, and A/B testing is how you act on that - changing things deliberately and letting real behavior, not opinion, decide what ships.
How a test runs
The mechanics are straightforward and always the same shape:
- Form a hypothesis. "Changing the CTA from Buy now to Get started will raise sign-ups." A test without a hypothesis is just fishing.
- Split traffic randomly. Each visitor is assigned to A or B and stays there (a persistent bucket), so one person always sees one version.
- Measure one primary metric. Track the event that matters - the sign-up, the purchase - for each group.
- Compare, and decide with statistics. The raw rates are never enough; the question is whether the difference is real.
That last step is the entire game. A variant will almost always show some difference. Whether that difference means anything is a statistical question, not a visual one.
Significance and sample size
Two concepts separate a real result from a coin flip. Statistical significance asks: if there were truly no difference between A and B, how likely is a gap this big by pure chance? The convention is to require that likelihood (the p-value) to fall below 5% before believing the result - though significance alone does not tell you the effect is large, only that it is probably not zero.
Sample size is the other half. Small differences need large numbers of visitors to detect reliably; a test on a low-traffic page can run for months and never reach a conclusion. You calculate the needed sample size before starting - from your baseline rate, the smallest improvement worth caring about, and the confidence you want - and you commit to running until you hit it. Which sets up the single most common mistake.
The ways it lies
A/B testing is easy to run and easy to fool yourself with. The recurring traps:
- Peeking. Checking results daily and stopping the moment B looks significant. Because random noise crosses the threshold constantly, early peeking manufactures winners that are not real. Decide the sample size and duration up front, and wait.
- Testing too many things. Run twenty tests and, by chance alone, one clears a 5% threshold with no real effect. The more variants and metrics you check, the more false positives you invite.
- Ignoring business cycles. A test that runs Tuesday to Thursday misses weekend behavior. Run for full weeks to average out the rhythm of real traffic.
- Winning the metric, losing the goal. A scarier headline may lift clicks and hurt trust. Watch guardrail metrics, not just the one you are optimizing.
The through-line: the statistics exist to stop you believing noise. Skip them and an A/B test is just a confident way to ship random changes.
Client-side vs server-side
Two ways to implement the split, with a real tradeoff:
- Client-side. JavaScript swaps the variant in the browser after load. Easy for marketers to run on content and layout, but it risks a flicker - the original flashes before the variant replaces it - and adds a script to the page.
- Server-side. The server decides the variant and sends the right version directly. No flicker, no extra client weight, and it can test deeper changes (pricing, logic, backend behavior) - but it needs engineering to build.
The rule of thumb: content and copy tests fit client-side tools; anything touching performance, pricing, or core logic belongs server-side. The flicker point is not cosmetic - a client-side test that hurts your Core Web Vitals can cost you more in ranking than the test wins in conversions.
When to reach for it
A/B testing rewards traffic and punishes impatience. It shines when you have enough visitors to reach significance in a reasonable time and a clear metric to move - checkout flows, sign-up pages, pricing, high-traffic landing pages. It is the wrong tool for a low-traffic site (you will never gather the sample), for huge redesigns (too many variables changing at once to learn anything), or for decisions where the answer is obvious and the test is just procrastination.
Used well, it is the most honest instrument in marketing analytics: it does not tell you what people say they prefer, it shows you what they actually do. Used badly - peeked at, under-powered, over-run - it is a machine for confirming whatever you already wanted to believe. The statistics are the difference, which is why this is a 201 and not a 101.