The A/B test is the scientific method of advertising: you compare two versions of one element to find out which performs better. It is a staple of digital marketing courses and is often applied badly. This guide gives a rigorous approach with worked numbers.
The principle
You expose two comparable groups to two different variants, changing only one element at a time, then compare a metric chosen in advance. If the only difference between A and B is the visual, the gap in results can reasonably be attributed to it.
What you can test:
- the creative: image, video, hook, opening shot;
- the text: headline, argument, tone;
- the call-to-action button;
- the audience;
- the placement or format;
- the landing page.
Step 1: write a hypothesis
A serious test starts with a sentence of the form: “If I change X, then Y should change, because Z.”
Example. “If I replace the product image with a 10-second video, CTR should rise, because motion draws more attention in the feed.”
Without a hypothesis you do not know what you are looking for, and you risk drawing conclusions from chance.
Step 2: pick a single variable
Testing two ads that differ in image, text and audience at once allows no conclusion: there is no way to know what produced the gap. One variable at a time. To test several elements, run tests in sequence: image first, then text, then button.
Step 3: choose the success metric
Choose the metric before launching, consistent with the objective:
| What you test | Suitable metric |
|---|---|
| Creative (attention) | CTR, view rate |
| Audience | Cost per result |
| Landing page | Conversion rate |
| Offer | CPA, ROAS |
Choosing the metric afterward, from those that favor the variant you prefer, is a classic bias. See Digital advertising formulas.
Step 4: split budget and audience fairly
Both variants must receive a comparable budget and be shown to comparable audiences, over the same period. Meta and TikTok provide testing tools that separate audiences so the same person does not see both versions. See Test with an A/B test and TikTok split tests.
Step 5: let the test run
Stopping a test as soon as one variant pulls ahead is the most common mistake. The first days are unstable, notably because of the learning phase (Optimize without breaking learning). Reference points:
- run for several days, ideally at least one full week to smooth out weekday and weekend variation;
- aim for a sufficient number of results per variant;
- do not edit the variants mid-test.
Step 6: read the result carefully
An observed gap is not necessarily a real gap: chance produces differences.
Worked example. Two visuals are each shown 10,000 times.
| Impressions | Clicks | CTR | |
|---|---|---|---|
| Variant A | 10,000 | 100 | 1.0% |
| Variant B | 10,000 | 130 | 1.3% |
B looks 30% better. A standard statistical test comparing two proportions gives roughly a 5% probability of seeing such a gap by pure chance here: you are at the edge of what is usually considered significant. A reasonable conclusion: B is probably better, but extending the test before rolling it out would be prudent.
If the same percentages had come from 1,000 impressions and 10 versus 13 clicks, the conclusion would be nil: three clicks of difference is chance. The smaller the sample, the less reliable the gap.
Platforms often show a confidence level or a probability of winning; use them as an aid, not as certainty.
Step 7: document and continue
For each test, record the hypothesis, variable, duration, results and decision. These notes prevent repeating the same mistakes. A winning test becomes the new baseline, which the next test tries to beat.
The eight classic mistakes
- Testing several variables at once.
- Stopping too early.
- Comparing different periods or budgets.
- Choosing the metric after seeing the results.
- Concluding on too little data.
- Editing a variant during the test.
- Generalizing a result to another product, audience or season without re-checking it.
- Forgetting that the most-clicked version is not always the most profitable: track conversion too.
Frequently asked questions
How long should an ad A/B test run? Several days, ideally a full week, and until each variant has enough results.
Can I test more than two variants? Yes (A/B/C test), but each extra variant dilutes the budget and requires more data.
What is the difference between an A/B test and automatic optimization? An A/B test is a protocol you control; automatic optimization shifts delivery continuously according to performance, without a comparable protocol.
Do you teach online advertising? Request access.