Reference
Every term you meet when planning, running or reading a test, defined in two or three sentences, with a link to the guide or tool that goes deeper. 43 terms, alphabetical.
5 terms
Showing two versions of a page or element to different visitors at the same time and measuring which one achieves a goal more often. It is the basic unit of experimentation; every other term on this page qualifies it.
Full explanation: What is A/B testing?
Go deeper: How to A/B test in WordPress
An A/B test with more than two versions, all compared against the same control. Each extra variant needs its own full sample, so the traffic a test needs grows with the number of arms.
Full explanation: What is A/B/n testing?
Go deeper: How many visitors a test needs
A small piece of code, loaded before the page renders, that hides content until the testing script has applied the variant. Cloud testing tools add one to every page, and it is a common source of slower loads.
Go deeper: Caching and CDN compatibility
How a visitor is placed into a variant. Cookieless tools keep the assignment in the browser's local storage or derive it from a random identifier, so the same person keeps seeing the same version on later visits.
Go deeper: Cookieless testing and GDPR
Limiting who takes part in a test, for example by device, by new versus returning visitor, or by a URL parameter. A narrower audience gives a cleaner answer about that group and takes proportionally longer to fill its sample. In ImproveAB it is a Pro setting; page targeting (where a test runs) is free.
Go deeper: Targeting in ImproveAB
2 terms
The control's conversion rate before the test starts. It is one of the three inputs to every sample-size calculation, and the lower it is, the more visitors a test needs.
Go deeper: Sample-size calculator
The probability, under a Bayesian model, that a variant is better than the control. It is easier to read than a p-value, but it is a different number answering a different question, and the two are not interchangeable.
Go deeper: A/B testing statistics made simple
8 terms
The address a page declares as its authoritative version. In a split URL test the variant page points its canonical at the original, so search engines index one page and the test does not compete with itself.
Go deeper: SEO settings for URL tests
The range within which the true uplift plausibly lies, given the data so far. A wide interval means the test has not yet pinned the effect down, even when the point estimate looks like a win.
Full explanation: What is a confidence interval?
Go deeper: Significance calculator
One minus the false-positive rate you accept. At 95% confidence you accept that one in twenty tests of a change that does nothing will still look like a winner by chance.
Go deeper: What 95% confidence means
The current version, also called A or the baseline. Every variant is measured against it, and it is the version you keep if nothing beats it.
Full explanation: Control vs variant in an A/B test
A visitor completing the action the test measures: reaching a page, clicking an element or link, submitting a form, or buying. What counts as a conversion is fixed by the goal before the test starts.
Full explanation: What is a conversion in marketing?
Conversions divided by visitors, expressed as a percentage. It is the number a test compares between variants, and the quantity every calculator on this site works with.
Full explanation: Conversion rate: definition and formula
Go deeper: Conversion rate calculator
Running a test without setting cookies, typically by keeping the assignment in browser local storage. It avoids cookie mechanics but does not by itself place a test outside privacy law, which covers local storage too.
Go deeper: Cookieless testing and GDPR
The label a testing tool uses to find the element under test, such as a class added to a section in a page builder. Class-based tools work in any builder or theme that lets you add a class to an element.
Go deeper: How class-based testing works
1 term
A test that swaps one element on a page, such as a headline, a button or a whole section, while everything else stays the same. It gives a clean answer about that one change.
Go deeper: How it works
4 terms
Missing a real improvement. Statistical power controls how often this happens: at the usual 80% power, one real effect in five goes undetected.
Declaring a winner when the variants are really the same. The confidence level caps how often this happens, but only if you read the result once, at the planned sample.
Go deeper: Why a winner did not hold
The visible switch from the original content to the variant while a testing script runs, also called a flash of original content. Flicker-free tools keep the tested element hidden until the variant is applied, so the original never flashes, with a failsafe timeout.
Go deeper: Caching, CDNs and flicker
Two schools of statistics. Frequentist methods such as z-tests and p-values ask how rare the data would be if there were no effect; Bayesian methods estimate the probability of the effect directly. Well-run tests reach the same decisions either way.
Full explanation: Frequentist vs Bayesian A/B testing
Go deeper: Statistics made simple
1 term
The specific conversion a test counts, chosen before the test starts. One goal per test keeps the answer clean; a test with several goals usually ends with the one that happened to look best.
Go deeper: Goals in ImproveAB
1 term
A falsifiable statement of what you expect and why, in the form "changing X to Y will increase Z". Writing one turns a hunch into a test that teaches something whichever way the result goes.
Full explanation: A/B test hypothesis: definition and template
Go deeper: Your first A/B test
1 term
A visitor being shown a variant. Impressions are the denominator of the conversion rate, so whether a tool counts them per visitor or per page view changes what its rates mean.
1 term
What a test taught you, regardless of its outcome. A recorded hypothesis and result keep a team from re-testing the same idea a year later.
Go deeper: Picking what to test
2 terms
The smallest uplift a test is designed to catch, chosen before it starts. It sets the sample size more than anything else: halving the MDE roughly quadruples the visitors needed.
Full explanation: Minimum detectable effect (MDE)
Go deeper: How many visitors a test needs
Testing combinations of several elements at once to learn how they interact. Every combination is its own arm, so an MVT needs far more traffic than an A/B test of the same elements one at a time.
Full explanation: What is multivariate testing?
1 term
A temporary lift caused by something being new rather than better. It fades as returning visitors get used to the change, which is one reason tests run for whole weeks rather than days.
Go deeper: Six ways a winner vanishes
3 terms
The probability of seeing a gap at least as large as the observed one if the variants were actually identical. A p-value under 0.05 corresponds to 95% confidence.
Full explanation: What is a p-value?
Go deeper: Significance calculator
Whether a test still works when pages are served from a cache or a content delivery network instead of being generated fresh. Tools that assign variants in the browser survive caching; server-side assignment usually does not.
Go deeper: Caching and CDN compatibility
Checking a running test and stopping the moment it looks significant. Because confidence swings widely early on, peeking produces far more false winners than the 5% the confidence level promises.
Go deeper: How long to run a test
8 terms
When the observed split between variants differs from the planned one by more than chance allows, for example 58/42 instead of 50/50. It is a sign that the test is broken and that its results cannot be trusted.
Full explanation: What is sample ratio mismatch (SRM)?
Go deeper: SRM checker
The number of visitors per variant a test needs before its result can be trusted, calculated from the baseline rate, the minimum detectable effect, the confidence level and the power. Decided before the test starts, never after.
Go deeper: Sample-size calculator
Regular changes in visitor behaviour by day of the week, time of month, or marketing calendar. Tests run in whole weekly cycles so both arms see the same mix of visitors.
Go deeper: Planning test duration
Breaking results down by a visitor attribute, such as device, after the test. Useful for seeing where an effect comes from, and dangerous when used to hunt for a segment that happens to look significant.
Another name for A/B testing. In practice it often means a URL split test, where the versions are separate pages and visitors are redirected, rather than an element test inside one page.
Full explanation: Split testing vs A/B testing
A test between two separate pages at different addresses. Visitors are sent to one or the other and kept there on return visits. Used when variants differ in too many ways to swap single elements.
Go deeper: How to split test two landing pages
The chance a test detects a real effect of the size you specified as the minimum detectable effect. 80% power is the convention; lower power means more real winners go unnoticed.
Go deeper: Power and sample size
A result is significant when chance alone would rarely produce a gap as large as the one observed. At the usual 95% level, rarely means less than one time in twenty. Significance says a difference is probably real, not that it is large.
Full explanation: Statistical significance in A/B testing
Go deeper: Significance calculator
2 terms
The share of visitors sent to each variant. An even split reaches significance fastest; uneven weights limit how many people see a risky variant, at the cost of a longer test.
The standard frequentist test for comparing two conversion rates. It is the formula behind most A/B significance and sample-size calculators, including the ones on this site.
Full explanation: The two-proportion z-test
Go deeper: Sample-size calculator
1 term
The improvement of a variant over the control. A move from 3.0% to 3.6% is a 20% relative uplift and a 0.6 percentage-point absolute one; most tools and this site quote the relative figure.
Full explanation: Uplift (lift) in A/B testing
Go deeper: Conversion rate and uplift calculator
1 term
Any version other than the control: B in a two-way test, and B, C, D and so on in an A/B/n test. The variant is what you are proposing to ship if it wins.
1 term
The variant declared better at the end of a test. Shipping it means making it the permanent page; a winner that does not hold up in production usually points to a sizing or peeking problem.
Go deeper: Why your test said +40% and nothing changed
Baseline, minimum detectable effect, daily traffic: the calculator turns them into a visitor count and a finish date.