Statistics

You do not need a statistics background to use Shuttlebase. This page starts with a short version, then explains each part of the methodology for readers interested in digging deeper.

The short version

Shuttlebase splits your visitors evenly between variants and keeps each visitor in the same variant for the entire experiment lifetime. It then uses standard statistical tests to check whether the difference between variants is real or just noise. Results appear in the dashboard as a significance level of high, medium, or low, together with a recommendation: deploy the winner, end the experiment, disable a variant, keep running, or review manually.

When Shuttlebase projects revenue impact, it uses a conservative estimate rather than the most optimistic number. The projections you see are ones the data actually supports.

The rest of this page explains how each piece works.

Traffic splitting

By default, traffic splits evenly between all variants, including the baseline. An even split reaches a decision fastest, because the test is limited by its smallest group.

If a variant is disabled or rejected during review, its traffic is redistributed. Depending on the situation, the freed traffic is spread evenly across the remaining variants, split in proportion to their existing shares, or returned to the baseline.

Sticky assignment

Each visitor is assigned to a variant on their first eligible visit and stays in that variant for the duration of the experiment. Assignment persists in the visitor’s browser, so returning visitors see the same version of your site every time. Details of how assignment is stored are at Cookies and storage.

Sticky assignment keeps the measurement clean. If visitors switched variants between sessions, their purchases could not be attributed to a single experience, and the comparison would break down.

The statistical tests

Shuttlebase uses frequentist statistics by default. It runs fixed comparisons between variants using two standard tests, chosen by metric type.

Conversion metrics: two-proportion z-test

Conversion-rate metrics compare two proportions, such as the share of visitors who added to cart in each variant. The two-proportion z-test checks whether the gap between those proportions is larger than chance would produce given the sample sizes. This is the standard test for exactly this comparison.

Revenue metrics: Welch’s t-test

Revenue is a continuous value that varies a lot from order to order, so it needs a different test. Shuttlebase uses Welch’s t-test, which compares the average value per visitor between variants. Unlike the plain t-test, Welch’s version does not assume both groups have the same variance. That assumption often fails for revenue data, where a few large orders can stretch one group’s spread.

Sample sizes: power analysis

Before and during an experiment, Shuttlebase uses standard power analysis to work out how many visitors are needed to detect an effect of a given size. This is why the dashboard does not call small differences early. On a modest sample, a small real effect and random noise look the same, and only more traffic can tell them apart.

Sample ratio mismatch detection

Shuttlebase continuously checks that the actual traffic split matches the intended one, using a chi-square test. This check is called sample ratio mismatch (SRM) detection.

If an experiment is set to a 50/50 split but the observed counts drift meaningfully away from that, something is interfering with assignment, such as a redirect or a script conflict. Results from a broken split cannot be trusted, so Shuttlebase flags the experiment instead of reporting misleading numbers.

How significance is reported

The dashboard shows significance as high, medium, or low rather than raw p-values. The label combines two checks: the confidence the statistical test reports, and whether the sample is large enough. The displayed level is always the more conservative of the two. A strong-looking lift on a thin sample shows as low, because it has not earned trust yet.

For how to act on each level, see Reading results.

Conservative uplift projections

Every measured effect comes with a confidence interval, a range the true effect plausibly falls in. When Shuttlebase projects the revenue impact of deploying a winner, it uses the lower bound of that interval, not the midpoint.

This makes projections deliberately cautious. A projection built on the lower bound is one the deployed change is likely to meet or beat. Projections built on the point estimate look better on a chart and disappoint in production.

The same cautious estimate drives the earned and locked-in figures on the account performance page, where each win’s contribution also fades over time. See Revenue uplift for the full model and Performance for how those figures combine.

The recommendations engine

Shuttlebase turns test results into one of five recommendations per experiment:

Recommendation	When it applies
Deploy winner	A variant beats the baseline on the primary metric with sufficient significance and clean guardrails.
End experiment	The variants are losing or the experiment cannot produce a useful result.
Disable variant	One variant is clearly underperforming while the rest of the experiment is still worth running.
Keep running	The data does not yet support a call in either direction.
Manual review	The situation needs human judgment, for example when the estimated revenue loss from a variant is large.

Recommendations are held back until an experiment has run for a minimum period and recorded enough conversions. This prevents calls based on early swings, which are common and usually misleading.

Depending on your automation settings, some recommendations execute automatically and others wait for your approval. See Automation settings for the options and Experiment terminal states for what each outcome does.