A/B test results & reporting
ShopGuide's built-in A/B testing splits your store's traffic across variants and measures the business impact using real Shopify order data. This page covers what the results actually contain.
Variants and assignment
A test is configured with a name, a start/end date, and a traffic split across up to three variants (e.g., control, chat, highlights), each assigned a target percentage of visitors. Each visitor is recorded with the variant they landed in, whether they viewed the chat widget, and whether they interacted with it.
What gets measured
For each variant, the results pull together:
- Users — unique visitors assigned to the variant
- Chat views / chat interactions — how many of those visitors opened chat, and how many actually sent a message
- Orders and revenue — pulled directly from Shopify orders tagged with the variant (via a
ab_test_variantorder attribute), filtered to your store's online sales channel and the test's date range - Conversion rate — orders in the variant ÷ users in the variant
- Average order value — total variant revenue ÷ variant order count
Results are available while a test is running (using the current date as the end of the window) and after it ends (using the configured end date).
What this isn't
ShopGuide's A/B testing does not compute statistical significance, p-values, or confidence intervals — there's no built-in test for whether an observed difference between variants is statistically reliable. There's also no geographic segmentation of results and no automated phased-rollout tooling. If you need statistical rigor before acting on a result, treat the numbers above as directional and apply your own significance testing to the underlying counts, especially with a small number of users or orders in either variant.