A/B test results & reporting
ShopGuide's built-in A/B testing splits traffic across variants and ties the outcome to real Shopify orders. Here's exactly what's in the results.
How visitors get assigned
A test has a name, a start and end date, and a traffic split across up to three variants — say, control, chat, and highlights — each with a target percentage. For every visitor, ShopGuide records which variant they landed in, whether they viewed the chat widget, and whether they interacted with it.
What the results actually contain
For each variant:
- Users — unique visitors assigned to it
- Chat views / chat interactions — how many opened the widget, and how many actually sent a message
- Orders and revenue — pulled directly from Shopify orders tagged with the variant (via an
ab_test_variantorder attribute), filtered to your online sales channel and the test's date range - Conversion rate — orders divided by users
- Average order value — total variant revenue divided by variant order count
You can read results while a test is still running, using the current date as the end of the window, or after it's finished, using the configured end date.
What's not calculated
No p-values, no confidence intervals, no statistical significance test. If a difference between variants looks real, that's a judgment call you're making, not a claim ShopGuide is making for you — treat the numbers as directional, especially with a small number of users or orders in either variant, and apply your own significance testing if you need to be sure before acting. There's also no geographic segmentation of results and no automated phased rollout.