A/B testing in ShopGuide
The A/B Test page, if it's enabled for your store, lets you actually measure whether the chat widget — and a few of its specific behaviors — move the needle. It's split into two parts: the Main Test and Advanced Tests.
Main Test: chat vs. control
One purpose-built experiment: does showing the chat widget improve outcomes compared to not showing it at all?
- Split — fixed 50/50 between control (no chat) and chat (chat shown)
- Duration — fixed at 14 days
- Scope — global, store-wide, not per page
Each run gets its own test ID, and starting a new one replaces whatever configuration was running before. Results show, per variant: unique users, chat views and interactions, orders and total revenue (pulled directly from Shopify orders tagged with that variant), and conversion rate (orders over users).
Advanced Tests
A second tab runs one focused test at a time on a specific widget behavior:
| Test | What it compares |
|---|---|
| Auto-Scroll Test | Whether auto-scrolling to the chat improves engagement |
| Launch Message Type Test | Dynamic, AI-generated launch messages vs. your static custom one |
| Proactive Message Test | Whether a proactive message bubble increases engagement and conversions |
Only one runs at a time — starting a new one stops whichever is active.
Setting one up
For the Main Test: open A/B Test, give it a name (or keep the default, "Chat vs Control Test"), and start it. The split and duration are fixed, so there's genuinely nothing else to configure. For an Advanced Test, pick a type from the dropdown, name it, and start it the same way.
Reading what comes back
Compare conversion rate, chat interactions, and revenue across variants. Because the split and duration are fixed for the Main Test, look for a difference that's large and holds up, rather than expecting a formal significance calculation — ShopGuide doesn't run one for you. If your store is small, 14 days might not be enough traffic to be confident either way; running it again during a comparable period is a reasonable next move if the result looks marginal.
A couple of things worth keeping in mind: let the Main Test run its full 14 days before deciding anything — the early data is noisy. Run one advanced test at a time, so you actually know what changed. And keep seasonality in view; a test that spans a holiday or a big promotion will tell you more about that event than about the chat widget.