A/B testing in ShopGuide
The Experiments page lets you measure whether a change to the chat widget actually moves the needle. Open it from Extras → A/B Testing. Several experiments can run at the same time, and each one splits your visitors between two or more variants.
Two ways to build an experiment
Describe a test
In the Describe a test card, type what you want to test in the box ("What do you want to test?") and click Build experiment. For example: "I added the local pickup button. Please make an A/B test for visible vs non-visible, 21 days."
ShopGuide reads your live storefront, finds the element you mean, builds the experiment, and checks that it really changes something. You then see a preview with:
- Hypothesis
- What it changes
- Variants
- Verification
- Schedule
If the request is unclear, you are asked a follow-up question first. Nothing goes live until you review the preview and click Launch experiment.
Write it in code
Click New Experiment to write the experiment yourself as a small JavaScript object: a name, and a list of variants. Each variant has an optional key and name, a weight (its share of visitors), and a run(ctx) function that executes in the visitor's browser once they are assigned to it. Weights don't need to add up to 100; they are normalized when you save. Click Start Experiment to save it. Syntax mistakes and unsupported patterns are caught before anything is saved.
The editor opens with a starter template (a bubble vs. no-bubble test, 50/50). The Read the full guide link on the page opens the complete reference.
The ctx object can:
| Call | What it does |
|---|---|
ctx.setFlag(key, value) | Turns a pre-wired widget behavior on or off for this visitor |
ctx.getFlag(key) | Reads a flag set earlier in the same run call |
ctx.hideElement(selector) / ctx.showElement(selector) | Hides or shows any element on the page by CSS selector |
ctx.addCSS(css) | Injects a style rule |
ctx.setText(key, value) | Sets a pre-wired text value |
What you can test
Flags and text values only work if the storefront actually reads them. These are the ones that do:
| Key | What it controls |
|---|---|
show_anchor_bubble | Shows or hides the floating chat launcher bubble. Only takes effect where Display Bubble is on in Settings. |
enable_proactive_messages | Shows or hides the proactive message near the launcher |
auto_scroll_enabled | Whether the page scrolls to the chat when it opens |
use_custom_launch | Uses your static launch message instead of the AI-generated one |
proactive_message (text) | The proactive message's wording |
bubble_icon (text) | The launcher icon: sparkles, chat, message, question, robot or search |
Hiding elements and injecting CSS works against anything on your page. Before launch, ShopGuide rejects keys or values nothing reads ("N widget settings can't be tested") and warns when a selector matches nothing ("Selectors not verified"), so an experiment can't quietly run for weeks and measure nothing.
Duration
New experiments default to 21 days, and 21 days is also the minimum. Run longer if you want a bigger sample. You can pick start and end dates when you create an experiment, and change the end date later with Change end date on the experiment card. A card shorter than the minimum shows a reminder to extend it.
Editing, stopping and deleting
Each experiment card shows a status badge: Active, Ended or Stopped.
- Edit reopens the experiment in the code editor. Saving updates it in place and keeps its start date.
- Stop ends the experiment for all future visitors and freezes its results as Final results. Those stored numbers never change afterwards. An experiment also stops assigning visitors at its end date, but its results are only frozen when you press Stop.
- Delete appears on stopped experiments. It removes the experiment and its saved results, and cannot be undone. A running experiment has to be stopped first.
Reading the results
Click View Results on a running experiment (stopped ones show their final results directly). The results table has one row per variant with these columns: Variant, Users, Orders, Revenue, Revenue / User, Avg Order Value, Conversion Rate, Add to Cart Rate and Chat Interactions.
Below the table, ShopGuide names the variant leading on revenue per visitor and how far behind the others are. Treat that as a direction, not a verdict: a short test often can't separate a real difference from random variation. Orders are re-read from Shopify each time you open the results, so a running test's revenue can still shift with refunds and order edits. Stopping the experiment pins the final figures.
Each visitor is assigned a variant once and keeps it on later visits (it is remembered in their browser).
A few habits make results more trustworthy:
- Let the experiment run its full duration before deciding. Early data is noisy.
- Change one thing per experiment, so you know what caused the difference.
- Keep seasonality in view. A test that spans a holiday or a big promotion tells you more about that event than about the chat widget.
The Global Kill Switch under Settings → General also stops experiments from loading while it is on.