🆎 A/B Testing
A/B Testing allows you to compare two theme variants by automatically alternating between them over time. Track conversions, analyze results, and determine which theme performs better.
A/B Testing is available exclusively on PRO and PRO Yearly plans.
How It WorksDirect link to How It Works
- Select two themes (Theme A = Control, Theme B = Variant)
- Set a schedule with start date, end date, and switch interval
- Leon alternates between themes automatically (e.g., every 24 hours)
- Track conversions via Shopify Web Pixel
- Analyze results to find your winning theme
PrerequisitesDirect link to Prerequisites
Before creating an A/B test, you need to set up tracking:
- Activate the Web Pixel (one-time, shop-level)
- Enable the A/B Test Tracker extension on both themes
See Set Up A/B Test Tracking for detailed step-by-step instructions.
Creating an A/B TestDirect link to Creating an A/B Test
The creation wizard has 6 steps:
Step 1: Test InformationDirect link to Step 1: Test Information
| Field | Description | Required |
|---|---|---|
| Name | Descriptive test name (max 100 characters) | Yes |
| Description | Purpose of the test (max 500 characters) | No |
Step 2: Theme SelectionDirect link to Step 2: Theme Selection
- Theme A (Control): Your baseline theme
- Theme B (Variant): The theme you're testing against
The form verifies that both themes have the tracking extension enabled.
Step 3: Start DateDirect link to Step 3: Start Date
- Select a future date and time
- Choose your timezone
- Toggle between 12/24-hour format
Step 4: End DateDirect link to Step 4: End Date
- Must be after the start date
- Recommendation: Run tests for at least 7 days to get meaningful data
Step 5: Switch IntervalDirect link to Step 5: Switch Interval
How often themes alternate:
| Interval | Best For |
|---|---|
| 6 hours | High-traffic stores, quick insights |
| 12 hours | Balanced testing |
| 24 hours | Most common, day-over-day comparison |
| 48 hours | Lower-traffic stores |
| 72 hours | Minimal switching, longer observation |
Option: Enable Auto-disable WebPixel to automatically turn off tracking when the test completes.
Step 6: SummaryDirect link to Step 6: Summary
Review all settings and click Create to start scheduling the test.
Understanding ResultsDirect link to Understanding Results
Tracked MetricsDirect link to Tracked Metrics
For each theme, Leon tracks:
| Metric | Description |
|---|---|
| Sessions | Unique visitor sessions |
| Page Views | Total pages viewed |
| Add to Cart | Visitors who added items |
| Orders | Completed purchases |
| Revenue | Total order value |
| Conversion Rate | Orders ÷ Sessions × 100% |
Winner DeterminationDirect link to Winner Determination
Leon determines a winner when:
- Each theme has at least 30 sessions (statistical minimum)
- Conversion rate difference is greater than 1%
- Confidence score reaches 95%
Possible results:
| Result | Meaning |
|---|---|
| Theme A Wins | Control performs better |
| Theme B Wins | Variant performs better |
| Tie | No significant difference |
| Insufficient Data | Need more sessions |
Managing A/B TestsDirect link to Managing A/B Tests
Test StatusDirect link to Test Status
| Status | Meaning |
|---|---|
| Scheduled | Created, waiting to start |
| In Progress | Currently running |
| Completed | Test finished |
| Paused | Temporarily stopped |
| Cancelled | Manually stopped |
Available ActionsDirect link to Available Actions
- View details: See results and period timeline
- Pause/Resume: Temporarily halt or continue the test
- Cancel: Stop the test permanently
- Edit: Modify settings (only for scheduled tests)
You can only run one A/B test at a time. Wait for the current test to complete or cancel it before starting a new one.
Example: Testing a New Homepage DesignDirect link to Example: Testing a New Homepage Design
Scenario: You redesigned your homepage and want to test if it improves conversions.
Setup:
- Theme A: Current homepage design
- Theme B: New homepage design
- Duration: 14 days
- Interval: 24 hours
Timeline:
| Day | Theme | What Happens |
|---|---|---|
| 1 | A | Baseline data collection |
| 2 | B | New design goes live |
| 3 | A | Compare with baseline |
| 4 | B | Continue alternating... |
| ... | ... | ... |
| 14 | B | Test completes |
Results: After 14 days, check which theme had better conversion rates.
Best PracticesDirect link to Best Practices
- Run tests long enough: At least 7 days to account for daily variations
- Avoid peak periods: Don't start tests during Black Friday or major sales
- Test one change at a time: Compare themes with a single significant difference
- Wait for statistical significance: Don't stop tests early based on initial results
- Document your tests: Use the description field to note what you're testing
TroubleshootingDirect link to Troubleshooting
"PRO Plan Required"
- A/B Testing requires a PRO or PRO Yearly subscription
- Upgrade your plan in Settings → Subscription
"Cannot create test - active test exists"
- You can only run one A/B test at a time
- Wait for the current test to complete or cancel it
"No data showing in results"
- Verify your tracking setup: Set Up A/B Test Tracking
- Wait at least one full period for data to appear
"Insufficient data for winner"
- Need at least 30 sessions per theme
- Extend the test duration or wait for more traffic
Next StepsDirect link to Next Steps
- Set Up A/B Test Tracking - Activate pixel and theme extensions
- Settings - Configure your timezone and preferences
- Sequences - Chain multiple theme changes together