A/B Testing - Multi-Variant Support with Weighted Traffic Distribution

A/B Testing just got a major upgrade. You can now test up to 5 theme variants in a single experiment, with full control over how traffic is distributed between them.

What's New?

Test More Than 2 Themes at Once

Previously, A/B tests were limited to Theme A vs Theme B. Now you can add 2 to 5 variants per test. Name each variant, assign a theme, and let the app handle the rotation.

Perfect for:

• Comparing multiple redesigns — Test 3 homepage layouts side by side
• Seasonal fine-tuning — Run 4 holiday themes and find the one that converts best
• Incremental changes — Compare your current theme against small variations

Weighted Traffic Distribution

Not all variants need equal exposure. Assign custom weights (1–10) to each variant to control how much traffic it receives. For example, give your current theme a weight of 4 and a new design a weight of 1 to test conservatively.

Two modes are available:

• Equal — Every variant gets the same share of traffic
• Custom — Fine-tune the weight of each variant individually

A visual bar shows the exact percentage breakdown, and a rotation summary explains how periods are distributed in practice.

Smart Cycle Compatibility

The app now detects when your test duration doesn't align with complete rotation cycles. If there's a mismatch, you'll see a warning with suggested end dates — pick one and confirm to adjust automatically. This ensures every variant gets its fair share of exposure.

Statistical Significance

Results are now backed by real statistics:

• 2 variants — Z-test for proportion comparison
• 3+ variants — Chi-squared test with Bonferroni correction for pairwise comparisons

Once enough data is collected, the app tells you which variant is winning and at what confidence level.

How It Works

1. Go to A/B Tests → Create
2. Name your test and add a description
3. Add your variants (2–5), pick a theme for each, and name them
4. Set your start and end dates
5. Choose a switch interval and configure traffic weights
6. Review the summary and launch

The app generates a weighted rotation schedule that interleaves variants proportionally to their weights, minimizing time-of-day bias.

Backward Compatible

Existing A/B tests continue to work exactly as before. The new multi-variant system runs alongside the original 2-theme setup — no migration needed on your end.

Availability

Multi-variant A/B Testing is available on the Scale plan.

---

Stop guessing which theme works best. Test them all at once, let the data decide, and ship the winner with confidence.