Experiments Workflow
Everything about running experiments in Tailor: from planning to promoting a winner.
A/B vs. Multi-Variant (A/B/C)
Most experiments should be a simple A/B test (control vs. one variant). Adding more variants fragments your traffic and slows learning.
When to Use Multi-Variant
Only add a third variant when you have enough conversion volume that splitting traffic three ways won't slow results to a crawl. Variants must be meaningfully different (not tiny copy tweaks).
If you're struggling to get steady conversions with two variants, stick to A/B. Sequential iteration (promote the winner, then test the next idea) is almost always better than testing five variants at once.
Setting & Changing the Control
The control variant is the baseline your treatment is measured against. If you want to change the control mid-experiment, the best practice is to stop the experiment, set the new control, and start a fresh test. Changing the control mid-flight contaminates the measurement window and can produce false positives.
Ramping Safely
Tailor defaults to a 50/50 traffic split, which works for the vast majority of experiments. Gradual ramping (starting at 10% and increasing) is only needed for very high-volume pages with sensitive outcomes.
Recommended Approach
- Launch at 50/50 unless you have very high traffic and a sensitive page
- Monitor results after launch
- Deramp immediately if you see a clear negative trend
Reading Results
Use this five-point framework to evaluate your experiment:
- 1Primary goal: did the variant improve meaningfully vs. control on your conversion goal?
- 2Volume: do you have enough conversion volume to trust the direction?
- 3Consistency: does the lift hold across days and major segments?
- 4Diagnostics: are clicks up without downstream lift? That's a warning sign (engagement without conversion).
- 5Confounds: did campaigns, deploys, or tracking change mid-test?
"Too Early" / Insufficient Data
"Too early" means there isn't enough signal on your goal to separate real lift from noise. There's no magic number for how long to wait. It depends on stable conversions per variant over time.
How to Speed Up Results
- Increase conversion volume by driving more qualified traffic
- Reduce variants (A/B instead of A/B/C) to concentrate traffic
- Temporarily use a higher-frequency proxy goal (e.g. form starts instead of form submits)
Broken tracking can make results look "too early" indefinitely. If data isn't accumulating, check that your conversion goal is firing correctly on both control and treatment before waiting longer.
Minimum Traffic Needed
There's no universal minimum, but the rule of thumb is: you need enough conversions that you're not reading noise. If your page converts at 2% and you get 100 visitors/week, that's about 2 conversions per variant per week. It will be very hard to detect lift at that volume.
For low-volume pages: stick to A/B (not multi-variant), make bigger and more meaningful changes, and consider a higher-frequency proxy goal to validate direction before committing to a longer test.
Promoting a Winner
When you're confident in the results, here's the workflow:
Confirm lift on primary goal
Make sure the improvement is on your real conversion goal, not just clicks or engagement metrics.
Ramp to 100%
Click "Ramp to 100%" to send all traffic to the winning variant.
Start the next test
Use the winner as the new control and test one new hypothesis. Don't pile multiple changes into one test.
Monitor for regression
Watch metrics for a few days after promotion. Novelty effects and promo bursts can fake a winner.
Stopping a Test Safely
To stop an experiment without losing data:
- 1Deramp the variant (set allocation to 0%). This stops exposure without deleting anything.
- 2Confirm the control experience is serving normally.
- 3Record the final readout: dates, traffic, conversions, and lift.
- 4Decide: promote the winner, iterate on the variant, or scrap it.
Deramp vs. Delete
Always deramp first. Deramping keeps the variant for future reference or re-testing. Only delete if you're certain you won't need it again. Available experiment actions are: ramp, ramp to 100%, deramp, and delete. There is no pause action.
Scheduling Experiments
Experiment scheduling (e.g. start Monday, end Friday) is not supported today. The workaround is to QA your experiment ahead of time, then launch and stop it manually. This is a known request and may be added in the future.