In 2000, Google ran its very first A/B testing to figure out the perfect number of search results per page. It crashed. The page loaded too slowly and the whole thing was a disaster. Today, Google runs over 10,000 A/B tests every single year. The lesson? Even the world’s most data-obsessed company had to start somewhere embarrassing.
If you are running a website, a product or a landing page and making design decisions based on what your boss thinks looks cool, you are essentially throwing darts blindfolded. A/B testing is the blindfold removal service. It is a simple, structured way to replace gut feelings with actual evidence about what your users want.
This guide covers everything from what A/B testing actually is, when it can help you and when it can waste your time, how to run it properly, the common mistakes people make, the tools worth using and where the discipline is heading.
TL;DR
- A/B testing is a controlled experiment where you show two versions of something to real users and measure which one wins based on data, not opinions.
- It replaces guesswork with evidence. But skip it if your traffic is too low or your fundamental UX is broken.
- Research and hypothesis, sample size calculation, execution and analysis. Miss any phase and your results are meaningless.
- Stopping too early, testing too many things at once, ignoring guardrail metrics and six other things that will quietly ruin your results.
- CUPED variance reduction, Multi-Armed Bandits and Sequential Testing for teams ready to go beyond basic split tests.
- The Facebook Emotional Contagion experiment proved that testing on humans without proper consent is a bad idea, legally and morally.
- AI will write the hypotheses, run the tests and pick the winner. Your job is to ask the right questions.
What is A/B Testing?
You have a webpage, and you make two versions of it. Version A is what you already have. Version B has one change, maybe a different button color, a new headline or a shorter form. You split your visitors randomly so half see A and half see B. Then you measure which version gets more people to do the thing you want them to do. The version that wins gets used. The one who loses gets retired. That is it. That is what is a/b testing in its purest form.
More formally, it is a randomized controlled experiment, which is the same scientific method James Lind used in 1747 when he split sick sailors into pairs and gave each pair a different potential scurvy cure. Citrus fruits won. The other sailors did not fare as well. A/B testing is essentially that, except your variants are button colors instead of lemons and nobody is dying either way.
How Does It Actually Work?
When a user visits your site, your A/B testing tool assigns them to a group, either control (Version A) or variant (Version B), using a stable identifier like their User ID or Device ID. This means the same person always sees the same version throughout the experiment. No switching mid-test. Consistency matters.
You collect data, wait and analyze. If Version B produces a statistically significant improvement in your chosen metric, you ship it. If it does not, you keep A and go back to the drawing board with a better hypothesis.
What Can You Test?
Almost anything that appears on a screen. The most common elements teams test include:
- Headlines and page copy
- Call-to-action button text, color, size and placement
- Page layouts and visual hierarchy
- Form length and field labels
- Pricing page structures
- Email subject lines and preview text
- Product images and thumbnails
For a/b testing for landing pages specifically, the highest-impact elements are almost always the headline, the hero section CTA and the social proof placement. Start there before touching anything else.
“In 2012, a Microsoft engineer tested a small change to how Bing displayed advertising headlines. Within hours, it produced a 12% revenue increase with no negative impact on user experience metrics.”
Why A/B Test? (And When NOT To)
The standard reason everyone gives for A/B testing is that it replaces the HiPPO, which stands for Highest Paid Person’s Opinion. The HiPPO phenomenon is when the most senior person in the room decides what the website should look like based on personal preference and everyone else politely agrees. A/B testing gives you something more powerful than seniority, that is, evidence.
Research from Optimizely shows that personalized experiences generated 41% more impact than general ones across 127,000 experiments. That kind of insight only comes from testing, not from meetings.
Good Reasons to Run an A/B Test
- You have a specific conversion metric you want to improve, like sign-ups, purchases or demo bookings.
- You have a real hypothesis based on user behavior, not just a hunch from your design team.
- You have enough traffic to reach statistical significance within a reasonable timeframe.
- You want to understand how small, isolated changes affect user behavior at scale.
When You Should NOT Run an A/B Test
This is the section most A/B testing guides skip. They want you to test everything. Here is the honest version:
Your traffic is too low
To get statistically reliable results, you typically need thousands of visitors per variant. If your site gets 500 visitors a month, a standard A/B test will take so long to complete that the results will be meaningless by the time you have them. Use qualitative user research instead. Platforms like Articos are designed exactly for this situation: they give you structured audience insights in under 30 minutes without needing a large pool of live traffic.
You have not fixed the obvious UX problems yet
If users are abandoning your checkout because they cannot find the payment button, changing the button color in an A/B test will not help. Fix the fundamental usability issues first. Then test the refinements. Nielsen Norman Group’s research has consistently shown that testing the wrong element because the actual problem was never identified is one of the top reasons A/B tests fail to produce useful results.
You are testing multiple changes at once
If you change the headline, the image, the CTA text and the layout all in one variant and it wins, you have no idea which change did the work. Test one thing at a time in A/B testing. If you genuinely need to test multiple interacting elements simultaneously, that is what multivariate testing is for.
Quick Rule of Thumb:
If your page gets fewer than 1,000 unique visitors per month, skip A/B testing and use simulated user research or moderated usability testing to find problems first.Fix the problems you find. Then A/B test the improvements once your traffic grows.
Insights in 30 minutes, not 12 weeks.
Try your first simulated user research for free.
How to Run an A/B Test: The 4-Phase Practitioner’s Roadmap
Running a proper A/B test is not complicated but it does have a specific order of operations. Skipping steps does not save time. It just produces bad data that gives you false confidence.
Phase 1: Research and Hypothesis Formation
Before you open any testing tool, you need to understand what problem you are actually trying to solve. This is the phase everyone rushes and then wonders why their tests never produce meaningful uplifts.
Start with data. Look at your analytics: bounce rates, drop-off points in your conversion funnel, heatmaps, session recordings and exit surveys. These will tell you where users are struggling. Only then do you form a hypothesis.
A strong hypothesis follows this structure:
“Changing [element] from [current state] to [new state] will increase [metric] because [reason based on user behavior data].”
That last part, the “because,” is what separates a real hypothesis from a guess dressed up in formal language. If you cannot explain the user behavior reason behind your hypothesis, you are still guessing. This is what a/b testing best practices start with: the “why” before the “what.”
Phase 2: Sample Size Calculation
This phase is where most teams embarrass themselves by skipping it entirely and then stopping the test when the results “look good.” Do not do this.
Before you launch, calculate the sample size you need using a power analysis. You need four inputs:
- Baseline conversion rate: what your current page converts at.
- Minimum Detectable Effect (MDE): the smallest improvement that would actually matter to your business.
- Significance level (alpha): typically 0.05, meaning you accept a 5% chance of a false positive.
- Statistical power: typically 0.80, meaning an 80% chance of detecting a real effect if one exists.
Free calculators like Evan Miller’s sample size calculator will do this math for you. The output tells you the minimum number of visitors per variant before you can trust the results.
Phase 3: Execution and Randomization
Launch the test. Assign users to groups using stable identifiers so the same user always sees the same version. Run the test for at least two full weeks, not because of an arbitrary rule but because user behavior varies by day of week. One week of data will miss the patterns that only show up across a full business cycle.
Do not touch the test while it is running. Don’t change the traffic split. Do not add a new variant midway through. Just let it run.
Phase 4: Analysis and Documentation
Once you hit your pre-calculated sample size, analyze the results. If your p-value is below 0.05 and the variant outperformed the control, you have a winner. Ship it.
But here is the part almost nobody does: document everything in a searchable knowledge base. Your failed tests are not failures. They are data about what your users do not respond to. Every test result, win or loss, should be recorded with the hypothesis, the metrics, the outcome and the interpretation. This is how experimentation compounds over time.
What metrics should you track? Set one primary metric before the test begins, the one that defines success. Then set guardrail metrics, the ones that must not get worse. For example, if you are testing a shorter checkout form, your primary metric might be form completion rate. Your guardrail metrics might be average order value and customer support ticket volume.
How Long Should an A/B Test Run?
Test duration is one of the most misunderstood aspects of A/B testing. The answer isn’t a number of days – it’s a number of conversions.
The right approach:
- Before starting the test, use a sample size calculator to determine how many visitors each variant needs
- Input: baseline conversion rate, minimum detectable effect, desired confidence level (95%)
- The calculator returns a sample size – run until both variants reach that number
- Minimum recommended duration: two full weeks, regardless of traffic volume, to capture weekly patterns
- Maximum: 8 weeks, beyond which external factors (seasonality, market shifts) start to confound results
Never end a test because one variant “looks like it’s winning.” Early results are volatile. Tests that look promising at day 5 frequently reverse by day 14.
Dangers of Stopping Tests Early
Stopping a test early is one of the most common – and costly – A/B testing mistakes. Here’s why it happens and why it’s wrong:
Why it’s tempting: Early results look decisive. The variant has a 25% lift on day 3. Why wait?
Why it’s wrong: Early traffic skews toward your most engaged, highest-intent visitors. Their behavior doesn’t represent your full audience. As more “average” visitors arrive, the effect size almost always compresses.
The statistical term for this is “optional stopping” or “peeking.” It inflates false positive rates significantly – tests ended when they first reach significance will produce incorrect conclusions roughly 30–40% of the time.
The fix: Pre-specify your sample size before the test starts and commit to running until you reach it.
Representative Sampling
A valid A/B test needs to capture a representative sample of your audience. That means:
- Running across at least one full business week (weekday vs. weekend behavior differs)
- Not starting tests immediately after a major campaign or announcement (traffic skews toward abnormal intent)
- Not ending tests mid-week, when one day’s traffic may represent an unrepresentative segment
Recommended Test Durations
| Traffic level | Typical minimum duration |
| < 500 visitors/week | 6–8 weeks minimum |
| 500–2,000 visitors/week | 3–4 weeks |
| 2,000–10,000 visitors/week | 2 weeks |
| > 10,000 visitors/week | 1–2 weeks, but still don’t end early |
These are minimums. Always defer to your sample size calculator over these rules of thumb.
Factors That Impact A/B Test Duration:
- Traffic volume – higher traffic = faster statistical significance
- Baseline conversion rate – lower baseline rates require more conversions to detect meaningful differences
- Minimum detectable effect – the smaller the improvement you’re trying to detect, the more data you need
- Test variance – highly variable metrics (like revenue per visitor) require more data than stable metrics (like click rate)
- Number of variants – A/B/n tests split traffic further, extending required run time per variant
How to Run Pre-Traffic A/B Tests with Articos
Most A/B testing advice skips the part where you have no traffic yet.
Split testing tools assume you already have thousands of daily visitors. You don’t. So you do what most early-stage builders do – stare at two versions of your homepage, poll your team, ask a friend who is politely supportive, and eventually pick the one that feels right. Then you spend your ad budget finding out you were wrong.
Articos handles this differently. Instead of sending real visitors to two variants and waiting weeks for statistical significance, it runs structured interviews with synthetic personas – AI-modeled users built around your specific target audience. You get a research-grade comparison report in about 30 minutes, before a single dollar touches paid media.
Here is exactly how that works, using a real scenario.
The Setup
Say you run a bootstrapped SaaS called Taskline – a project management tool for freelancers juggling multiple clients. You are about to launch your first Google Ads campaign with a $500 budget, and you are sitting on two homepage hero variants your team cannot agree on:
Variant A (feature-led): “Manage all your client projects in one place”
Variant B (outcome-led): “Never miss a deadline or lose a client again”
Same layout, same CTA, same dashboard screenshot. Only the headline and subheadline differ. Your team is 50/50. Rather than burn the campaign budget to find out, you open Articos.
Step 1: Choose Your Testing Intent
The first thing Articos asks is what you actually want to learn. This is where most testing tools skip past something important.
Articos gives you eight distinct testing intents to pick from: Conversion Clarity, Value Proposition, Trust and Credibility, CTA Effectiveness, Visual Appeal, Message Resonance, Information Hierarchy, and Overall Preference.
These are not just labels – they shape what the synthetic personas are looking for when they evaluate your variants. Testing for Message Resonance means the personas are specifically examining how emotionally and contextually connected each headline feels to someone in your target audience. Testing for Conversion Clarity would ask something different: which variant makes it clearer what the user is supposed to do next.
For the Taskline scenario, the question is straightforward: which headline actually lands with a freelancer who is stressed about losing clients and missing deadlines? That is Message Resonance. Select it and move on.
Step 2: Upload Your Variants
Articos accepts both landing page screenshots and URLs. We uploaded two variants separately side by side.
Upload Variant A and Variant B – in this case, full above-the-fold screenshots of both hero sections.
There is nothing to configure here beyond the uploads. No code snippets, no UTM parameters, no minimum traffic thresholds. The test does not depend on your site being live at all, which is exactly the point. You can run this on a Figma mockup, a Webflow draft, or a live staging URL – whatever stage you are at.
Step 3: Review the Generated Personas
This is where Articos does the work that would normally take you a week.
Based on the testing intent and your uploaded variants, Articos generates the synthetic personas it will interview. For the Taskline test, it builds profiles that match the target audience: freelance designers and creatives managing three to six active client projects, working independently, sensitive to deadline pressure and the anxiety of a client relationship going sideways.
Each persona comes with demographic context, psychographic attributes, and the specific behavioral patterns relevant to the use case. You are not getting a generic “35-year-old professional” – you are getting a persona that holds the anxieties, priorities, and language patterns of the actual person your headline needs to speak to.
Review them before the interviews run. If the personas look right, proceed. If the platform has generated something that does not match your actual target user, you can adjust before anything happens.
Step 4: The Interviews Run
Articos runs the interviews in parallel – each synthetic persona evaluates both variants through a structured AI-moderated conversation. The platform generates the interview script automatically based on your selected testing intent, so the questions probe specifically for message resonance rather than drifting into general feedback.
You do not need to write the questions. You do not need to recruit anyone, schedule a session, or wait for a participant to show up. The whole process runs in the background while you do something else.
For most tests, results are ready within 30 minutes.
Step 5: Read the Report
The output is a structured research report comparing variant performance across your personas, with a clear indication of which variant won and – more importantly – why.
For the Taskline test, Variant B outperformed Variant A on Message Resonance. The reasoning matters more than the score: the personas flagged that Variant A described a feature state (“manage projects in one place”) without connecting to the emotional weight of the problem. It tells you what the product does. Variant B, by contrast, names the specific fear that freelancers carry – missing a deadline, losing a client – which made it feel immediately relevant rather than just informative.
That is the kind of insight that changes how you write headlines, not just which one you pick this time. The report gives you the reasoning behind the result, not just a winner percentage.
What Makes This Different from Traditional A/B Testing
Traditional A/B testing answers “which variant performed better with our existing audience.” Articos answers something slightly different and often more useful: “which variant will resonate with the people we are trying to reach, before we reach them.”
That distinction matters most in three situations:
Pre-launch. You have no traffic data. Testing is the only way to make an informed call.
Paid campaigns. You have a fixed budget and limited runway to find a winner. Burning ad spend on a losing headline is expensive discovery.
High-stakes rewrites. A homepage refresh, a positioning pivot, a new ICP – changes where you want signal before you commit to rebuilding around the wrong message.
Articos does not replace live traffic testing at scale. Once you have volume, real-world conversion data will always tell you more. What it does is close the gap between “we think this will work” and “we have actual evidence” – at a stage when most teams are making calls on instinct alone.
For the Taskline scenario, the test took one afternoon and cost less than a single day of wasted ad spend. The headline that ran on the campaign was Variant B. The reasoning from the report made it easy to defend to stakeholders. And there was no need to wait for the data to confirm what synthetic research had already surfaced.
That is the practical value of running the test before the traffic arrives.
Ready to test your own variants? Start a free Articos trial →
The 7 Common Mistakes in A/B Testing
Consider this the hall of shame. Every one of these mistakes is made constantly by teams that think they are running rigorous experiments.
Mistake 1: Peeking
You launch your test on Monday. By Wednesday, it looks like Variant B is winning. You stop the test and declare victory. Congratulations. You have just inflated your false positive rate to roughly 14% instead of the 5% you planned for. Stopping a test early because it looks promising is called peeking and it is one of the most common ways teams fool themselves with their own data.
Mistake 2: The Multiple Comparison Fallacy
You track 12 different metrics in one test. Three of them show statistically significant improvements. You get excited. But when you test 12 metrics with a 5% significance threshold, pure random chance alone will produce a false positive in at least one metric most of the time. If you track multiple metrics, apply a Bonferroni correction to your alpha threshold.
Mistake 3: The Novelty Effect
Users notice changes. Sometimes they click on something new, not because it is better but because it is different. This initial engagement spike almost always fades within one to two weeks. Running a test for less than two weeks risks mistaking novelty for improvement.
Mistake 4: Selection Bias
If your randomization is broken and certain user segments end up disproportionately in one group, your results reflect the behavior of that segment, not your full user base. Always run a Sample Ratio Mismatch (SRM) check before interpreting results.
Mistake 5: Ignoring Guardrail Metrics
Your variant increased sign-ups by 15%. But page load time increased by 800 milliseconds and support ticket volume went up 30%. Did you win? Define guardrail metrics before launching any test and treat them as non-negotiable.
Mistake 6: Testing Without a Qualitative Foundation
Testing the wrong thing confidently is not better than not testing at all. If your hypothesis is based on internal opinions rather than observed user behavior, your variations will not address the actual problem. A/B testing ux research and qualitative methods work together. Use user interviews, session recordings and heatmaps to identify the problem. Use A/B testing to validate the solution.
Mistake 7: Abandoning the Experiment Knowledge Base
Most teams document their wins and forget their losses. This is backwards. Failed tests are your most valuable asset because they tell you exactly what your users do not respond to. Document every test. Build institutional memory. Let future hypotheses be informed by the full history of what you have learned.
Ethics, Privacy and Informed Consent in A/B Testing
In 2014, researchers from Facebook and Cornell University published a study where they had quietly manipulated the News Feeds of 689,003 users to test whether emotions could be spread through social networks. They showed some users more negative posts. They showed others more positive posts. Then they measured whether users’ own posts changed as a result.
The results were interesting. The public reaction was not. The study was met with widespread condemnation because the participants had never given specific informed consent to be in a psychological experiment. Facebook argued that users had agreed to its Data Use Policy. Ethicists and regulators disagreed vigorously.
The lesson for practitioners is this: there is a meaningful difference between optimizing a checkout button and manipulating someone’s emotional state. Most commercial A/B testing falls comfortably within acceptable ethical boundaries. But the boundary exists and understanding where it is matters.
What Privacy Laws Require
- Requires transparency in data collection and processing. Users have the right to know how their behavior is being analyzed.
- Gives consumers the right to opt out of the sale of their personal data, which can affect targeting and segmentation in experiments.
- Any experimentation involving patient data requires specific compliance. Platforms like VWO and Kameleoon offer HIPAA-compliant versions.
Privacy-First Experimentation
The industry is moving toward cookieless tracking and warehouse-native testing, where experiment data stays within your own data infrastructure rather than being shared with third-party tools. Tools like GrowthBook and Eppo are built for this model. If GDPR compliance is a concern for your team, this architecture is worth exploring.
“Cambridge Analytica harvested the personal data of up to 87 million Facebook users without direct consent to build psychographic profiles for political advertising, resulting in a $5 billion FTC fine against Facebook.”
Choosing the Right A/B Testing Tools
There is a tool for every budget, technical level and use case. Here is how to think about the decision rather than just listing names and hoping one sounds familiar.
Free and Open Source A/B Testing Tools
If you want free a/b testing tools without compromising on quality, GrowthBook is the standout option. It is open source, warehouse-native (it runs tests on your own data in Snowflake or BigQuery) and trusted by teams that care about data privacy. It is more technically demanding to set up but it gives you full control.
Google Optimize was the default free option for years before Google shut it down in 2023. The gap it left has been partially filled by GrowthBook and by Varify.io, which offers a cookieless, GDPR-friendly alternative at a flat monthly rate.
Best A/B Testing Tools by Use Case
| Tool | Best For | Pricing |
| GrowthBook | Open source, privacy-first teams, warehouse-native testing | Free / Low-cost |
| VWO | Growing teams needing heatmaps + AI suggestions + A/B in one platform | $200-$500/mo |
| Optimizely | Enterprise teams running omnichannel experiments at scale | $50,000+/yr |
| Statsig | Developer teams wanting auto-CUPED and feature flag management | Usage-based |
| Varify.io | Budget-conscious GDPR-compliant teams, cookieless testing | ~119 EUR/mo |
| Kameleoon | Enterprise AI personalization with HIPAA compliance | $35,000+/yr |
One thing worth mentioning: the best a/b testing tools for your team specifically depend on where you are in the research process. If you need to validate ideas before you have enough traffic to run a proper A/B test, a platform like Articos fills that gap. It simulates structured audience conversations and gives you instant landing page reaction data without needing live traffic at all. Think of it as the research phase that makes your eventual A/B tests far more likely to produce meaningful results.
The Future of A/B Testing
Here is where things get genuinely interesting and slightly unsettling, depending on your job title.
AI-Generated Hypotheses
Currently, the hardest part of A/B testing is forming good hypotheses. It requires qualitative research, behavioral data analysis and creative thinking about what might resonate with users. Generative AI is increasingly capable of doing this at scale by analyzing behavioral patterns and producing test variants with copy tailored to specific user segments.
Microsoft and Google have been running over 10,000 experiments annually for years. AI systems that can automate hypothesis generation will push that number into the hundreds of thousands, with human oversight focused on strategy and ethical review rather than individual test design.
Multi-Armed Bandits Replacing Traditional Tests
As real-time personalization becomes the norm, static A/B tests with fixed traffic splits will increasingly give way to adaptive algorithms that optimize continuously. The experiment never fully ends. The experience never stops improving. This is particularly true for content-heavy platforms and e-commerce, where the opportunity cost of showing a suboptimal variant is measured in real revenue.
The Three Stages of Experimentation Maturity
Most organizations currently sit in one of three stages:
- Reactive: Testing happens occasionally when someone has a strong opinion about a change. Results are rarely documented.
- Systematic: Regular testing cadences exist. Teams have defined metrics. Data begins informing decisions instead of just confirming them.
- Strategic: Experimentation is embedded in the product development workflow. AI assists with hypothesis generation. The Experiment Knowledge Base drives compounding returns over time.
The gap between Stage 1 and Stage 3 is not primarily a technology gap. It is a culture gap. Organizations that invest in experimentation culture consistently outperform those that treat A/B testing as a quarterly activity.
“Barack Obama’s 2007 presidential campaign used A/B testing on their website to test four different donation buttons alongside six different accompanying images. The winning combination significantly increased online fundraising and volunteer sign-ups.
Conclusion
Let’s be honest. A/B testing has a reputation for being the thing companies do instead of actually talking to their users. And sometimes that reputation is earned. Teams run button color tests when what they really need is to understand why users are leaving the page in the first place.
But done properly, A/B testing is one of the most powerful tools available for making product decisions with real evidence. The keyword is “properly.” That means qualitative research before the test, a legitimate power analysis before launching, running for a full business cycle, documenting everything, including the failures and treating statistical significance as the beginning of the conversation rather than the end.
The companies that run 10,000 experiments a year are not doing that because they have more opinions than you. They are doing it because they have built systems for learning from their users continuously and at scale.
Start with one well-formed hypothesis. Run one clean test. Document what you learned. Then do it again. That is the entire playbook.
And if you are not yet at the traffic level where A/B testing makes sense, start with the research phase. Tools like Articos give you structured user insights in under 30 minutes, so your eventual tests are built on real user intelligence rather than internal assumptions.
Check the Articos blog for more guides on evidence-based decision making across the full research stack.
FAQs: A/B Testing
A/B testing is a randomized experiment where you show two versions of something to real users and measure which performs better. In user research, it is the quantitative method that validates whether a change actually improves user behavior, as opposed to qualitative methods that explain why behavior happens.
Start with the elements closest to your primary conversion goal: the headline and hero section CTA on landing pages, the checkout flow in e-commerce and the sign-up form in SaaS products. These have the highest impact-to-effort ratio and will generate the most meaningful data fastest
Use this structure: “Changing [element] to [new state] will improve [metric] because [observed user behavior data].” The “because” clause is what makes a hypothesis testable rather than a guess. If you cannot point to qualitative data supporting it, run user research first.
Define one primary metric before launch, the number that decides the winner. Then define two to three guardrail metrics that must not decrease. Common primary metrics include conversion rate, click-through rate and revenue per visitor. Common guardrail metrics include page load time, error rate and support ticket volume.
A/B testing is the validation engine of CRO. It confirms whether a change that qualitative research suggested would actually help improve conversion at scale. Without A/B testing, CRO is just a collection of opinions about what might work better.
There is no universal threshold; it depends on your baseline conversion rate and the size of improvement you want to detect. As a rough guide, most teams need at least 1,000 unique visitors per variant for meaningful results. Use a power analysis calculator to get the exact number for your situation.
At a minimum of two full weeks, which covers one complete business cycle and accounts for day-of-week variation in user behavior. Run longer if you have low traffic. Do not stop early because results look good; that is the peeking mistake that inflates your false positive rate.
GrowthBook is the strongest free and open-source option, particularly for teams that care about data privacy and warehouse-native analytics. Varify.io offers an affordable entry point for GDPR-sensitive teams. Both are solid starting points before committing to enterprise platforms.
When both variants have reached your pre-specified sample size AND you’ve hit 95% statistical confidence. Don’t end based on time alone or based on early results.
This means either the difference between variants is smaller than your minimum detectable effect, or there is no meaningful difference. Both are valid learnings. Document what you tested and move on.
Only if the harm is significant and immediate (e.g., the variant is actively breaking conversion). Otherwise, let the test run – early apparent losers sometimes recover.
