Skip to content
Peer-reviewed86% recall validated against Baymard Institute and Nielsen Norman Group›

What Is Experimental Design?

Experimental design is the plan that makes a test fair. See the types, key variables, a 120-vs-60 participant example and the mistakes that spoil results.

Experimental Design Definition

Experimental Design is the plan for a study that tests whether changing one thing causes a different result. It fixes what you change, what you hold constant, who is assigned to each condition and what you measure before any data comes in, so a difference in the outcome can be traced to the change.

Key takeaways

  • A sound design has a variable you change on purpose, a group to compare against and random assignment of who sees what.
  • The classic principles are randomization, replication and blocking. Ronald Fisher's 1926 and 1935 books helped establish them.
  • Between-subjects designs give each participant one condition. Within-subjects designs give each participant every condition, which needs fewer people but a counterbalanced order.
  • Without random assignment or a control group, a result shows a link between two things but can't show that one caused the other.
  • Articos supplies participants and analysis for pre-launch tests in under 30 minutes. The design choices stay with you.

Explained

What Is Experimental Design?

Decide the comparison first. Collect the data second.

Experimental design is the structure of a study: which variable you change (the independent variable), which outcome you measure (the dependent variable), which conditions participants are assigned to and how everything else is kept the same. That structure is what lets you say the change caused the outcome, because other explanations have been designed out.

Studies that assign conditions at random are true experiments. Studies that compare groups that already exist, such as customers who picked different plans, are quasi-experiments, and their results carry more doubt because the groups may have differed before the study began. Observational research watches what happens without changing anything.

Why it matters

Why Experimental Design Matters

A number that moves after a change might have moved anyway. Design is what lets you attribute it.

Cause

Random assignment and a control group let you say the change produced the result. Without them you have a correlation, and the usual rival explanations stay open.

Fairness

Groups formed by chance start out alike on average. A difference at the end can't be traced to who happened to land in which group.

Reuse

A written design (variable, control, unit, measure) can be repeated by someone else next quarter, which is how results build on each other.

How it works

How Experimental Design Works, Step by Step

The method is a sequence of decisions. Each one is cheap to make before data exists and expensive to fix after.

All visitorsrandom 50 / 50 splitGroup A · Controlsees the current versionGroup B · Variantsees one changeMeasure one metricsign-ups, purchasesSignificant lift?ship B, or keep A
  1. 1
    Plan

    Name the Variables

    List what you will change (the independent variable), what you will measure (the dependent variable) and what could interfere (confounding variables). Anything that could interfere is held constant or randomized.

  2. 2
    Plan

    Write a Testable Hypothesis

    "Naming the outcome in the headline will raise clarity ratings." One change, one expected effect and one measure that could prove you wrong.

  3. 3
    Plan

    Set the Treatment Levels and a Control

    Decide the versions of the independent variable, and include a control that gets the current version or no change at all.

  4. 4
    Build

    Choose How to Assign Participants

    Between-subjects gives each participant one condition. Within-subjects gives every participant all of them, in a randomized or reversed order (counterbalancing) so that order does not color the results. A randomized block design sorts participants by a shared trait first, then randomizes within each group.

  5. 5
    Run

    Randomize, Run and Replicate

    Assign at random, give every group the same instructions and conditions, and repeat the measurement or the whole study when you can. Replication separates a real effect from one lucky run.

  6. 6
    Decide

    Analyze, Decide and Record

    Compare the outcome across conditions with a method chosen in advance. Write down the hypothesis, the result and the decision so the next design starts from it.

Key elements

Key Elements and Types of Experimental Design

ElementWhat it meansExample
Independent variableThe factor you change on purpose.Headline wording
Dependent variableThe outcome you measure to see whether the change had an effect.Clarity rating from 1 to 7
Control groupThe baseline group that gets the current version or no treatment.The existing headline
Experimental unitThe smallest thing assigned to a condition on its own.One participant
Random assignmentEvery unit has the same chance of landing in any condition.A random number drawn per participant
ReplicationRepeating measurements or the whole study to separate real effects from noise.Re-running with a new sample
BlockingGrouping similar units first, then randomizing within each group, to cut known variation.Block by new and returning customers

Types of Experimental Design

DesignHow it worksBest for / Watch for
Completely randomizedEach unit is assigned to a condition at random.Best for: simple comparisons with similar units. Watch for: chance can produce uneven groups in small samples.
Randomized blockUnits are grouped by a shared trait, then randomized within each group.Best for: samples where a known trait affects the outcome. Watch for: the trait has to be measured before assignment.
Between-subjectsEach participant sees one condition.Best for: tasks where seeing one version would bias reactions to another. Watch for: needs more participants and has person-to-person noise.
Within-subjectsEach participant sees every condition.Best for: small samples and side-by-side comparisons. Watch for: order and learning effects, so counterbalance.
FactorialTwo or more variables change together and every combination is tested.Best for: finding interactions, such as headline and price. Watch for: the number of conditions multiplies quickly.

Example

Experimental Design Example: Testing Two Headlines

An illustrative case: a team wants to know whether a headline that names the outcome is rated clearer than the current one. Both designs below test the same question.

Design 1 · Between-subjects
Same question, two ways to staff it
Between-subjects: 60 people see Headline A and a different 60 people see Headline B. Within-subjects: 60 people see both headlines, with 30 seeing A first and 30 seeing B first.
Each person rates one headline
120participants
60ratings per headline
1headline each
Design 2 · Within-subjectsFewer participants
Same question, two ways to staff it
Between-subjects: 60 people see Headline A and a different 60 people see Headline B. Within-subjects: 60 people see both headlines, with 30 seeing A first and 30 seeing B first.
Each person rates both
60participants
60ratings per headline
2headlines each

Same question, same 60 ratings per headline. The between-subjects design needs 120 people and keeps every rating independent. The within-subjects design needs 60 people but has to counterbalance the order, because the first headline someone reads colors how they read the second. Neither design is better in general. The right choice depends on how many people you can reach and how much carryover the task creates.

When to use it

When Experimental Design Applies, and When It Doesn't

Use Experimental Design When

  • You want to know whether a change causes an outcome
  • You can control what each participant sees
  • You can assign conditions at random
  • A wrong decision costs more than the time to test

It Struggles When

  • The groups already exist and can't be randomized, so the result is quasi-experimental
  • The variable can't be manipulated, such as age or country
  • The effect takes months to show up
  • The test setting differs too much from real use
Pros of Experimental DesignCons of Experimental Design
Isolates cause and effectNeeds planning before any data exists
Can be repeated by someone elseControlled settings can differ from real life
Makes results comparable across studiesTests only the variables you thought to change
Settles disagreements with evidenceNeeds enough participants to detect a realistic effect

Common mistakes

Common Experimental Design Mistakes to Avoid

01

Changing More Than One Thing

If version B has a new headline and a new image, you can't say which one moved the result.

One variable per test, or plan a factorial design.
02

Skipping Random Assignment

Letting people pick their version, or giving the new one to early adopters, builds the difference into the groups.

Assign by chance.
03

Running Without a Control

Comparing this month with last month mixes your change with everything else that changed.

Run control and treatment at the same time.
04

Ignoring Order Effects

In a within-subjects design, the first version people see shapes how they judge the second.

Counterbalance the order.
05

Counting the Wrong Unit

Three answers from one person are not three independent observations, and treating them that way inflates confidence.

Analyze at the level you randomized.
06

Choosing the Measure After the Fact

Picking whichever number moved turns noise into a finding.

Name the dependent variable in the hypothesis.

How Articos helps

How to Run an Experiment Before Launch With Articos

Recruiting participants is the slowest part of most designs. Articos replaces that step for pre-launch tests, and you keep the design.

Recruited participant study6–8 weeksArticos pre-testUnder 30 minutes

Write the design in the morning. Read the results by lunch.

Recruited participant study (6–8 weeks per cycle):

  • Recruiting and scheduling set the timeline
  • No-shows and cancellations shrink the sample
  • Incentives and fees add cost to every study

With Articos:

You write the hypothesis, choose what changes and decide what stays fixed. A panel of AI personas grounded in behavioral science acts as the participants, each with a different stance toward your idea, and returns a report with scores, reasons and direct quotes in under 30 minutes. A study of two versions puts both in front of the same panel and compares them side by side. Articos tests how personas react, so confirm a high-stakes winner with real users or a live test.

Test two versions free →

Putting it into practice

Putting Experimental Design Into Practice

Write the Design on One Page

Record the hypothesis, variable, control, unit and measure before you build anything. A one-page design stops the measure from drifting once you see data.

Name Your Experimental Unit

Decide whether the unit is a person, a session or a page, then randomize and analyze at that level so repeated measures aren't counted as separate people.

Pre-Test Before You Spend Real Traffic

Screen versions with a persona panel first, then send only the strongest into a live test with real users.

FAQ

Still Wondering?

Questions teams ask before designing their first experiment.

01What are the main types of experimental design?

Completely randomized and randomized block designs describe how units are randomized. Between-subjects and within-subjects designs describe how participants meet the conditions. Factorial designs change two or more variables together to test how they interact.

02What is the difference between between-subjects and within-subjects design?

In a between-subjects design each participant experiences one condition, which avoids learning effects but needs more people. In a within-subjects design each participant experiences every condition, which needs fewer people but requires counterbalancing so that order does not bias the result.

03What is an experimental unit?

The smallest thing that is assigned to a condition independently: one participant, one page visit, one classroom or one plot of land. Randomization and analysis both work at this level, so counting repeated measures from one unit as separate units overstates confidence.

04What is the difference between an experiment and a quasi-experiment?

An experiment assigns units to conditions at random. A quasi-experiment compares groups that already exist, so the groups may have differed before the change, and the result supports a weaker claim about cause.

05How many participants does an experiment need?

There is no single number. The count depends on the size of the effect you want to detect, how variable your measure is and how much false-positive risk you accept. Run a power calculation before the study, and add participants for any subgroup you want to report separately.

06Can I run an experiment before my product launches?

Yes, with participants who see prototypes or drafts instead of the live product. Recruiting them is the slow part, which is where Articos helps: AI personas stand in as participants and return a report in under 30 minutes. Treat that as a pre-launch read, then confirm important decisions with real users.

Get Your Experiment's Results in Under 30 Minutes

14-day trial · 1,000 credits · No credit card