What Is Experimental Design?
Experimental design is the plan that makes a test fair. See the types, key variables, a 120-vs-60 participant example and the mistakes that spoil results.
Experimental Design is the plan for a study that tests whether changing one thing causes a different result. It fixes what you change, what you hold constant, who is assigned to each condition and what you measure before any data comes in, so a difference in the outcome can be traced to the change.
Key takeaways
- A sound design has a variable you change on purpose, a group to compare against and random assignment of who sees what.
- The classic principles are randomization, replication and blocking. Ronald Fisher's 1926 and 1935 books helped establish them.
- Between-subjects designs give each participant one condition. Within-subjects designs give each participant every condition, which needs fewer people but a counterbalanced order.
- Without random assignment or a control group, a result shows a link between two things but can't show that one caused the other.
- Articos supplies participants and analysis for pre-launch tests in under 30 minutes. The design choices stay with you.
Explained
What Is Experimental Design?
Decide the comparison first. Collect the data second.
Experimental design is the structure of a study: which variable you change (the independent variable), which outcome you measure (the dependent variable), which conditions participants are assigned to and how everything else is kept the same. That structure is what lets you say the change caused the outcome, because other explanations have been designed out.
Studies that assign conditions at random are true experiments. Studies that compare groups that already exist, such as customers who picked different plans, are quasi-experiments, and their results carry more doubt because the groups may have differed before the study began. Observational research watches what happens without changing anything.
Why it matters
Why Experimental Design Matters
A number that moves after a change might have moved anyway. Design is what lets you attribute it.
Random assignment and a control group let you say the change produced the result. Without them you have a correlation, and the usual rival explanations stay open.
Groups formed by chance start out alike on average. A difference at the end can't be traced to who happened to land in which group.
A written design (variable, control, unit, measure) can be repeated by someone else next quarter, which is how results build on each other.
How it works
How Experimental Design Works, Step by Step
The method is a sequence of decisions. Each one is cheap to make before data exists and expensive to fix after.
- 1Plan
Name the Variables
List what you will change (the independent variable), what you will measure (the dependent variable) and what could interfere (confounding variables). Anything that could interfere is held constant or randomized.
- 2Plan
Write a Testable Hypothesis
"Naming the outcome in the headline will raise clarity ratings." One change, one expected effect and one measure that could prove you wrong.
- 3Plan
Set the Treatment Levels and a Control
Decide the versions of the independent variable, and include a control that gets the current version or no change at all.
- 4Build
Choose How to Assign Participants
Between-subjects gives each participant one condition. Within-subjects gives every participant all of them, in a randomized or reversed order (counterbalancing) so that order does not color the results. A randomized block design sorts participants by a shared trait first, then randomizes within each group.
- 5Run
Randomize, Run and Replicate
Assign at random, give every group the same instructions and conditions, and repeat the measurement or the whole study when you can. Replication separates a real effect from one lucky run.
- 6Decide
Analyze, Decide and Record
Compare the outcome across conditions with a method chosen in advance. Write down the hypothesis, the result and the decision so the next design starts from it.
Key elements
Key Elements and Types of Experimental Design
| Element | What it means | Example |
|---|---|---|
| Independent variable | The factor you change on purpose. | Headline wording |
| Dependent variable | The outcome you measure to see whether the change had an effect. | Clarity rating from 1 to 7 |
| Control group | The baseline group that gets the current version or no treatment. | The existing headline |
| Experimental unit | The smallest thing assigned to a condition on its own. | One participant |
| Random assignment | Every unit has the same chance of landing in any condition. | A random number drawn per participant |
| Replication | Repeating measurements or the whole study to separate real effects from noise. | Re-running with a new sample |
| Blocking | Grouping similar units first, then randomizing within each group, to cut known variation. | Block by new and returning customers |
Types of Experimental Design
| Design | How it works | Best for / Watch for |
|---|---|---|
| Completely randomized | Each unit is assigned to a condition at random. | Best for: simple comparisons with similar units. Watch for: chance can produce uneven groups in small samples. |
| Randomized block | Units are grouped by a shared trait, then randomized within each group. | Best for: samples where a known trait affects the outcome. Watch for: the trait has to be measured before assignment. |
| Between-subjects | Each participant sees one condition. | Best for: tasks where seeing one version would bias reactions to another. Watch for: needs more participants and has person-to-person noise. |
| Within-subjects | Each participant sees every condition. | Best for: small samples and side-by-side comparisons. Watch for: order and learning effects, so counterbalance. |
| Factorial | Two or more variables change together and every combination is tested. | Best for: finding interactions, such as headline and price. Watch for: the number of conditions multiplies quickly. |
Example
Experimental Design Example: Testing Two Headlines
An illustrative case: a team wants to know whether a headline that names the outcome is rated clearer than the current one. Both designs below test the same question.
Same question, same 60 ratings per headline. The between-subjects design needs 120 people and keeps every rating independent. The within-subjects design needs 60 people but has to counterbalance the order, because the first headline someone reads colors how they read the second. Neither design is better in general. The right choice depends on how many people you can reach and how much carryover the task creates.
When to use it
When Experimental Design Applies, and When It Doesn't
Use Experimental Design When
- You want to know whether a change causes an outcome
- You can control what each participant sees
- You can assign conditions at random
- A wrong decision costs more than the time to test
It Struggles When
- The groups already exist and can't be randomized, so the result is quasi-experimental
- The variable can't be manipulated, such as age or country
- The effect takes months to show up
- The test setting differs too much from real use
| Pros of Experimental Design | Cons of Experimental Design |
|---|---|
| Isolates cause and effect | Needs planning before any data exists |
| Can be repeated by someone else | Controlled settings can differ from real life |
| Makes results comparable across studies | Tests only the variables you thought to change |
| Settles disagreements with evidence | Needs enough participants to detect a realistic effect |
Common mistakes
Common Experimental Design Mistakes to Avoid
Changing More Than One Thing
If version B has a new headline and a new image, you can't say which one moved the result.
One variable per test, or plan a factorial design.Skipping Random Assignment
Letting people pick their version, or giving the new one to early adopters, builds the difference into the groups.
Assign by chance.Running Without a Control
Comparing this month with last month mixes your change with everything else that changed.
Run control and treatment at the same time.Ignoring Order Effects
In a within-subjects design, the first version people see shapes how they judge the second.
Counterbalance the order.Counting the Wrong Unit
Three answers from one person are not three independent observations, and treating them that way inflates confidence.
Analyze at the level you randomized.Choosing the Measure After the Fact
Picking whichever number moved turns noise into a finding.
Name the dependent variable in the hypothesis.How Articos helps
How to Run an Experiment Before Launch With Articos
Recruiting participants is the slowest part of most designs. Articos replaces that step for pre-launch tests, and you keep the design.
Write the design in the morning. Read the results by lunch.
Recruited participant study (6–8 weeks per cycle):
- Recruiting and scheduling set the timeline
- No-shows and cancellations shrink the sample
- Incentives and fees add cost to every study
With Articos:
You write the hypothesis, choose what changes and decide what stays fixed. A panel of AI personas grounded in behavioral science acts as the participants, each with a different stance toward your idea, and returns a report with scores, reasons and direct quotes in under 30 minutes. A study of two versions puts both in front of the same panel and compares them side by side. Articos tests how personas react, so confirm a high-stakes winner with real users or a live test.
Test two versions free →Putting it into practice
Putting Experimental Design Into Practice
Write the Design on One Page
Record the hypothesis, variable, control, unit and measure before you build anything. A one-page design stops the measure from drifting once you see data.
Name Your Experimental Unit
Decide whether the unit is a person, a session or a page, then randomize and analyze at that level so repeated measures aren't counted as separate people.
Pre-Test Before You Spend Real Traffic
Screen versions with a persona panel first, then send only the strongest into a live test with real users.
Keep learning
Related Terms and Guides
Go deeper on the ideas behind a reliable design.
FAQ
Still Wondering?
Questions teams ask before designing their first experiment.
01What are the main types of experimental design?
Completely randomized and randomized block designs describe how units are randomized. Between-subjects and within-subjects designs describe how participants meet the conditions. Factorial designs change two or more variables together to test how they interact.
02What is the difference between between-subjects and within-subjects design?
In a between-subjects design each participant experiences one condition, which avoids learning effects but needs more people. In a within-subjects design each participant experiences every condition, which needs fewer people but requires counterbalancing so that order does not bias the result.
03What is an experimental unit?
The smallest thing that is assigned to a condition independently: one participant, one page visit, one classroom or one plot of land. Randomization and analysis both work at this level, so counting repeated measures from one unit as separate units overstates confidence.
04What is the difference between an experiment and a quasi-experiment?
An experiment assigns units to conditions at random. A quasi-experiment compares groups that already exist, so the groups may have differed before the change, and the result supports a weaker claim about cause.
05How many participants does an experiment need?
There is no single number. The count depends on the size of the effect you want to detect, how variable your measure is and how much false-positive risk you accept. Run a power calculation before the study, and add participants for any subgroup you want to report separately.
06Can I run an experiment before my product launches?
Yes, with participants who see prototypes or drafts instead of the live product. Recruiting them is the slow part, which is where Articos helps: AI personas stand in as participants and return a report in under 30 minutes. Treat that as a pre-launch read, then confirm important decisions with real users.
Get Your Experiment's Results in Under 30 Minutes
14-day trial · 1,000 credits · No credit card