# How to Run A/B Tests When Your Product Has Low Traffic
> Run A/B tests on low traffic: test big swings, one metric, clean before/after windows, 5-user sessions, fake doors and a simple log.

```yaml
url: "https://makerhunt.io/studio/how-to-run-ab-tests-when-your-product-has-low-traffic"
markdown: "https://makerhunt.io/studio/how-to-run-ab-tests-when-your-product-has-low-traffic.md"
title: How to Run A/B Tests When Your Product Has Low Traffic
type: studio_article
category: Growth
tags: "A/B testing, low traffic, experiments, conversion, fake door test, user testing, makers, Growth"
featured_image: "https://txmhk1zrnc.ufs.sh/f/xSkWTCqmKWx9rqFO472TCM8expg3JaYRjSXLVoZNbv9Df0wE"
published_at: "2026-10-09T09:52:19.413Z"
updated_at: "2026-10-09T09:52:19.976Z"
```

## About this studio article

Run A/B tests on low traffic: test big swings, one metric, clean before/after windows, 5-user sessions, fake doors and a simple log.

## Article

If your product gets a few hundred visitors a week, a classic A/B test will usually take months to give you a trustworthy answer, so stop splitting traffic on small tweaks. Instead, test big changes one at a time, judge them on a single metric close to money or activation, compare clean before-and-after windows with guardrails, and back every decision with five quick user sessions. That is how you run A/B tests when your product has low traffic: fewer, bolder experiments, written down, with a stop date set before you start.

This guide is for makers with somewhere between 100 and 1,000 visitors a week. You will get the math in plain words, the methods that work at your size, a simple test log, a made-up example product, and a 14-day plan you can start on Monday.

## Why do classic A/B tests fail at low traffic?

A normal A/B test splits visitors into two groups, shows each group a different version, and waits until the difference in results is big enough that chance is an unlikely explanation. The waiting is the problem. How long you wait depends on three things: your current conversion rate, the size of the lift you want to detect, and how many visitors you get.

Here is a rough worked example. I ran the numbers with the standard two-proportion sample size formula, at the usual 5% significance level and 80% power, and you can reproduce them in [Evan Miller's free A/B test sample size calculator](https://www.evanmiller.org/ab-testing/sample-size.html). The numbers are approximate and the traffic figure is just an example:

Baseline signup rate

Lift you want to detect

Visitors needed per version (approx.)

Weeks at 400 visitors/week split 50/50

3%

20% relative (3% to 3.6%)

about 13,900

about 70

3%

50% relative (3% to 4.5%)

about 2,500

about 13

3%

100% relative (3% to 6%)

about 750

about 4

Read that table twice. A button colour change that might lift signups by a few percent would need more visitors than many small products see in a year. Only very large effects can be detected in a few weeks. That single fact should shape everything you test.

There is a second trap. If you keep checking a running test and stop the moment the dashboard says "significant", the result is far less reliable than it looks. Evan Miller explains this in [How Not To Run an A/B Test](https://www.evanmiller.org/how-not-to-run-an-ab-test.html): the significance math assumes the sample size was fixed in advance, and repeated peeking raises the rate of false positives. In one of his simulated scenarios, checking after every observation pushed the false positive rate to 26.1% instead of the expected 5%. Small products are especially tempted to peek, because every signup feels like news.

## So what should you test when traffic is small?

Test big swings, not polish. A big swing is a change that a visitor would notice in the first five seconds and that could plausibly double or halve your result. Examples:

- **A different promise in the headline.** "Track your invoices" versus "Get paid 2 weeks faster" is a real swing. Changing "Start free" to "Try free" is not.
- **A different audience on the page.** Rewriting the whole page for one specific buyer instead of "teams and individuals".
- **A different first step.** Letting people try the core action before creating an account, versus asking for an email first.
- **Removing a whole section or step.** Cutting a signup form from six fields to two, or dropping a setup screen.
- **Adding proof of how it works.** A short demo near the top. If you go this route, here is a guide on [filming a 90-second product demo for cold visitors](https://makerhunt.io/studio/how-to-film-a-90-second-product-demo-that-converts).

Skip button colours, icon swaps, word-level copy edits and font changes until you have far more traffic. They are not useless, you just cannot measure them yet, so they waste your test slots. Pricing changes deserve their own careful process, so leave them out of this weekly rhythm.

A rule of thumb I use (an opinion, not a law): if you cannot explain in one sentence why the change might move the number a lot, it is too small to test right now. Ship it if you like it, but do not call it an experiment.

## Which metric should a low-traffic test use?

Pick one primary metric, and pick it as close to revenue or real usage as your volume allows. Page clicks are easy to count but often lie. A headline that promises more can win clicks and lose real users.

A useful order to choose from, from best to weakest:

1. **Paid conversions**, if you get enough of them to see a pattern within a month.
2. **Activation**: the share of new signups who complete the core action (sent the first invoice, published the first page, booked the first class).
3. **Signups**, if activation is still too rare to count.
4. **Clicks on the main call to action**, only as a last resort, and only for top-of-page changes.

Write down how the metric is calculated before you start, so you cannot quietly change the definition later. If you have not set up basic tracking yet, start with this guide to [tracking product metrics before your first 100 users](https://indiehunt.io/studio/how-to-track-product-metrics-before-100-users), then come back.

Also pick one or two **guardrails**: numbers that must not get worse. Common ones are refund requests, support messages per new user, and the share of signups who never come back after day one. A change that lifts signups but doubles confused support emails is not a win.

## How do you run a before-and-after test without fooling yourself?

When you cannot split traffic, run a sequential test: measure a "before" window, ship the change, then measure an "after" window of the same length. It is weaker than a true split test because other things change over time too, but you can make it much more honest with a few rules.

### Rules for a clean sequential test

- **Fix the windows in advance.** For example, two full weeks before and two full weeks after, Monday to Sunday, so weekday patterns cancel out.
- **Change one thing.** If you also launched on a directory, sent a newsletter, or fixed a big bug, the test is muddied. Note it in the log and extend or restart.
- **Compare by traffic source.** Look at the metric separately for search, direct and referral traffic. If the lift only appears in one source and that source had a spike, be careful.
- **Set your decision rule before you look.** For example: "Keep the change if activation is clearly higher and no guardrail is worse. Revert if activation is lower. If it is roughly the same, keep whichever version is simpler."
- **Do not check the result daily.** Check traffic is flowing and nothing is broken, but read the metric only when the window ends.

"Clearly higher" is a judgment call at small numbers. My rough rule (again, an opinion): with only dozens of conversions per window, treat anything under a 30 to 50% relative change as noise unless the five-user sessions below point the same way.

### When a real split test still makes sense

If a change is huge (a new first step, a new page for a new audience) and you have a few hundred visitors a week, a 50/50 split can still work. Use the sample size calculator before starting, write the required number of visitors in your log, and run until you hit it, even if the early numbers look exciting. If the calculator says 40 weeks, do not run it. Use the sequential method instead.

![image](https://txmhk1zrnc.ufs.sh/f/xSkWTCqmKWx93PVoYx1duKC2cTLjUHtkW4FmRlen8BZa9JpY)

## Why do 5-user tests matter more than your dashboard?

Numbers tell you whether something changed. Watching people tells you why. At low traffic, the "why" is usually more valuable, and it is far cheaper to get.

Jakob Nielsen's well-known article [Why You Only Need to Test with 5 Users](https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/) (Nielsen Norman Group, 2000) argues that a small qualitative test with five people uncovers most usability problems, and that it is better to run several small rounds than one large study. It is about usability problems, not conversion rates, so do not use five people to declare a winner. Use them to find out whether a change is worth testing at all, and to explain the result afterwards.

### How to run one in an afternoon

1. Find five people who match your buyer. Recent signups, people in a community you are part of, or makers who already gave you feedback. If you need a fresh source, here is how to [get brutal feedback from other makers](https://makerhunt.io/studio/how-to-get-brutal-feedback-from-makers-before-you-launch-publicly).
2. Book 20-minute video calls. Ask them to share their screen.
3. Give one task: "You just found this page from a search. Figure out if it is for you, and if so, get started."
4. Stay quiet. Ask "What are you thinking right now?" when they pause. Do not explain the product.
5. Write down every moment of hesitation, every misread, and the exact words they use to describe what the product does.

Run it on the current version before a test and on the new version after. If three or more people stumble on the same thing, that is your next big swing.

## What is a fake-door test and when should you use one?

A fake-door test (also called a painted-door test) measures demand for something before you build it. You add a button, menu item or plan option for a feature that does not exist yet. When someone clicks it, they see a short, honest message: "This is not built yet. Want us to tell you when it is?" with an email field or a one-question form.

It works well at low traffic because you are not trying to detect a small lift. You are asking a yes-or-no question: does anyone care enough to click? A few dozen clicks from the right users can be enough to decide.

### Rules that keep fake doors fair

- **Be honest immediately.** The message right after the click must say the feature does not exist yet. Never take payment for it.
- **Keep it small.** One fake door at a time, removed within two to three weeks.
- **Follow up.** Email everyone who left their address with one question about what they hoped it would do. Those replies are often worth more than the click count.
- **Do not stack them.** A product full of buttons that lead nowhere feels broken and costs you trust.

## How often should you ship a change?

One change per week, or one per two weeks if your traffic is very low. That rhythm does three things: it keeps each window clean, it stops you from piling five changes on top of each other, and it gives you a steady list of learnings instead of a vague memory of "we tried some stuff".

Keep a short queue of the next three ideas, ranked by how big the swing is and how cheap it is to ship. When a test ends, the next one starts the following Monday.

## What should go in a simple test log?

A spreadsheet is enough. One row per test, with these columns:

Column

What to write

ID and name

T07: "Try before signup"

Hypothesis

"Letting visitors build one schedule before creating an account will raise activation, because people can see the result first."

Method

Sequential (2 weeks before / 2 weeks after), split test, fake door, or 5-user session

Primary metric

Exact definition, for example "signups who publish a schedule within 7 days"

Guardrails

Support messages per new user; day-one return rate

Dates

Start date and end date, written before starting

Decision rule

What result means keep, revert, or keep the simpler one

Other events

Launches, outages, newsletter mentions during the window

Result

Before and after numbers, plus notes from user sessions

Decision and next step

Kept / reverted / follow-up test

The "Other events" column is the one people skip and later regret. At low traffic, a single mention in a newsletter can swamp any change you made.

![image](https://txmhk1zrnc.ufs.sh/f/xSkWTCqmKWx910jJWCb54BKHuiYIbGfnwyaSkxvDTCMsrW6Q)

## When should you stop a test?

Stop on the end date you wrote down. Not earlier because it looks good, not later because you hope it will turn around. There are only three good reasons to end early:

- **Something is broken.** Signups drop to zero, a form errors out, payments fail. Revert immediately and log it.
- **A guardrail goes clearly bad.** For example, a flood of confused support messages from new users.
- **Outside traffic swamps the window.** A big launch or press spike makes the window useless. Note it and restart later.

If the result is flat, that is still a result. Keep the simpler version, write down what you learned, and move to a bigger swing. Most tests at small scale end flat. That is normal and it tells you the change was not big enough to matter yet.

## What does this look like in practice? (Invented example)

Everything in this example, including the product, the people and the numbers, is made up to show the method. It is not a real case study.

**Kilnwise** is an imaginary scheduling tool for small pottery studios that run weekly classes. Its maker gets around 350 visitors a week, mostly from search and a couple of craft forums. About 12 people sign up each week, and about 4 of them publish their first class schedule within 7 days.

The maker wanted to test a green "Start free" button against a blue one. The sample size calculator made it obvious that such a small change would need many months of traffic, so the idea went into a "later" list.

Instead, the maker ran five 20-minute sessions with studio owners from a forum. Four of the five said, in different words, "I don't know if this handles drop-in students." That sentence became the first test.

- **T01 (sequential, 2 weeks each):** new headline and first screen built around "classes and drop-ins in one calendar", with a screenshot of exactly that. Primary metric: signups who publish a schedule within 7 days. Guardrail: support messages per new user. Before: 8 activated out of 25 signups. After: 15 activated out of 27. No launches during either window. Support messages were about the same. Decision: keep, then confirm with three more user sessions, which no longer raised the drop-in question.
- **T02 (fake door, 2 weeks):** a "Sell class packs" item in the menu. 19 clicks, 11 emails left, 6 replies to the follow-up question. Four of the six wanted something different from what the maker had imagined (gift vouchers, not packs). Decision: do not build packs yet; explore vouchers.
- **T03 (sequential):** cut the signup form from five fields to email and password. Result: about the same activation. Decision: keep the shorter form because it is simpler, and log it as flat.

Three tests, six weeks, one clear win, one avoided build, one flat result. That is a good month at this scale.

## What are the common traps?

- **Testing tiny things because they are easy.** Easy to ship, impossible to measure.
- **Peeking and stopping early.** The early numbers will swing around. Wait for the end date.
- **Changing several things at once.** You will not know which change did it.
- **Using a vanity metric.** Clicks and time on page can rise while real users fall.
- **Ignoring outside events.** One newsletter mention can look like a winning test.
- **Trusting a tool's "95% chance to win" badge at tiny volumes.** Check how many conversions sit behind it. If it is a handful, treat it as a hint.
- **Never writing it down.** Without a log, you will rerun the same failed ideas six months later.
- **Leaving fake doors up forever.** They turn into broken promises.

## Your 14-day plan

1. **Day 1:** Pick your primary metric and one or two guardrails. Write the exact definitions.
2. **Day 2:** Check that the metric is tracked correctly. Pull the last four weeks so you know your normal weekly range.
3. **Day 3:** Create the test log spreadsheet with the columns above.
4. **Days 4 to 5:** Run five 20-minute user sessions on your current page or first-run flow. Write down every stumble.
5. **Day 6:** Turn the most common stumble into one big-swing hypothesis. Put two more ideas in the queue.
6. **Day 7:** Plug your numbers into a sample size calculator. Decide: split test or sequential. Write the dates and decision rule.
7. **Day 8:** Ship the change on a Monday, or start the split test. Note the start in the log.
8. **Days 9 to 13:** Do not read the metric. Check only that nothing is broken. Log any outside events. Optionally set up one fake door for a feature you are unsure about.
9. **Day 14:** Review the fake door clicks and follow-up replies, confirm the main test end date, and line up the next test so it starts the Monday after this one ends.

## FAQ

### How much traffic do I need to run an A/B test?

It depends on your conversion rate and how big a change you want to detect. Put your own numbers into a sample size calculator before starting. As a rough guide from the table above, detecting a small lift at a low conversion rate can take tens of thousands of visitors per version, while a very large lift may need only hundreds.

### Are A/B testing tools worth it at low traffic?

Often not yet. A feature flag or a simple code switch, plus your analytics and a spreadsheet, covers sequential tests and the occasional large split test. Paid tools are more useful once you have enough traffic to test smaller changes.

### Can I trust a before-and-after comparison?

Less than a proper split test, but much more than guessing, if you fix the windows in advance, change one thing, log outside events, and confirm the result with user sessions.

### Is a fake-door test dishonest?

Not if you are upfront the moment someone clicks, never take money, and remove it within a few weeks. If it would embarrass you to explain it to a user, do not run it.

### How long should each test run?

At least one full week, and usually two, so that weekday and weekend patterns balance out. Set the length before you start and stick to it.

## Where to go from here

At low traffic, experiments are less about statistics and more about discipline: big changes, one metric, fixed dates, a written log and real conversations. Do that for a couple of months and you will know your users better than most teams with ten times your traffic.

And if you want more people to try your product while you test, list it on [MakerHunt](https://makerhunt.io), where makers share what they are building and get early users who actually give feedback.

## Links

- Article (HTML): https://makerhunt.io/studio/how-to-run-ab-tests-when-your-product-has-low-traffic
- AI-friendly Markdown: https://makerhunt.io/studio/how-to-run-ab-tests-when-your-product-has-low-traffic.md

## Explore

- [Studio index](https://makerhunt.io/studio)
- [Blog](https://makerhunt.io/blog)
- [Browse projects](https://makerhunt.io/projects)

_HTML version: https://makerhunt.io/studio/how-to-run-ab-tests-when-your-product-has-low-traffic_