# Experiments

Experiments finds the tests hidden in your launches, ads in one campaign that differ in one thing, and calls each one when the data can.

![Experiments: the filters All 3, Collecting 0, Decided 0, No difference 1 and Stopped early 2, and the first experiments, a Hook test called No difference and a Landing Page test Stopped early](https://docs.omnymous.com/media/analytics/creatives/experiments/list.6b4b6b87.avif)

You never declare an experiment in Omnymous. Whenever ads inside one campaign differ in exactly one thing, such as their hook, their offer or the page they lead to, and agree on everything else, Omnymous treats them as a test and follows it. Open it under **Analytics**, then **Creatives**, then **Experiments**.

## How an experiment is found

An experiment is a set of ads in one platform campaign. Each group of ads that carries the same value of the varied dimension is an **arm**: one hook against another, one offer against another, or an atom against its absence. Everything else the ads carry is **held**: the same angle, avatar, product and format in every arm.

The arms are compared only over the days every arm delivered. The varied dimension decides what is measured: a hook test on CTR, an offer or a page test on orders per click.

Ads that differ in two things at once are never an experiment: no comparison could say which of the two made the difference. When choosing one thing fixes others, such as a landing page that brings its own product and angle, the experiment still counts, and its card names what moved with it.

## The list

![The status filters All 3, Collecting 0, Decided 0, No difference 1 and Stopped early 2, the dimension filters Hook, Landing Page and Offer, and the Derive button](https://docs.omnymous.com/media/analytics/creatives/experiments/filters.73a1bb6b.avif)

The first row filters by status, with how many experiments have each. The second filters by the dimension that was varied. **Derive** looks for experiments again now; Omnymous also does it every time it recomputes your analytics.

### Statuses

| Name              | What it does                                                                                                                                                           |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Collecting**    | Every arm is still delivering, and the arms cannot be told apart yet.                                                                                                  |
| **Decided**       | The test ruled out "no difference" between the two best arms at the 5% level, after at least 3 days of overlapping delivery. Checking it every day does not weaken it. |
| **No difference** | The arms are equivalent: the interval of their ratio sits within 10% either side of equal. More spend will not change the answer.                                      |
| **Stopped early** | Delivery stopped more than 7 days ago, before the arms could be told apart. It is the most common way a test fails.                                                    |

## An experiment

![A Landing Page test, at the conversion stage on orders per click: two pages, the Omega-3 report at 3.9% and the Focus Stack page at 2.2%, Stopped early, with what was held and the campaign it ran in](https://docs.omnymous.com/media/analytics/creatives/experiments/card.611d942f.avif)

Each card names the dimension tested, the stage and the measure, and its status. Each arm shows its ads, its rate and how much it delivered. A sentence says what the data show, such as which arm led and how sure that was when delivery stopped.

**Held** lists what every arm had in common. The last line says where the test ran: the campaign, whether the arms shared an ad set and so one audience, whether they were launched from Omnymous, the days compared, and the spend.

| Name                               | What it does                                                                                                        |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **One audience (one ad set)**      | Every arm ran in one ad set, to the same people.                                                                    |
| **Same targeting in every ad set** | Each arm had an ad set of its own, and the ad sets targeted the same people.                                        |
| **Audiences not verified**         | The arms ran in ad sets whose targeting Omnymous cannot see. The result holds only if they reached the same people. |

## Inside an experiment

Press a card to open it on the right.

![The Landing Page test opened: Stopped early, the verdict on orders per click with the winner 1.74× the runner-up, a 90% interval of 1.32–2.28× and P(winner is better) over 99%, and each arm with its rate, spend, impressions, clicks, sessions, orders and ROAS](https://docs.omnymous.com/media/analytics/creatives/experiments/rail.4da87974.avif)

**Verdict** compares the leading arm with the runner-up: the ratio of their rates, its 90% interval, and the probability that the leader is truly better. Under them, the test's own figure and the bar it has to clear. **Arms** gives each arm's delivery and money, then its creative.

Further down: the daily rate of each arm, everything **Held constant**, what **Moved with it**, and the **Model card**. When an experiment is decided, **Next batch** proposes the untried pairings next to the winner, with what each is expected to bring and a button that opens Studio on them.

## Questions

### Why is a test I ran missing?

Its ads differed in more than one thing, or never delivered on the same days, or ran in different campaigns. To make a test Omnymous can read, put the ads in one campaign, ideally one ad set, and change one thing.

### Why does a test stay Collecting so long?

Small differences need a lot of delivery to tell apart, and the test is strict so that checking it every day does not call a winner by chance.

### Can a decided test change its mind?

No. Once decided, an experiment keeps its verdict. Its figures keep updating as the ads run.

---

Source: https://docs.omnymous.com/analytics/creatives/experiments