Experiments
Experiments finds the tests hidden in your launches, ads in one campaign that differ in one thing, and calls each one when the data can.
You never declare an experiment in Omnymous. Whenever ads inside one campaign differ in exactly one thing, such as their hook, their offer or the page they lead to, and agree on everything else, Omnymous treats them as a test and follows it. Open it under Analytics, then Creatives, then Experiments.
How an experiment is found
An experiment is a set of ads in one platform campaign. Each group of ads that carries the same value of the varied dimension is an arm: one hook against another, one offer against another, or an atom against its absence. Everything else the ads carry is held: the same angle, avatar, product and format in every arm.
The arms are compared only over the days every arm delivered. The varied dimension decides what is measured: a hook test on CTR, an offer or a page test on orders per click.
Ads that differ in two things at once are never an experiment: no comparison could say which of the two made the difference. When choosing one thing fixes others, such as a landing page that brings its own product and angle, the experiment still counts, and its card names what moved with it.
The list
The first row filters by status, with how many experiments have each. The second filters by the dimension that was varied. Derive looks for experiments again now; Omnymous also does it every time it recomputes your analytics.
Statuses
| Name | What it does |
|---|---|
| Collecting | Every arm is still delivering, and the arms cannot be told apart yet. |
| Decided | The test ruled out "no difference" between the two best arms at the 5% level, after at least 3 days of overlapping delivery. Checking it every day does not weaken it. |
| No difference | The arms are equivalent: the interval of their ratio sits within 10% either side of equal. More spend will not change the answer. |
| Stopped early | Delivery stopped more than 7 days ago, before the arms could be told apart. It is the most common way a test fails. |
An experiment
Each card names the dimension tested, the stage and the measure, and its status. Each arm shows its ads, its rate and how much it delivered. A sentence says what the data show, such as which arm led and how sure that was when delivery stopped.
Held lists what every arm had in common. The last line says where the test ran: the campaign, whether the arms shared an ad set and so one audience, whether they were launched from Omnymous, the days compared, and the spend.
| Name | What it does |
|---|---|
| One audience (one ad set) | Every arm ran in one ad set, to the same people. |
| Same targeting in every ad set | Each arm had an ad set of its own, and the ad sets targeted the same people. |
| Audiences not verified | The arms ran in ad sets whose targeting Omnymous cannot see. The result holds only if they reached the same people. |
Inside an experiment
Press a card to open it on the right.
Verdict compares the leading arm with the runner-up: the ratio of their rates, its 90% interval, and the probability that the leader is truly better. Under them, the test's own figure and the bar it has to clear. Arms gives each arm's delivery and money, then its creative.
Further down: the daily rate of each arm, everything Held constant, what Moved with it, and the Model card. When an experiment is decided, Next batch proposes the untried pairings next to the winner, with what each is expected to bring and a button that opens Studio on them.