# When a number has moved

Two measurements will almost always differ. Most of that difference is sampling noise.
Shruwd calls a change **movement** only when it passes both gates below.

## Both gates must pass

**Gate 1 — the difference is statistically distinguishable.** The two periods are
compared with a two-proportion test, and the 95% intervals must not overlap.

**Gate 2 — the difference is big enough to care about.** The change must be at least
**5 percentage points**.

Either gate alone gives the wrong answer. Gate 1 by itself will flag a change of half a
percentage point once the sample is large. Gate 2 by itself will flag pure noise when the
sample is small. Together they mean a reported movement is both real and worth your
attention.

## Testing many prompts at once

Checking dozens of prompts every week, a few will look significant by chance alone —
forever, no matter how well the product works.

Shruwd applies a false-discovery-rate correction across every comparison in one
evaluation, so the flagged movements are the ones that survive being one of many tests.
The batch is one brand's comparisons; results are never pooled across brands.

The practical effect: a change you can see in the chart may not be labelled as movement.
That is the correction doing its job.

## When the engine changed underneath you

Providers update the models behind their answers. When that happens between two periods,
the comparison is marked **model changed**.

The difference may be real, or it may be the new model behaving differently. Shruwd
flags it rather than silently attributing it to something you did.

> **Note**
>
> This same rule decides the verdict on a finding you have fixed, with the fix's prompts
> compared together as well as one by one. See [Rechecks](https://shruwd.io/docs/findings/rechecks.md).

## What this means in practice

- Do not read week-to-week wobble as progress or decline. Look at the range, not the point.
- A fix that produced no labelled movement has not been proven to work — that is honest,
  and it is the point of measuring.
- Real movement usually needs a real change. Fixes that alter a single page rarely shift a
  prompt-set-wide number by 5 points.

## Next steps

- [Confidence and sample size](https://shruwd.io/docs/metrics/confidence-and-sample-size.md) — where the ranges come from
- [Rechecks](https://shruwd.io/docs/findings/rechecks.md) — proving a specific fix worked