Metrics

When a number has moved

Shruwd calls a change movement only when it passes two gates and survives a correction for testing many prompts at once.

Two measurements will almost always differ. Most of that difference is sampling noise. Shruwd calls a change movement only when it passes both gates below.

Both gates must pass

Gate 1 — the difference is statistically distinguishable. The two periods are compared with a two-proportion test, and the 95% intervals must not overlap.

Gate 2 — the difference is big enough to care about. The change must be at least 5 percentage points.

Either gate alone gives the wrong answer. Gate 1 by itself will flag a change of half a percentage point once the sample is large. Gate 2 by itself will flag pure noise when the sample is small. Together they mean a reported movement is both real and worth your attention.

Testing many prompts at once

Checking dozens of prompts every week, a few will look significant by chance alone — forever, no matter how well the product works.

Shruwd applies a false-discovery-rate correction across every comparison in one evaluation, so the flagged movements are the ones that survive being one of many tests. The batch is one brand's comparisons; results are never pooled across brands.

The practical effect: a change you can see in the chart may not be labelled as movement. That is the correction doing its job.

When the engine changed underneath you

Providers update the models behind their answers. When that happens between two periods, the comparison is marked model changed.

The difference may be real, or it may be the new model behaving differently. Shruwd flags it rather than silently attributing it to something you did.

This same rule decides the verdict on a finding you have fixed, with the fix's prompts compared together as well as one by one. See Rechecks.

What this means in practice

  • Do not read week-to-week wobble as progress or decline. Look at the range, not the point.
  • A fix that produced no labelled movement has not been proven to work — that is honest, and it is the point of measuring.
  • Real movement usually needs a real change. Fixes that alter a single page rarely shift a prompt-set-wide number by 5 points.

Next steps