Fail Rate

Fail rate measures how often a test’s terminal status on a timeline is failing. A commit counts as a failure only if the test ended red after all retries.

Fail rate answers a different question than flip rate:

• Flip rate captures volatility — how often the test changes its outcome between commits.
• Fail rate captures failure frequency — what share of commits ended in failure.

Use fail rate to find tests that consistently fail (or fail often enough to be a problem) regardless of whether they also sometimes pass. Use flip rate to find tests that alternate between passing and failing.

Formula

For a given test in a timeline:

Fail Rate = Failures / Invocations

Where:

• Invocations is the number of commits where the test ran (any status other than skipped).
• Failures is the number of commits where the test’s status was failed or regressed.

If a test was never invoked in the window, fail rate is not defined.

Note

Failures are counted from the test’s commit-level status after retries. Flakiness.io first computes a test status per commit, and only commits whose terminal status is failed or regressed count as failures. A flaked commit (failed then passed on retry) counts as one invocation with zero failures, because the test eventually passed.

Examples

Commit history	Failures	Invocations	Fail Rate
✅ ✅ ✅ ✅ ✅	0	5	0%
❌ ❌ ❌ ❌ ❌	5	5	100%
✅ ✅ ✅ ❌ ❌	2	5	40%
✅ ❌ ✅ ❌ ✅	2	5	40%
✅ 🟡 ✅ ✅ ✅	0	5	0%
❌ 🟡 ❌ ✅ ❌	3	5	60%

(🟡 = flaked: failed then passed on retry within the same commit.)

Note that the third and fourth rows have the same fail rate (40%) but very different flip rates — fail rate ignores ordering, flip rate doesn’t.

Filtering by fail rate

Use the fail> operator in FQL to narrow tests by their fail rate:

fail>0%         # Tests that have failed at least once on the timeline
fail>50%        # Tests that fail more than half the time
fail<=25        # Tests with fail rate at most 25%