Getting Started

You'll need

• a GitHub monorepo — tests and code live in the same repository. Flakiness.io keys results by commit, so test results and the code that produced them must share the same git history.
• a GitHub Actions workflow that runs your tests.

You'll have

• your first report visible on the Flakiness.io dashboard, with a GitHub check posted on new pull requests.

1. Sign in to Flakiness.io via GitHub.

   Note about “Act on your behalf”

   The Flakiness.io GitHub App only asks for read-only permissions. There is a long thread on GitHub’s community forum discussing the “Act on your behalf” wording here.

   Sign in

2. Create an Organization.

   Organization

   An Organization is a workspace that contains projects. Members share access control and billing. Think of it as your team or company account. See Access Control.

   Create Organization

3. Install the Flakiness.io GitHub App. The App is the bridge between your repository and Flakiness.io: it grants read-only access to commit history and permission to post PR checks.

   Install the GitHub App

   Installing alone does nothing

   Installing the App is just the first half — the repository isn’t actually connected to Flakiness.io yet. You’ll wire it up in the next step by creating a Project and linking it to this repo.

4. Create a Project inside the organization, linked to your GitHub repository.

   Project

   A Project maps 1:1 to a GitHub repository. It owns the test history, analytics, and settings for that repo.

5. Copy your flakinessProject identifier. Open your project page on Flakiness.io; the URL looks like https://flakiness.io/<orgSlug>/<projectSlug>. Your identifier is that <orgSlug>/<projectSlug> pair — you’ll paste it into the reporter config in the next step.

   What is flakinessProject?

   The orgSlug/projectSlug identifier tells the reporter which project to upload results to. For example, a project at https://flakiness.io/my-org/my-app has flakinessProject: 'my-org/my-app'.

6. Upload test results from CI. Pick your runner — each tile shows the install, config, and CI integration you need.

   Playwright Pytest Vitest Jest CucumberJS Bun Rust JUnit XML C Custom

   Install the reporter:

   npm install -D @flakiness/playwright

   Add it to your config:

   playwright.config.ts

   import { defineConfig } from '@playwright/test';
   
   export default defineConfig({
     reporter: [['@flakiness/playwright', {
       flakinessProject: 'my-org/my-app'
     }]],
   });

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Playwright Test supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx playwright install --with-deps
         - run: npx playwright test

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Expose an access token via FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx playwright test

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   Install the reporter:

   uv add --dev pytest-flakiness
   # or: pip install pytest-flakiness

   The reporter auto-activates once installed.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission and specify a FLAKINESS_PROJECT to upload the report to: Pytest supports Github OIDC for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       env:
         FLAKINESS_PROJECT: my-org/my-app
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with:
             python-version: '3.12'
         - run: pip install -e '.[test]'
         - run: pytest

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> pytest

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   Install the reporter:

   npm install -D @flakiness/vitest

   Add it to your config:

   vitest.config.ts

   import { defineConfig } from 'vitest/config';
   import flakinessReporter from '@flakiness/vitest';
   
   export default defineConfig({
     test: {
       reporters: ['default', flakinessReporter({
         flakinessProject: 'my-org/my-app'
       })],
     },
   });

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Vitest supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx vitest run

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx vitest run

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   Install the reporter:

   npm install -D @flakiness/jest

   Add it to your config:

   jest.config.ts

   import type { Config } from 'jest';
   
   const config: Config = {
     testLocationInResults: true,
     reporters: [
       'default',
       ['@flakiness/jest', { flakinessProject: 'my-org/my-app' }],
     ],
   };
   
   export default config;

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: Jest supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx jest

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx jest

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   Install the reporter:

   npm install -D @flakiness/cucumberjs

   Add it to your config:

   cucumber.mjs

   export default {
     default: {
       format: ['@flakiness/cucumberjs'],
       formatOptions: {
         flakinessProject: 'my-org/my-app',
       },
     },
   };

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: CucumberJS supports GitHub OIDC and uses flakinessProject for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: '20'
         - run: npm ci
         - run: npx cucumber-js

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Set FLAKINESS_ACCESS_TOKEN (from your project’s settings) and run your tests:

   FLAKINESS_ACCESS_TOKEN=<token> npx cucumber-js

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   bun test has built-in JUnit XML output. Run it with --reporter=junit, then use the Flakiness CLI to convert and upload the report. See Ingesting JUnit XML for the full ingest flow.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the flakiness upload step uses GitHub OIDC for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: oven-sh/setup-bun@v2
         - run: bun install
         - run: bun test --reporter=junit --reporter-outfile=./junit.xml
         - name: Convert and upload to Flakiness.io
           if: always()
           run: |
             curl -LsSf https://cli.flakiness.io/install.sh | sh
             flakiness convert-junit ./junit.xml --category bun --project my-org/my-app
             flakiness upload ./flakiness-report/report.json

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Install the CLI, set FLAKINESS_ACCESS_TOKEN (from your project’s settings), run your tests with the JUnit reporter, then convert and upload:

   curl -LsSf https://cli.flakiness.io/install.sh | sh
   bun test --reporter=junit --reporter-outfile=./junit.xml
   flakiness convert-junit ./junit.xml --category bun --project my-org/my-app
   FLAKINESS_ACCESS_TOKEN=<token> flakiness upload ./flakiness-report/report.json

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   Rust doesn’t have a native Flakiness reporter. The simplest path is cargo-nextest, a drop-in cargo test replacement that emits JUnit XML.

   Adopt cargo-nextest. If your project doesn’t already use it, install it locally and switch your test command from cargo test to cargo nextest run. See the nextest installation guide for the recommended install for your OS.

   Configure the ci profile to emit JUnit XML:

   .config/nextest.toml

   [profile.ci.junit]
   path = "junit.xml"

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the flakiness upload step uses GitHub OIDC for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         - uses: dtolnay/rust-toolchain@stable
         - run: cargo nextest run --profile ci
         - name: Convert and upload to Flakiness.io
           if: always()
           run: |
             curl -LsSf https://cli.flakiness.io/install.sh | sh
             flakiness convert-junit ./target/nextest/ci/junit.xml --category rust --project my-org/my-app
             flakiness upload ./flakiness-report/report.json

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Install the CLI, set FLAKINESS_ACCESS_TOKEN (from your project’s settings), run tests with cargo-nextest, then convert and upload:

   curl -LsSf https://cli.flakiness.io/install.sh | sh
   cargo nextest run --profile ci
   flakiness convert-junit ./target/nextest/ci/junit.xml --category rust --project my-org/my-app
   FLAKINESS_ACCESS_TOKEN=<token> flakiness upload ./flakiness-report/report.json

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   For runners without a native reporter, produce JUnit XML, then use the Flakiness CLI to convert and upload it. See Ingesting JUnit XML for the full ingest flow.

   Configure your CI:

   GitHub Actions

   Grant the id-token: write permission: the flakiness upload step uses GitHub OIDC for secretless authentication.

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read       # for actions/checkout
         id-token: write      # for Flakiness.io OIDC
       steps:
         - uses: actions/checkout@v4
         # ... your existing test steps, producing JUnit XML in ./test-results ...
         - name: Convert and upload to Flakiness.io
           if: always()
           run: |
             curl -LsSf https://cli.flakiness.io/install.sh | sh
             flakiness convert-junit ./test-results --project my-org/my-app
             flakiness upload ./flakiness-report/report.json

   For fork PRs or more advanced cases, see GitHub Actions Setup and GitHub Actions for Open Source Projects.

   Other CI

   Install the CLI, set FLAKINESS_ACCESS_TOKEN (from your project’s settings), run your tests, then convert and upload:

   curl -LsSf https://cli.flakiness.io/install.sh | sh
   <your test command producing JUnit XML in ./test-results>
   flakiness convert-junit ./test-results --project my-org/my-app
   FLAKINESS_ACCESS_TOKEN=<token> flakiness upload ./flakiness-report/report.json

   See CI Configuration (non-GitHub) for per-provider details (GitLab CI, CircleCI, Jenkins, etc.).

   No official reporter for your runner? You have three options — see Custom Integration.

7. Push a commit and open the dashboard. Your first report should appear within seconds of the test job finishing.

   Tip

   If the upload didn’t happen, check your CI logs. The reporter prints a confirmation line on success ([flakiness.io] ✓ Uploaded as ...) and a descriptive error on failure.

Next steps

• Pull Requests & Regressions — interpret the PR check and decide mergeability.
• Slack Notifications — daily alerts on failures and regressions.
• Sharded Tests — configure CI matrix sharding.
• AI Agents — let Claude Code / Codex / Cursor investigate flaky tests.