GitHub Actions

There are two ways to authenticate Flakiness.io uploads in GitHub Actions: GitHub OIDC (recommended) and Access Token.

GitHub OIDC (Recommended)

GitHub OIDC lets your workflow upload reports to Flakiness.io without storing any secrets. The workflow requests a short-lived token from GitHub and exchanges it with Flakiness.io automatically.

Prerequisites

• A reporter or tool that supports OIDC — all official reporters and the Flakiness CLI support OIDC.
• The flakinessProject option configured in your reporter (format: orgSlug/projectSlug).

Setup

1. Configure flakinessProject in your reporter so it knows which project to upload to. The value is orgSlug/projectSlug. Refer to your test runner’s documentation for the exact configuration syntax.

   Example

   For a project hosted on https://flakiness.io/flakiness/pytest-flakiness, the flakinessProject value would be flakiness/pytest-flakiness.

2. Add OIDC permissions to your workflow:

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       permissions:
         contents: read # Required for actions/checkout
         id-token: write # Required for GitHub OIDC
       steps:
         # ... setup steps

That’s it — no secrets to create or rotate. The reporter handles token exchange automatically.

Note

All official reporters and the Flakiness CLI support OIDC. Refer to your test runner’s documentation for configuration details.

Access Token

For reporters that don’t yet support OIDC, or as a fallback, you can use a traditional secret-based approach.

1. Navigate to your GitHub repository Settings → Secrets and variables → Actions.

2. Click New repository secret, name it FLAKINESS_ACCESS_TOKEN, and paste the access token from your Flakiness.io project settings.

3. In your workflow file, pass the secret as an environment variable to the step that runs tests:

   .github/workflows/tests.yml

   jobs:
     test:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         # ... setup steps (Node.js, Python, etc.)
         - name: Run tests
          env:
            FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
           run: npx playwright test   # or pytest, etc.

Pull Requests from Forks

GitHub Actions does not expose repository secrets or OIDC tokens to workflows triggered by fork pull requests (pull_request event). This means the reporter cannot upload directly from a fork PR workflow, regardless of which auth method you use.

The workaround uses two workflows:

1. The PR workflow runs tests and uploads the report folder as a GitHub artifact.
2. A trusted workflow in the base repository, triggered by workflow_run, downloads that artifact and uploads it to Flakiness.io.

Step 1 — Save the report as an artifact

Add an upload-artifact step at the end of your existing PR workflow:

.github/workflows/tests.yml

# ... your regular test steps ...
- name: Upload Flakiness report artifact (fork PRs only)
  if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
  uses: actions/upload-artifact@v4
  with:
    name: flakiness-report-${{ github.job }}-${{ strategy.job-index }}
    path: flakiness-report/
    retention-days: 1

Tip

The artifact name includes ${{ github.job }} and ${{ strategy.job-index }} so that matrix builds can upload one artifact per job without name conflicts.

Step 2 — Create the upload workflow

Create a new workflow that runs in the context of the base repository (where secrets and OIDC are available).

Using GitHub OIDC (no secrets required — the report must contain flakinessProject):

.github/workflows/flakiness-upload-fork-prs.yml

name: Upload Flakiness.io report (fork PRs)
on:
  workflow_run:
    # Must match the name(s) of workflows that produce flakiness-report artifacts
    workflows: ["Tests"]
    types: [completed]

jobs:
  upload-flakiness-report:
    runs-on: ubuntu-latest
    if: >-
      (github.event.workflow_run.conclusion == 'success' || github.event.workflow_run.conclusion == 'failure')
      && github.event.workflow_run.event == 'pull_request'
      && github.event.workflow_run.head_repository.full_name != github.event.workflow_run.repository.full_name
    permissions:
      actions: read
      contents: read
      id-token: write
    steps:
      - name: Install Flakiness CLI
        run: curl -LsSf https://cli.flakiness.io/install.sh | sh

      - name: Download flakiness-report artifacts
        env:
          GH_TOKEN: ${{ github.token }}
          RUN_ID: ${{ github.event.workflow_run.id }}
        run: gh run download "$RUN_ID" --repo "$GITHUB_REPOSITORY" --pattern 'flakiness-report-*' --dir .

      - name: Upload to Flakiness.io
        run: find . -path '*/flakiness-report-*/report.json' -exec flakiness upload {} \;

Using Access Token (fallback when OIDC is not available):

.github/workflows/flakiness-upload-fork-prs.yml

name: Upload Flakiness.io report (fork PRs)
on:
  workflow_run:
    # Must match the name(s) of workflows that produce flakiness-report artifacts
    workflows: ["Tests"]
    types: [completed]

jobs:
  upload-flakiness-report:
    runs-on: ubuntu-latest
    if: >-
      (github.event.workflow_run.conclusion == 'success' || github.event.workflow_run.conclusion == 'failure')
      && github.event.workflow_run.event == 'pull_request'
      && github.event.workflow_run.head_repository.full_name != github.event.workflow_run.repository.full_name
    permissions:
      actions: read
      contents: read
    steps:
      - name: Install Flakiness CLI
        run: curl -LsSf https://cli.flakiness.io/install.sh | sh

      - name: Download flakiness-report artifacts
        env:
          GH_TOKEN: ${{ github.token }}
          RUN_ID: ${{ github.event.workflow_run.id }}
        run: gh run download "$RUN_ID" --repo "$GITHUB_REPOSITORY" --pattern 'flakiness-report-*' --dir .

      - name: Upload to Flakiness.io
        env:
          FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
        run: find . -path '*/flakiness-report-*/report.json' -exec flakiness upload {} \;