Playwright

Demo

Report demo is available at https://flakiness.io/flakiness/demo-attachments

The official Flakiness.io reporter for Playwright Test. The open source implementation is available at github.com/flakiness/playwright.

A custom Playwright test reporter that generates Flakiness Reports from your Playwright test runs. The reporter automatically converts Playwright test results into the standardized Flakiness JSON format, capturing test outcomes, attachments, system utilization, and environment information. This enables you to track test flakiness over time, analyze patterns across different environments, and identify regressions in your test suite.

Quick Start

1. Install the Flakiness.io Playwright reporter:

   npm install -D @flakiness/playwright

2. Configure the reporter in your playwright.config.ts file:

   playwright.config.ts

   import { defineConfig } from '@playwright/test';
   
   export default defineConfig({
     reporter: [
       ['@flakiness/playwright']
     ],
   });

3. Run your tests. The report will be automatically generated in the ./flakiness-report folder:

   npx playwright test

4. View the interactive report:

   npx flakiness show ./flakiness-report

Tip

Make sure to add flakiness-report folder to your .gitignore. You can also use the Flakiness CLI to view reports locally.

CI/CD Integration

Reports are automatically uploaded to Flakiness.io at the end of each test run when authentication is configured. If no credentials are present, the report is still generated locally and can be uploaded manually later.

When uploading succeeds, you’ll see a confirmation in your terminal:

  45 passed (1.5m)
[flakiness.io] ✓ Uploaded as https://flakiness.io/flakiness/flakiness/run/6574

Upload Method	Status	Guide
GitHub OIDC v1.1.0+	Supported	GitHub Actions
Access Token	Supported	GitHub Actions
Other CI	Supported	Other CI Providers

Tip

GitHub OIDC is the recommended method — no secrets to manage. It requires the flakinessProject config option. See the GitHub Actions guide for setup instructions.

Configuration

The reporter accepts the following configuration options:

endpoint?: string

Custom Flakiness.io endpoint URL for uploading reports. Defaults to the FLAKINESS_ENDPOINT environment variable, or https://flakiness.io if not set.

Use this option to point to a custom or self-hosted Flakiness.io instance.

reporter: [
  ['@flakiness/playwright', { endpoint: 'https://custom.flakiness.io' }]
]

token?: string

Access token for authenticating with Flakiness.io when uploading reports. Defaults to the FLAKINESS_ACCESS_TOKEN environment variable.

If no token is provided, the reporter will attempt GitHub OIDC when flakinessProject is set. If neither is available, the report is generated locally without uploading.

reporter: [
  ['@flakiness/playwright', { token: 'your-access-token' }]
]

outputFolder?: string

Directory path where the Flakiness report will be written. Defaults to flakiness-report in the current working directory, or the FLAKINESS_OUTPUT_DIR environment variable if set.

reporter: [
  ['@flakiness/playwright', { outputFolder: './test-results/flakiness' }]
]

open?: 'always' | 'never' | 'on-failure'

Controls when the report viewer should automatically open in your browser after test completion.

• 'on-failure' (default): Opens the report only if tests failed and running in an interactive terminal (not in CI)
• 'always': Always opens the report after test completion (when running in an interactive terminal)
• 'never': Never automatically opens the report

reporter: [
  ['@flakiness/playwright', { open: 'always' }]
]

collectBrowserVersions?: boolean

When enabled, the reporter will launch each browser type used in your Playwright projects to detect and record the actual browser version. This information is added to the environment metadata.

Note: This option requires launching browsers, which adds overhead to report generation. Enable only when browser version information is critical for your analysis.

reporter: [
  ['@flakiness/playwright', { collectBrowserVersions: true }]
]

flakinessProject?: string

The Flakiness.io project to upload reports to, in orgSlug/projectSlug format. Required for GitHub OIDC authentication. Defaults to the FLAKINESS_PROJECT environment variable.

reporter: [
  ['@flakiness/playwright', { flakinessProject: 'my-org/my-project' }]
]

Environment Variables

The reporter respects the following environment variables:

• FLAKINESS_ACCESS_TOKEN: Access token for Flakiness.io uploads (equivalent to token option)
• FLAKINESS_PROJECT: Flakiness.io project in orgSlug/projectSlug format (equivalent to flakinessProject option)
• FLAKINESS_ENDPOINT: Custom Flakiness.io endpoint URL (equivalent to endpoint option)
• FLAKINESS_OUTPUT_DIR: Output directory for reports (equivalent to outputFolder option)
• FK_ENV_*: Custom environment variables prefixed with FK_ENV_ are automatically included in the environment’s metadata. The prefix is stripped and the key is converted to lowercase.

Example:

export FK_ENV_DEPLOYMENT=staging
export FK_ENV_REGION=us-east-1

This will result in the environment containing:

{
  "metadata": {
    "deployment": "staging",
    "region": "us-east-1"
  }
}

Flakiness.io platform will create a dedicated history for tests executed in each unique environment. This means tests run with FK_ENV_DEPLOYMENT=staging will have a separate timeline from tests run with FK_ENV_DEPLOYMENT=production, allowing you to track flakiness patterns specific to each deployment environment.

Example Configuration

Here’s a complete example with all options:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['@flakiness/playwright', {
      flakinessProject: 'my-org/my-project',
      endpoint: process.env.FLAKINESS_ENDPOINT,
      token: process.env.FLAKINESS_ACCESS_TOKEN,
      outputFolder: './flakiness-report',
      open: 'on-failure',
      collectBrowserVersions: false,
    }]
  ],
  // ... rest of your config
});