Get started with tests in Pipelines

Tests is in open beta and available to Bitbucket Pipelines customers on Standard and Premium plans.

How test reporting works in a Pipeline step

If your build step generates test results, pipelines will automatically detect and show any failures in the web interface. This currently works with JUnit and Maven Surefire XML formats, regardless of the language they are written in.

Any user with read access to the repository can view the test results in the pipelines log panel. If failed test results are found, the log view will change to highlight the failed tests, showing the reason and the stack-trace if present. The full log view is also available using the Build tab.

The Tests tab will only show if you have failed tests to review.

Failed test limits

Only the first 100 failed test results are displayed.

Perquisites

Your test framework must generate JUnit-style XML reports.
Bitbucket Pipelines must be enabled for your repository.

Go to your Bitbucket repository and select Test management.
Review the Test summaries page to see aggregated data for each test, including failure rate, average duration, and variance duration.
Use the search bar or filters to find specific tests by name, label, or state.
Select a test summary to view detailed test executions for that test, including buildNumber, commit message, lastExecuted, execution duration, and outcome.
[Optional] To mark a test as Flaky, go to the Test state column, then choose Flaky from the dropdown menu.
[Optional] To mark a test as Quarantined, select More actions (…) for the test and mark it as Quarantined.

If the repository meets the eligibility criteria and the frameworks generate XML format reports as described below, then the tests will be generated automatically.

Configure test reporting

To enable test reporting, make sure that build test reports are generated in one of the supported default locations (with a directory depth of 3 levels):

./**/surefire-reports/**/*.xml

./**/failsafe-reports/**/*.xml

./**/test-results/**/*.xml

./**/test-reports/**/*.xml

./**/TestResults/**/*.xml

The test report file scanner will begin searching from the base directory of your build: /opt/atlassian/pipelines/agent/build

If the test reports are in some path other than the default path mentioned above, it can be defined like the example provided below:

Example

artifacts:
  upload:
   - name: "custom test reports"
     type: "test-reports"
     paths:
     - "custom_path/*.xml"
     - "**/custom_path2/*.xml"

Configuring test runners to generate XML format test reports

Some test frameworks, such as TestNG, generate multiple JUnit XML reports containing the same tests but with different suite names.

In this case, you will see multiple tests with duplicated test names but different suites on the test management page, which can cause confusion.

Maven Surefire plugins

For Maven build jobs, no special configuration is required if you are using Maven Surefire Plugin. The reports are automatically generated when the unit or integration maven test goals are executed.

PHPUnit

For PHPUnit test reports, you should explicitly specify the --log-junit parameter to generate the test reports output to a particular location.

Example bitbucket-pipelines.yml

image: php:7.1.1
pipelines:
  default:
    - step:
        script:
          - apt-get update && apt-get install -y unzip
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer require phpunit/phpunit
          - vendor/bin/phpunit --log-junit ./test-reports/junit.xml
    

dotnet

For dotnet test reporting, you can use JUnitTestLogger to generate compatible XML test results.

Using JUnitTestLogger to output JUnit format directly:

Example bitbucket-pipelines.yml
image: mcr.microsoft.com/dotnet/sdk pipelines: default: - step: script: - dotnet add package JUnitTestLogger --version 1.1.0 - dotnet test --logger "junit"

Junit/TestNG (with Maven)

By default, JUnit and TestNG generate test reports in XML format. Run the command below to generate test reports in target/surefire-reports.

mvn clean test

Below is an example of how you can add it to your Bitbucket Pipelines script:

pipelines:
  default:
    - step:
        name: Build and Test
        image: maven:3.8.4-openjdk-11
        script:
          - mvn clean install 

PyTest

Generate a JUnit XML report with Pytest using the --junit-xml flag. Specify the path to store the test report, for example, test-results/<report-name>.xml.

Below is an example of how you can add it to your Bitbucket Pipelines script:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - pip
        script:
          - pytest --junit-xml=test-results/junit.xml 

Mocha

Add the following package in dependencies in package.json: “mocha-junit-reporter": "^2.2.1" or use npm to install it using npm install --save-dev mocha-junit-reporter.

...
  "scripts": {
    "test": "mocha test/**/*.test.js --reporter mocha-junit-reporter,
  },
  "devDependencies": {
    "mocha-junit-reporter": "^2.2.1"
  },
...

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

TestCafe

Install or include the following dependency: npm install --save-dev testcafe-reporter-junit
Then update package.json with the following test script:

{
  "scripts": {
    "test": "testcafe chrome:headless tests --reporter junit:test-reports/junit.xml",
  }
}

The above applies to Google Chrome. Configuration may vary slightly by browser. Reports will be created in the test-reports folder or the specified path in the command.

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Jest

Use jest-junit to generate a report in Junit XML format.

...
  "scripts": {
    "test": "jest --reporters=default --reporters=jest-junit",
  },
  "devDependencies": {
    "jest": "^29.7.0",
    "jest-junit": "^16.0.0"
  },
...

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

WebdriverIO

Install @wdio/junit-reporter as a devDependency in your package.json using the command below:

npm install @wdio/junit-reporter --save-dev

2. Configure the WebdriverIO config file:

export const config = {
  // ... other config options
  
  reporters: [
    'spec', // Optional: for console output
    ['junit', {
      outputDir: 'test-results',        // Directory where reports will be saved
      outputFileFormat: () => 'junit.xml' // Filename format
    }]
  ],
  
  // ... rest of your config
};

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Playwright

The following line needs to be added in Playwright config under reporter section in the playwright.config.ts file to generate this test report:

['junit', { outputFile: 'test-results/junit.xml' }]

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Cypress

Add the following package in dependencies in package.json: “mocha-junit-reporter": "^2.2.1"
or use npm to install it using npm install --save-dev mocha-junit-reporter.
Run Cypress with the required options in your Bitbucket Pipelines configuration:

npx cypress run --reporter mocha-junit-reporter --reporter-options \
  "mochaFile=test-reports/junit/test-results-[hash].xml,\
   testsuitesTitle=true,\
   suiteTitleSeparatedBy=' / ',\
   rootSuiteTitle='Cypress Tests'"

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npx cypress run --reporter mocha-junit-reporter --reporter-options "mochaFile=test-reports/junit/test-results-[hash].xml, testsuitesTitle=true, suiteTitleSeparatedBy=' / ', rootSuiteTitle='Cypress Tests'"

NUnit

Use the popular JunitXml.TestLogger (71M downloads) from NuGet Package Manager to generate test reports in JUnit XML format. The latest stable version is 7.0.2. More details are available here.

// Install the Junit Test Logger via package reference:
<PackageReference Include="JunitXml.TestLogger" Version="7.0.2" />

Update your pipeline script to generate the report in JUnit XML format. See the example below:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - dotnetcore
        script:
          - echo "Restoring NuGet packages..."
          - dotnet restore
          - echo "Building solution..."
          - dotnet build --configuration Release --no-restore
          - echo "Running tests with JUnit XML output..."
          - mkdir -p TestResults test-reports test-results
          - dotnet test --configuration Release --no-build --logger "junit;LogFilePath=TestResults/test-results.xml"

Selenium/Appium

Selenium is a tool that simulates browser actions. It, however, is not a testing framework; rather, it must be used in combination with testing frameworks like JUnit, TestNG, Mocha, or pytest to run organized tests.

These testing frameworks have their own setting for generating JUnit XML style reports, however, Selenium in itself can’t generate any reports as it is nothing more than a tool that simulates actions.

Similarly, Appium supports mobile testing with test frameworks such as WebdriverIO and Mocha. Ensure the test framework generates Appium test reports in JUnit XML format.

Other test runners

For other test runners, make sure that the compatible XML test results are generated as a part of your build in one of the default supported locations.

Was this helpful?

It wasn't accurateIt wasn't clearIt wasn't relevant

Still need help?

The Atlassian Community is here for you.

Ask the Community

Bitbucket Support

Bitbucket is getting a new navigation

Get started with tests in Pipelines

How test reporting works in a Pipeline step

Configure test reporting

Configuring test runners to generate XML format test reports

Maven Surefire plugins

PHPUnit

dotnet

Junit/TestNG (with Maven)

PyTest

Mocha

TestCafe

Jest

WebdriverIO

Playwright

Cypress

NUnit

Selenium/Appium

Other test runners

Still need help?