Captain is an open source CLI that supercharges testing capabilities across 15 different testing frameworks

  • By RWX
  • Last update: Apr 8, 2023
  • Comments: 12


🌐   🐦 @rwx_research   πŸ’¬ discord   πŸ“š documentation

Captain is an open source CLI that can:

  • detect and quarantine flaky tests
  • automatically retry failed tests
  • partition files for parallel execution
  • generate comprehensive test failure summaries

See the announcement blog post and documentation to learn more.

Getting Started

The Captain CLI is easy to integrate into a build process running on any CI platform. See the documentation on getting started. We’re happy to help with any integrations. Say hello on Discord or reach out at [email protected]


Please read for information around our development & contribution process.



  • 1

    Captain v1.9.11 - Release Candidate

    Note: As this is our first release on releases/v1, the diff & commit history on this PR are not complete. To see all changes from the previous release, go to

    Release notes (to be published) are below:



    • Ability to explicitly disable Cloud mode through the config file.
    • Additional warning messages if Captain is misconfigured.
    • A fallback to the pull-request name / number if Captain can't determine a commit message in GitHub Actions.
    • Ability to manually set a "title" for a test-suite execution in Captain via CAPTAIN_TITLE or --title. This can be useful if Captain is unable to derive commit messages from your environment.
    • Additional options to configure the partitioning delimiter (can now be set via CAPTAIN_DELIMITER)
    • Support for passing the suite-id as an argument to captain add and captain remove. Previously this only worked with the --suite-id flag.


    • Some error messages were modified to be easier to understand.
  • 2

    Send up a title with all providers

    This will let us have fewer cases in Cloud where we show "N/A" for the title. Right now we use the commit message for this, but we don't always have a commit message available.

  • 3

    Expose the new reporter options

    This required a few bug fixes that I just lumped into this PR, but long story short:

    • Uses the config file in this repo
    • Fixes how the config file's reporters setting and cli flag reporters interact
    • Exposes the github step summary and markdown summary reporters
  • 4

    title falls back to first line of commit message

    if title isn't set but commit message is, all providers falls back to commit message

    previously most providers were doing this. This PR

    • ensures it happens for the generic provider
    • ensures it happens late -- if the commit message is overridden somehow, the title is pulled from the latest commit message

    additionally, it makes all of the provider-specific MakeProvider methods private

  • 5

    Captain v1.10.1 - Release Candidate



    • When fixing the interaction between the config file and flags while specifying reporters in #16, a subtle bug was introduced which caused an error when specifying reporters via CLI flag and none via config file.
  • 6

    Integration test the reporters

    Adds integration tests for the reporters both with and without a config file (that they produce files, nothing about the content since that may change version to version and is itself unit tested)

  • 7

    Create GitHub releases through GitHub Actions

    This modifies our current CD process in the following way:

    1. Releases happen on a releases/* branch - we'll have one for each major version. Right now, that's releases/v1.
    2. All releases need to happen through a pull request. The "Release Requirements" action will ensure that a version bump took place. Other requirements can be found in RFC 30.
    3. Once merged, Github Actions will create a new tag & release for the version from version.go. Additionally, the latest major version tag & release (v1) will be updated. The action will further also upload assets to the release. This looks something like this:

    Note that the existing release pipeline is unchanged - we'll also upload releases to S3 and update versions.json for now.

    After this process, someone will have to manually move the release out of "draft" status. This is so we get a chance to update the release notes or make any other changes if necessary. Subscribers to this repo will get notified once there is a new release in the "published" state.

  • 8

    No-op on github-step-summary reporter when GITHUB_STEP_SUMMARY is blank

    With the old behavior, it was a little annoying to use github-step-summary in the config file because running the CLI anywhere but in GitHub Actions (e.g. locally for testing) would result in an error. With this behavior, if you're not on GitHub, we just won't try to write the summary.

  • 9

    Fixes a few markdown summary issues

    • Ensure quarantined and canceled tests end up in the right section
    • Combine message and backtrace to better handle multiline message
    • Strip ANSI escape codes in the markdown summary
  • 10

    backwards compatibility testing

    this PR

    • checks for changes in the integration test suites between all releaesed versions after 1.10.2
    • for each of those, runs the integration tests
    • the integration tests prefix the top level description with the version
      • and disable some over-constrained assertions within the tests

    going forward, wrap tests or assertions you don't want included in the backwards compatibility tests in withoutBackwardsCompatibility, e.g.

    withoutBackwardsCompatibility(func() {
  • 11

    If no reporters are set in the suite config, make a new config map

    The changes in #16 introduced new problems with reporter configurations. map[string]string in Go defaults to nil, not an empty map 😞 Right now, if you don't specify config file reporters but do try to specify a reporter via CLI flag, you get a nil pointer error.

    This fixes that. I'll get tests in place ASAP (how we discussed them in Slack), but this'll fix the bug (tested by hand locally with config file only, flags only, and both).

  • 12

    Introduce a new release process

    Introduces a new release process for Captain. Instead of tracking release branches (for production), we'll trigger a workflow which will:

    • Create a draft release (or overwrite an existing draft release, if we need to re-cut a release)
    • Build the binaries
    • Upload those binaries to the draft release
    • Wait for approval
      • For SOC2 and to give an opportunity to tweak the release notes
    • Publish that release
    • Update (or create) an aliased major-version release
    • Upload the binaries from the new version's release to the major-version release
    • Upload the binaries from the new version's release to cloudfront (S3)

    Additionally, this workflow supports unstable and testing releases. This PR doesn't introduce it yet, but on merges to main, we'll trigger this workflow to create an unstable release tagged with the commit sha and an aliased release unstable which always points to the most recent commit on main. The testing release allows for engineers to create testing-SHA releases based on commits in their branches if necessary. unstable and testing releases do not require approval.

    Noteworthy: we no longer upload versions.json. I checked datadog and no one is using this now.

    I've tested this fairly extensively on my fork- the only bits that haven't been actually invoked are the aws s3 cp and aws cloudfront create-invalidation calls (since I didn't bother setting up a place to test that; I did, however, change them to echos to check the arguments).

    Waiting for approval

    Screenshot 2023-04-11 at 11 16 26 AM

    Approved production release

    Screenshot 2023-04-11 at 1 56 31 PM

    Rejected production release

    Screenshot 2023-04-11 at 1 56 42 PM

    unstable release

    Screenshot 2023-04-11 at 1 57 02 PM

    testing release

    Screenshot 2023-04-11 at 1 58 44 PM