Captain is an open source CLI that can:
- detect and quarantine flaky tests
- automatically retry failed tests
- partition files for parallel execution
- generate comprehensive test failure summaries
See the announcement blog post and documentation to learn more.
The Captain CLI is easy to integrate into a build process running on any CI platform. See the documentation on getting started. We’re happy to help with any integrations. Say hello on Discord or reach out at [email protected]
Please read CONTRIBUTING.md for information around our development & contribution process.
Captain v1.9.11 - Release Candidate
Note: As this is our first release on
releases/v1, the diff & commit history on this PR are not complete. To see all changes from the previous release, go to https://github.com/rwx-research/captain/compare/e14a5b9...b0f3f7d9af93c66f62a2e0b6f8546b899ec7dd75
Release notes (to be published) are below:
--title. This can be useful if Captain is unable to derive commit messages from your environment.
captain remove. Previously this only worked with the
Send up a title with all providers
This will let us have fewer cases in Cloud where we show "N/A" for the title. Right now we use the commit message for this, but we don't always have a commit message available.
Expose the new reporter options
This required a few bug fixes that I just lumped into this PR, but long story short:
title falls back to first line of commit message
if title isn't set but commit message is, all providers falls back to commit message
previously most providers were doing this. This PR
additionally, it makes all of the provider-specific
Captain v1.10.1 - Release Candidate
Integration test the reporters
Adds integration tests for the reporters both with and without a config file (that they produce files, nothing about the content since that may change version to version and is itself unit tested)
Create GitHub releases through GitHub Actions
This modifies our current CD process in the following way:
releases/*branch - we'll have one for each major version. Right now, that's
version.go. Additionally, the latest major version tag & release (
v1) will be updated. The action will further also upload assets to the release. This looks something like this:
Note that the existing release pipeline is unchanged - we'll also upload releases to S3 and update
After this process, someone will have to manually move the release out of "draft" status. This is so we get a chance to update the release notes or make any other changes if necessary. Subscribers to this repo will get notified once there is a new release in the "published" state.
No-op on github-step-summary reporter when GITHUB_STEP_SUMMARY is blank
With the old behavior, it was a little annoying to use
github-step-summaryin the config file because running the CLI anywhere but in GitHub Actions (e.g. locally for testing) would result in an error. With this behavior, if you're not on GitHub, we just won't try to write the summary.
Fixes a few markdown summary issues
backwards compatibility testing
going forward, wrap tests or assertions you don't want included in the backwards compatibility tests in
If no reporters are set in the suite config, make a new config map
The changes in #16 introduced new problems with reporter configurations.
map[string]stringin Go defaults to
nil, not an empty map 😞 Right now, if you don't specify config file reporters but do try to specify a reporter via CLI flag, you get a nil pointer error.
This fixes that. I'll get tests in place ASAP (how we discussed them in Slack), but this'll fix the bug (tested by hand locally with config file only, flags only, and both).
Introduce a new release process
Introduces a new release process for Captain. Instead of tracking release branches (for production), we'll trigger a workflow which will:
Additionally, this workflow supports
testingreleases. This PR doesn't introduce it yet, but on merges to main, we'll trigger this workflow to create an
unstablerelease tagged with the commit sha and an aliased release
unstablewhich always points to the most recent commit on main. The
testingrelease allows for engineers to create
testing-SHAreleases based on commits in their branches if necessary.
testingreleases do not require approval.
Noteworthy: we no longer upload
versions.json. I checked datadog and no one is using this now.
I've tested this fairly extensively on my fork- the only bits that haven't been actually invoked are the
aws s3 cpand
aws cloudfront create-invalidationcalls (since I didn't bother setting up a place to test that; I did, however, change them to echos to check the arguments).
Waiting for approval