Note: As this is our first release on releases/v1, the diff & commit history on this PR are not complete. To see all changes from the previous release, go to https://github.com/rwx-research/captain/compare/e14a5b9...b0f3f7d9af93c66f62a2e0b6f8546b899ec7dd75

Release notes (to be published) are below:

Changelog

Added

• Ability to explicitly disable Cloud mode through the config file.
• Additional warning messages if Captain is misconfigured.
• A fallback to the pull-request name / number if Captain can't determine a commit message in GitHub Actions.
• Ability to manually set a "title" for a test-suite execution in Captain via CAPTAIN_TITLE or --title. This can be useful if Captain is unable to derive commit messages from your environment.
• Additional options to configure the partitioning delimiter (can now be set via CAPTAIN_DELIMITER)
• Support for passing the suite-id as an argument to captain add and captain remove. Previously this only worked with the --suite-id flag.

Changed

• Some error messages were modified to be easier to understand.

This will let us have fewer cases in Cloud where we show "N/A" for the title. Right now we use the commit message for this, but we don't always have a commit message available.

This required a few bug fixes that I just lumped into this PR, but long story short:

• Uses the config file in this repo
• Fixes how the config file's reporters setting and cli flag reporters interact
• Exposes the github step summary and markdown summary reporters

if title isn't set but commit message is, all providers falls back to commit message

previously most providers were doing this. This PR

• ensures it happens for the generic provider
• ensures it happens late -- if the commit message is overridden somehow, the title is pulled from the latest commit message

additionally, it makes all of the provider-specific MakeProvider methods private

Changelog

Fixed

• When fixing the interaction between the config file and flags while specifying reporters in #16, a subtle bug was introduced which caused an error when specifying reporters via CLI flag and none via config file.

Adds integration tests for the reporters both with and without a config file (that they produce files, nothing about the content since that may change version to version and is itself unit tested)

This modifies our current CD process in the following way:

1. Releases happen on a releases/* branch - we'll have one for each major version. Right now, that's releases/v1.
2. All releases need to happen through a pull request. The "Release Requirements" action will ensure that a version bump took place. Other requirements can be found in RFC 30.
3. Once merged, Github Actions will create a new tag & release for the version from version.go. Additionally, the latest major version tag & release (v1) will be updated. The action will further also upload assets to the release. This looks something like this:

Note that the existing release pipeline is unchanged - we'll also upload releases to S3 and update versions.json for now.

After this process, someone will have to manually move the release out of "draft" status. This is so we get a chance to update the release notes or make any other changes if necessary. Subscribers to this repo will get notified once there is a new release in the "published" state.

With the old behavior, it was a little annoying to use github-step-summary in the config file because running the CLI anywhere but in GitHub Actions (e.g. locally for testing) would result in an error. With this behavior, if you're not on GitHub, we just won't try to write the summary.

• Ensure quarantined and canceled tests end up in the right section
• Combine message and backtrace to better handle multiline message
• Strip ANSI escape codes in the markdown summary

this PR

• checks for changes in the integration test suites between all releaesed versions after 1.10.2
• for each of those, runs the integration tests
• the integration tests prefix the top level description with the version
  • and disable some over-constrained assertions within the tests

going forward, wrap tests or assertions you don't want included in the backwards compatibility tests in withoutBackwardsCompatibility, e.g.

withoutBackwardsCompatibility(func() {
  Expect(something).To(Equal("something-that-will-likely-change-between-versions"))
})

The changes in #16 introduced new problems with reporter configurations. map[string]string in Go defaults to nil, not an empty map 😞 Right now, if you don't specify config file reporters but do try to specify a reporter via CLI flag, you get a nil pointer error.

This fixes that. I'll get tests in place ASAP (how we discussed them in Slack), but this'll fix the bug (tested by hand locally with config file only, flags only, and both).

Introduces a new release process for Captain. Instead of tracking release branches (for production), we'll trigger a workflow which will:

• Create a draft release (or overwrite an existing draft release, if we need to re-cut a release)
• Build the binaries
• Upload those binaries to the draft release
• Wait for approval
  • For SOC2 and to give an opportunity to tweak the release notes
• Publish that release
• Update (or create) an aliased major-version release
• Upload the binaries from the new version's release to the major-version release
• Upload the binaries from the new version's release to cloudfront (S3)

Additionally, this workflow supports unstable and testing releases. This PR doesn't introduce it yet, but on merges to main, we'll trigger this workflow to create an unstable release tagged with the commit sha and an aliased release unstable which always points to the most recent commit on main. The testing release allows for engineers to create testing-SHA releases based on commits in their branches if necessary. unstable and testing releases do not require approval.

Noteworthy: we no longer upload versions.json. I checked datadog and no one is using this now.

I've tested this fairly extensively on my fork- the only bits that haven't been actually invoked are the aws s3 cp and aws cloudfront create-invalidation calls (since I didn't bother setting up a place to test that; I did, however, change them to echos to check the arguments).

Waiting for approval

Approved production release

Rejected production release

unstable release

testing release