🥒 Cucumber PoC 🥒

[laurazard]

What I did:

I got a bit curious from what we discussed regarding a PoC using Cucumber, so I started playing around with it! It needs a ton of refactoring/reorganizing (it's currently really ugly), and a good look regarding the testing utils and parallelization.

So far, steps implemented are:

• (Given) a compose file [xxx]: sets the compose file to get piped into the compose command
• (When) I run "compose [command]": runs the specified compose command with the given fixture piped in
• (Then) output contains [xxx]: checks that the output of the last command contains [xxx]
• (...) exit code is [i]: checks that the exit code of the last command is [i]
• (...) [service] service is [status]: runs compose ps, checks that the specific service is the desired status

This allows us to define tests such as:

Feature: Simple service up

Background:
    Given a compose file
        """
        services:
          simple:
            image: alpine
            command: top
        """

Scenario: compose up
    When I run "compose up -d"
    Then the output contains "simple-1  Started"
    And service "simple" is "running"

Run the cucumber feature tests with go test -v -run ^TestCucumberFeatures$ github.com/docker/compose/v2/pkg/e2e/cucumber

Output should look something like:

Signed-off-by: Laura Brehm [email protected]

(not mandatory) A picture of a cute animal, if possible in relation with what you did

[laurazard]

For comparison:

	t.Run("abort-on-container-exit", func(t *testing.T) {
		res := c.RunDockerComposeCmdNoCheck(t, "-f", "./fixtures/cascade-stop-test/compose.yaml", "--project-name", projectName, "up", "--abort-on-container-exit")
		res.Assert(t, icmd.Expected{ExitCode: 1, Out: `should_fail-1 exited with code 1`})
		res.Assert(t, icmd.Expected{ExitCode: 1, Out: `Aborting on container exit...`})
	})

	t.Run("exit-code-from", func(t *testing.T) {
		res := c.RunDockerComposeCmdNoCheck(t, "-f", "./fixtures/cascade-stop-test/compose.yaml", "--project-name", projectName, "up", "--exit-code-from=sleep")
		res.Assert(t, icmd.Expected{ExitCode: 137, Out: `should_fail-1 exited with code 1`})
		res.Assert(t, icmd.Expected{ExitCode: 137, Out: `Aborting on container exit...`})
	})

	t.Run("exit-code-from unknown", func(t *testing.T) {
		res := c.RunDockerComposeCmdNoCheck(t, "-f", "./fixtures/cascade-stop-test/compose.yaml", "--project-name", projectName, "up", "--exit-code-from=unknown")
		res.Assert(t, icmd.Expected{ExitCode: 1, Err: `no such service: unknown`})
	})

turns into:

Feature: Stop

Background:
    Given a compose file
        """
        services:
          should_fail:
            image: alpine
            command: ls /does_not_exist
          sleep: # will be killed
            image: alpine
            command: ping localhost
        """

Scenario: Cascade stop
    When I run "compose up --abort-on-container-exit"
    Then the output contains "should_fail-1 exited with code 1"
    And the output contains "Aborting on container exit..."
    And the exit code is 1

Scenario: Exit code from
    When I run "compose up --exit-code-from sleep"
    Then the output contains "should_fail-1 exited with code 1"
    And the output contains "Aborting on container exit..."
    And the exit code is 137

Scenario: Exit code from unknown service
    When I run "compose up --exit-code-from unknown"
    Then the output contains "no such service: unknown"
    And the exit code is 1

[nicksieger]

Really like where this is going so far. I think it's worth trying out.

I'm wondering if deleting the corresponding existing tests that these cucumber tests replace would help with evaluating it? That way we can compare the current e2e test with a cucumber version in the same PR.

[nicksieger]

What's behind the need for this replace, out of curiosity?

[laurazard]

cucumber/godog currently allows for passing in a testing.T instance to run scenarios as subtests with, but it does not make that testing.T instance available to the scenario initializer, which was needed for passing to our e2e.NewCLI() that uses some of it's helper functions. I'll make a PR to see if I can contribute that upstream soon :)

[laurazard]

Thanks! :)

I removed cascade_stop_test.go which was replaced by stop.feature and compose_down_test.go which was replaced by down.feature, and their corresponding fixtures, so the comparison is clearer in the PR.

[laurazard]

From here, moved the new tests into a new e2e module to separate out test dependencies.

[thaJeztah]

So, it's a bit messy to do so, but it may be good to have make targets to verify vendoring / go mod files to make sure they're kept in sync (to prevent situations where the e2e tests test with different versions than the main module) - (replace rules are a pain! 😞).

In containerd there's some script to do that; called from https://github.com/containerd/containerd/blob/39f7cd73e7cc3e1d24f3557adfce5b68484136f7/Makefile#L441-L456

[laurazard]

Sounds good! That's definitely a concern now having two separate sets of dependencies to keep in sync. I'll take a look at it, thanks :)

[laurazard]

@thaJeztah took a look at it and added a Makefile target validate-modules here to verify it, using the script from containerd :)

[laurazard]

The CI isn't triggering 😢 Cursory google search seems to suggest I close and reopen the PR, so I'm going to try that.

[glours]

Looks cool, we'll need to figure out how to import fixtures for tests using huge compose files

[milas]

🥒 🥒 🥒 🥒

[laurazard]

🥳