Cypress Bamboo Plans

Background

The Cypress test suite contains system tests verifying the most important features of Artemis. System tests test the whole system and therefore require a complete deployment of Artemis first. In order to prevent as many faults (bugs) as possible from being introduced into the develop branch, we want to execute the Cypress test suite whenever new commits are pushed to a Git branch (just like the unit and integration test suites).

To accomplish this we need to be able to dynamically deploy multiple different instances of Artemis at the same time. An ideal setup would be to deploy the whole Artemis system using Kubernetes. However, this setup is too complex at the moment. The main reason for the complexity is that it is very hard to automatically setup Docker containers for the external services (Jira, Bitbucket and Bamboo) and connect them directly with Artemis.

Therefore, the current setup only dynamically deploys the Artemis server and configures it to connect to the prelive system, which is already properly setup in the university data center.

Artemis Deployment on Bamboo Build Agent

Every execution of the Cypress test suite requires its own deployment of Artemis. The easiest way to accomplish this is to deploy Artemis locally on the build agent, which executes the Cypress tests. Using docker-compose we can start a MySQL database and the Artemis server locally on the build agent and connect it to the prelive system in the university data center.

Artemis Deployment on Bamboo Build Agent for Cypress

Artemis Deployment on Bamboo Build Agent for Cypress

In total there are three Docker containers started in the Bamboo build agent:

  1. MySQL

    This container starts a MySQL database and exposes it on port 3306. The container automatically creates a new database ‘Artemis’ and configures it with the recommended settings for Artemis. The Cypress setup reuses the already existing MySQL docker image from the standard Artemis Docker setup.

  2. Artemis

    The Docker image for the Artemis container is created from the already existing Dockerfile. When the Bamboo build of the Cypress test suite starts, it retrieves the Artemis executable (.war file) from the Artemis build plan. Upon creation of the Artemis Docker image the executable is copied into the image together with configuration files for the Artemis server.

    The main configuration of the Artemis server is contained in the application.yml file. However, this file does not contain any security relevant information. Security relevant settings like the credentials to the Jira admin account in the prelive system are instead passed to the Docker container via environment variables. This information is accessible to the Bamboo build agent via Bamboo plan variables.

    The Artemis container is also configured to depend on the MySQL container and uses health checks to wait until the MySQL container is up and running.

  3. Cypress

    Cypress offers a variety of docker images to execute Cypress tests. We use an image which has the Cypress operating system dependencies and a Chrome browser installed. However, Cypress itself is not installed in these images. This is convenient for us because the image is smaller and the Artemis Cypress project requires additional dependencies to fully function. Therefore, the Artemis Cypress Docker container is configured to install all dependencies (using npm ci) upon start. This will also install Cypress itself. Afterwards the Artemis Cypress test suite is executed.

    The necessary configuration for the Cypress test suite is also passed in via environment variables. Furthermore, the Cypress container depends on the Artemis container and is only started once Artemis has been fully booted.

Bamboo webhook

The Artemis instance deployed on the build agent is not publicly available to improve the security of this setup. However, in order to get the build results for programming exercise submissions Artemis relies on a webhook from Bamboo to send POST requests to Artemis. To allow this, an extra rule has been added to the firewall allowing only the Bamboo instance in the prelive system to connect to the Artemis instance in the build agent.

Timing

As mentioned above, we want the Cypress test suite to be executed whenever new commits are pushed to a Git branch. This has been achieved by adding the Cypress Github build plan as a child dependency to the Artemis Build build plan. The Artemis Build build plan is triggered whenever a new commit has been pushed to a branch.

The Cypress build plan is only triggered after a successful build of the Artemis executable. This does imply a delay (about 10 minutes on average) between the push of new commits and the execution of the Cypress test suite, since the new Artemis executable first has to be built.

NOTE: The Cypress test suite is only automatically executed for internal branches and pull requests (requires access to this GitHub repository) not for external ones. In case you need access rights, please contact the maintainer Stephan Krusche.

Automatic flaky test detection based on changed code

In addition to our regular Cypress execution, we also run a special experimental build plan that attempts to detect flaky tests based on the changed code. To do this, we have some special Docker configurations that are specific to this build plan.

  1. Docker Image Extensions

    We extend the existing Dockerfile to create the Docker image for the Artemis container. For the flaky test detection build plan, we need to change the Artemis startup and add the unzip dependency. To do this, we have a special Dockerfile that extends the original one and adds these changes. The Dockerfile can be found here. To do this, the regular image has to be built and tagged with artemis:coverage-latest.

    Additionally, we need Java in the Cypress container for the flaky test detection, so we have a special Dockerfile for the Cypress container that extends the original one and adds the Java installation. This Dockerfile can be found here.

  2. Docker Compose Changes

    The Docker Compose file for the flaky test detection is located here. This file includes some overrides for the regular Docker Compose file. The main differences are that we use the extended Dockerfiles for the Artemis and Cypress containers, and we also change the Cypress startup command to include our coverage analysis. To use the overrides, you can run the following command: docker-compose -f cypress-E2E-tests.yml -f cypress-E2E-tests-coverage-override.yml up.

This setup allows us to run the flaky test detection build plan in parallel with the regular Cypress build plan. If there is no overlap between the changed code and the files covered by failed tests, we label plan executions with the suspected-flaky label.

Artemis Deployment in Test Environment

There is another build plan on Bamboo which executes the Cypress test suite. This build plan deploys the latest Artemis executable of the develop branch on an already configured test environment (test server 3) and executes the Cypress test suite against it. This build plan is automatically executed every 8 hours and verifies that test server 3 is working properly.

Artemis Deployment on test environment for Cypress

Artemis Deployment on test environment for Cypress

The difference of this setup is that the Artemis server is deployed on a separate environment which already contains the necessary configuration files for the Artemis server to connect to the prelive system. The Docker image for the Cypress container should be exactly the same as the Cypress image used in the docker-compose file for the deployment on a Bamboo build agent.

Maintenance

The Artemis Dockerfile as well as the MySQL image are already maintained because they are used in other Artemis Docker setups. Therefore, only Cypress and the Cypress Docker image require active maintenance. Since the Cypress test suite simulates a real user, it makes sense to execute the test suite with the latest Chrome browser. The Cypress Docker image we use always has a specific Chrome version installed. Therefore, the docker-compose file as well as the build plan configuration for the Cypress tests on test server 3 should be updated every month to make sure that the latest Cypress image for the Chrome browser is used.