Juggling multiple build stages and test environments with TravisCI

Note: Babbel is no longer using TravisCI, however this article may be useful for others that use Travis.

At Babbel, we employ continuous integration to detect issues early in the development process. Our CI does not only automate the build and run different test suits but also handles some tasks for documentation purposes. With so many moving parts, the setup is not trivial. We (one of the Web teams) share our TravisCI setup here on Babbel Bytes in the hope that some of you find it helpful for your own applications.

Our application and its unit tests are written in JavaScript. Additionally, we have Selenium based UI tests written in Ruby. We use TravisCI to run all our tests and build the application for deployment. After a deployment, we also need to build a so-called Storybook and upload it to AWS S3. Storybook is a UI Development Environment. Finally, we check for vulnerabilities using a service called Snyk.

---

This means there are five different job types (from now on called stages).

1. Run unit tests
2. Run UI tests
3. Build application for deployment
4. Run Snyk
5. Build and upload the storybook

We want these stages to be executed sequentially, and whenever a stage fails, we want the pipeline to be stopped immediately after it. This will save us time and TravisCI computing resources.

For running the UI tests, we are not using an external service provider (browser/mobile farm), but instead run the tests on TravisCI itself, and the test suite’s total runtime is quite long. For this reason, we cannot run the UI tests on every push/PR/merge, as this would delay our deployments. We decided to be bold and merge code first and run the UI tests afterwards, and roll back or deploy a fix in case an issue came up. Before we deploy to production, however, we still do manual testing in our staging environment for all code changes that require it.

Also – when running sequentially – the UI tests’ runtime is too long for a single TravisCI job. The job would time out before the UI tests have finished. So we decided to parallelize them. Instead of writing our own parallelization script, we found a nice solution offered by TravisCI itself, the TravisCI build stages. As TravisCI puts it:

“Build stages is a way to group jobs, and run jobs in each stage in parallel, but run one stage after another sequentially.”

This is exactly what we wanted. With this solution, we could run each of our five stages sequentially, and the UI tests (and potentially other jobs) in parallel.

And there is more! By enabling the TravisCI Conditions, it becomes possible to run the stages depending on GitHub information. We are using branch, type, and commit_message to decide when to run which stage(s).

To keep the TravisCI runtime as short as possible, we decided to neither run the UI tests, nor build the storybook, nor use Snyk in case we

• open a PR, or
• push to any branch that is not called master or integration

We do, however, run the unit tests and build the application.

When we push to master, we run all stages except the UI tests. As mentioned before, we run the UI tests after a merge because the UI test suite’s runtime is too long at the moment. Once we’ll have improved the tests’ speed, we will re-evaluate our setup.

So when do we run the UI tests? For now,

• When we push to a branch called integration
• As part of a daily TravisCI cron job
• When we specify run_tests in the last commit message of a push to any branch

To tell TravisCI that one wants to use stages, they are quite simply listed (in order of execution!) in the stages section of the travis.yml. Each stage has a name and an optional condition. Here is our stages configuration:

stages:
  - name: Unit Tests
    if: type != cron AND branch != integration
  - name: Ui tests
    if: type = cron OR branch = integration OR commit_message =~ run_tests
  - name: Deployment
    if: type != cron AND branch != integration
  - name: Snyk
    if: branch = master AND type = push
  - name: StorybookS3
    if: branch = master AND type = push

Our standard TravisCI environment (the env section of the travis.yml) is configured to run the UI tests, split up using the matrix option. The environment variables configured in env - matrix are passed to the script section that starts the UI tests.

script: ./run_ui_tests $TEST_PART

env:
  matrix:
    - TEST_PART=1
    - TEST_PART=2
    - TEST_PART=3

run_ui_tests is a simple bash file that handles the setup for the UI tests:

#!/bin/bash

TESTS_PART_1=(
 "test1.feature"
 "test2.feature"
 "test3.feature"
)
TESTS_PART_2=(
 "test4.feature"
 "test5.feature"
 "test6.feature"
)
# .. etc

for test_name in “${!TESTS_TO_EXECUTE}”
  TESTS_TO_EXECUTE=TESTS_PART_$1[@]
do
  ./ui_test_run_script "$test_name.feature"
done

This is how we parallelize the UI tests – all TravisCI env - matrix jobs are run in parallel, although there can be a limit on how many parallel TravisCI instances are allowed per repository. In fact, our project is one of the larger ones at Babbel, so in order not to block other teams, we gave ourselves a limit. This can be configured under https://travis-ci.com/YOUR_COMPANY/YOUR_REPO/settings:

We are also auto canceling jobs in case we push to a branch on which a job is currently running:

Now that we have env set but don’t want to run the UI tests in the other stages, we need to overwrite it with either another or no environment variables. This is done in the jobs - include section of the travis.yml.

jobs:
  include:
    - stage: StorybookS3
      env: # OVERWRITE WITH EMPTY
      install: install_script_for_storybook
      script: bundle exec s3_website push

    - stage: Unit tests
      env: TARGET=target_environment
      install: install_script_for_unit_tests
      script: run_script_for_unit_tests
      after_success: upload_coverage

    - stage: some other stage..

Here are the stages related parts of our .travis.yml, from top to bottom:

install:
  - # install whatever is necessary to run the UI tests

script: .run_ui_tests $TEST_PART

env:
  matrix:
    - TEST_PART=1
    - TEST_PART=2
    - [...]

# the following line is needed to enable the TravisCI build conditions
conditions: v1

stages:
  - name: Unit Tests
    if: type != cron AND branch != integration
  - name: Ui Tests
    if: type = cron OR branch = integration OR commit_message =~ run_tests
  - name: Deploy
    if: type != cron AND branch != integration
  - name: Snyk
    if: branch = master AND type = push
  - name: StorybookS3
    if: branch = master AND type = push

jobs:
  include:
    - stage: Unit Tests
      env: TARGET=target_environment
      install: ./install_unit_tests_dependencies
      script: ./run_unit_tests
      after_success: ./upload_coverage

    - stage: StorybookS3
      env: # no environment variables - overwrite with empty
      install: ./install_storybook_dependencies
      script: ./build_storybook
      after_success: ./upload_storybook

    - stage: Deploy
      name: Deploy to staging
      env: ENVIRONMENT=staging
      install: ./install_deployment_dependencies
      script: ./deploy

    - stage: Deploy
      name: Deploy to production
      env: ENVIRONMENT=production
      install: ./install_deployment_dependencies
      script: ./deploy

    - stage: Snyk
      env: # no environment variables - overwrite with empty
      install: ./install_snyk_dependencies
      script: ./run_snyk

A full run on TravisCI for the master branch, containing all build stages, looks like this:

Alternative approach for parallelizing

We realize that there is another way of parallelizing the UI tests; it can be done using only stages without env - matrix:

jobs:
  include:
    - stage: UI tests
      name: part1
      install: .install_ui_test_dependencies
      script: .run_ui_tests 1
    - stage: UI tests
      name: part2
      install: .install_ui_test_dependencies
      script: .run_ui_tests 2

We did, however, feel that this solution made the travis.yml file longer and harder to read. TravisCI build stages are still a beta feature, and matrix will probably be enabled for them sooner or later, and then we will most likely change our approach.

To sum up, in this article you have been introduced to the challenges of setting up continuous integration for a medium-sized project. We showed you how to group jobs in stages, run tasks within one stage in parallel, and run stages conditionally. We hope you find our ideas useful and are looking forward to hearing your thoughts!

The author would like to thank Annalise Nurme for providing the drawing of the juggler.

Photo by Alex Kotliarskyi on Unsplash