Travis CI #

Standards, values, and other information relevant to the NYPL Engineering Team.

Travis CI

What does Travis CI do?

CI: Continuous Integration

From Travis CI documentation

Continuous Integration is the practice of merging in small code changes frequently - rather than merging in a large change at the end of a development cycle. The goal is to build healthier software by developing and testing in smaller increments.

Process

In NYPL Digital’s CI/CD stack, when a pull request is merging with, e.g., development branch, a series of steps is taken to build, test, and deploy to the branch.

Travis CI is a tool with options to deploy to our services of choice, e.g. AWS Elastic Beanstalk.

.travis.yml

Travis CI looks for a configuration file named .travis.yml to receive instructions on how to build, test, and deploy an application. This means the following:

The file MUST be located at the root level of the code repository, such as NYPL/staff-picks/.travis.yml.
The file MUST be named .travis.yml, with a leading period in the file name.

Build

A basic build on Travis CI consists of the following steps

install: installs required dependencies for the application
script: runs the build script(s) for the application

There are many steps in between the two configuration stages mentioned above that can be added for further customization. Here’s a list of frequently used Travis CI configuration stages on NYPL applications, from top to bottom of .travis.yml file:

language: Programming language of the code repository
Runtime environment version managers such as rvm for ruby, node_js for javascript with node_js, etc.
sudo to indicate whether sudo privileges are required.
before_install to include any applications that are needed to build and/or test an application, but are not included as part of the standard virtualization machine generated by Travis.
addons is similar to before_install, for quick shortcuts if you already know which APT package you want to install
cache to cache content that does not often change, to speed up your build process, e.g. node_modules directory from node apps.
install: Same description as mentioned above.
before_script: Install dependencies required to build the application
script: Same description as mentioned above.
before_deploy: Used for user-friendly messages to notify user of deployment.
deploy: Sets up watched branches and AWS credentials needed to deploy to, e.g. Elastic Beanstalk
after_deploy: Used for user-friendly messages to notify user deployment is successfully triggered.

Test

Testing is often run during the script configuration stage as part of the build.

Deploy

Travis has a deploy configuration stage, with a list of preconfigured providers you can use for deployments.

For new Travis users:

Sign in to travis-ci.com using your Github account
Create a personal access token on Github and give it the scope listed here (make sure you use the travis-ci.com scope, NOT the travis-ci.org scope)
From the commandline run travis login --pro --github-token <your_token>

To configure Travis to deploy:

From the root directory of your app

Find the encrypted command stored [in nypl-digital-dev parameter store](https://us-east-1.console.aws.amazon.com/systems-manager/parameters/production/travis/add_aws/description?region=us-east-1&tab=Table#list_parameter_filters=Name:Contains:travis) which will automatically add encrypted credentials for AWS_ACCESS_KEY_ID_DEVELOPMENT, AWS_SECRET_ACCESS_KEY_DEVELOPMENT, AWS_SECRET_ACCESS_KEY_QA, AWS_SECRET_ACCESS_KEY_PRODUCTION, AWS_ACCESS_KEY_ID_QA, and AWS_ACCESS_KEY_ID_PRODUCTION to `.travis.yml`. (If this is a new Travis integration - or new to you - [ensure the repo is locally associated with the right Travis endpoint](#failure-to-decrypt-environmental-variables).)
Run the command in the root directory of your app

Add deploy entries to .travis.yml for each deploy hook. Beanstalk example: ```yml deploy:
- provider: elasticbeanstalk skip_cleanup: true access_key_id: “$AWS_ACCESS_KEY_ID_DEVELOPMENT” secret_access_key: “$AWS_SECRET_ACCESS_KEY_DEVELOPMENT” region: us-east-1 app: discovery-api # This is the name of the app env: discovery-api-dev # This is the specific deployment bucket_name: elasticbeanstalk-us-east-1-224280085904 # This is common across EB deployments in a given account bucket_path: discovery-api-dev # Conventionally this should equal the env name on: repo: NYPL-discovery/discovery-api branch: development # specify the branch for travis to watch
```
Lambda example:
```yml
  deploy:
          - provider: script
  skip_cleanup: true
  script: npm run deploy-development
  on:
  branch: development
```

Building PRs

The Travis “Settings” page for a given repo allows you to enable two kinds of builds:

“Build pushed branches”: Create a build for all pushed branches and commits (i.e. build when “my-feature” branch is pushed, build when “development” is updated in any way)
“Build pushed pull requests”: Create a build any time a PR is created by checking out the target branch, locally merging the PR branch into it, and running the test suite against the result.

Both options are enabled by default. The former serves as an essential baseline check. The latter adds assurance that the result of the merge will continue to pass.

Note that the deploy section is not evaluated in the context of a pushed pull request.

Troubleshooting

Chrome

When using Chrome to conduct tests such as with karma, Chrome is not installed by default on the virtualization environment.

The idea is to install Chrome as an apt addon and enabling display within the Travis virtualization machine so Chrome can run with a monitor on the background.

addons:
  chrome: stable  # have Travis install chrome stable.
  ...
before_script:
  - export DISPLAY=:99.0  # Export display to Travis
  - sh -e /etc/init.d/xvfb start  # Start graphical operations

Headless Chrome

The installation of Chrome would still be required even if chrome-headless is used by karma or other test suites.

Failure to Decrypt Environmental Variables

If it seems your encrypted environmental variables aren’t being set in builds and/or you see something like the following in your build log, this section is for you:

Setting environment variables from .travis.yml
$ export CpxjaQ=[secure]
...

One reason Travis will fail to properly decrypt environmental variables is if there’s a mismatch between the Travis endpoint used to encrypt the values and the Travis endpoint used to run builds. Travis is in the process of moving from travis-ci.org to travis-ci.com. All builds will eventually migrate to the latter. Our repos are currently split between them, with most of the newer Travis integrations existing on .com.

To determine whether .org or .com is correct for a given repo, simply check the Travis URL of any attempted build.

To determine which endpoint is actually used to encrypt on your local machine, check ~/.travis/config.yml. For example:

endpoints:
  https://api.travis-ci.org/:
    access_token: [snip]
  https://api.travis-ci.com/:
    access_token: [snip]
repos:
  NYPL-discovery/discovery-front-end:
    endpoint: https://api.travis-ci.org/
  NYPL-discovery/schemaservice:
    endpoint: https://api.travis-ci.com/

The endpoints section indicates I’ve authenticated against both .org and .com at some point. Under repos, we see that builds for discovery-front-end are expected to be run on .org whereas builds for schemaservice are expected to be run on .com. Consequently any time I run travis encrypt ... locally for either of those repos, the encryption will be tied to those endpoints.

When you run travis encrypt ..., the command makes a guess about whether the integration will be run on .org or .com. It makes that guess based on the contents of the repos section described above and, failing a match there, falls back on your default endpoint (see notes on travis endpoint ... below). When travis encrypt defaults to the wrong endpoint, correct that by re-running your travis encrypt ... command with ` –com or –org at the end. (This will add the missing entry to your ~/.travis/config.yml` for subsequent calls.)

To ensure all new Travis integrations default to a .com association for you going forward:

travis endpoint --com --set-default

$TRAVIS_BRANCH

Many apps use $TRAVIS_BRANCH environmental variable in .travis.yml to inspect the current branch. This allows us to DRY our deployment code by using only a single deploy provider that uses $TRAVIS_BRANCH to control the place deployed to. Example

Note that Travis will not attempt to deploy anything in the context of a PR build. For example, when Travis checks out “development” to locally merge “my-feature” and run tests, Travis will not thereafter attempt to deploy to “development” even though $TRAVIS_BRANCH is “development”; It skips deploy altogether for PR builds.

Recycled [failed] builds appearing in PRs

When creating a PR in a repo with “Build pushed pull requests” enabled, Travis will show two builds: One for the “push” and one for the PR. Typically one creates a PR to merge a recently pushed branch, so both builds will appear “pending”. If the source branch was updated a while ago, the build status of the “push” may already be visible when the PR is created. You’ll therefore see one pending “PR” build and one “push” build showing the status of the last push to that branch. If the last push to the source branch was a failure, that build will appear as a failure in your PR. This is sensible and good, but may be distressing because it suggests your PR triggered the failure, but in fact the failure may be old and/or unrelated.

A concrete example: When creating a PR to merge “development” into “qa”, we immediately saw that the “push” build was a failure. On clicking through, the failure was found to be in the deployment stage. This was distressing because it initially appeared a deployment was being executed in response to a PR, but in fact the deployment was triggered by the original push to “development” (i.e. we simply hadn’t noticed the original failure.)