DevOps Pipelines: Build and Deploy

Posted by Darin Pantley on December 16, 2017

Getting Started: A Basic Pipeline

Every time software is updated, it needs to be built and deployed. In other words, source code is useful when it is presented to end users in its final usable form. This is broadly true for modern web apps that run continuously, CLI tools used by developers and automation, compiled languages, scripted languages, and many other types of software.

If you don’t already have a CI/CD pipeline created by a team of dedicated DevOps engineers, you might roll your own pipeline that looks something like this:

  1. Build
    • compile the code, run unit tests, etc.
  2. Deploy
    • initialize infrastructure in the cloud, launch your app, etc.

For the purposes of this article, I will assume that you are broadly familiar with the concept of building a CI/CD pipeline. Rather than talk about its necessity, let’s discuss the general structure and the different types of functionality that can be added to a pipeline.

Making Progress: A More Complex Pipeline

Some automation may need to run before or after the Build and Deploy stages. To account for this, we can split the pipeline into more granular stages:

  • Pre-Build
  • Build
  • Post-Build
  • Pre-Deploy
  • Deploy
  • Post-Deploy

You can name these stages whatever you like. For example, you may want to call your pre-build stage “Static Analysis” and your post-build stage “Bake”. Regardless of what you call each stage, it’s nice to have a logical place to put new automation when the need arises.

Pre-Build

This is a good place to perform static analysis on your source code. No code has been built yet, so your only inputs to this stage are your source code and whatever tools you can find to analyze the source code.

  • Static Analysis
    • Bugs
    • Code coverage
    • Cyclomatic complexity
    • Linting
    • Performance
    • Security

Build

Use this stage to convert your code into executable artifacts.

  • Build the source code
    • Incremental builds
    • Distributed builds

Post-Build

Once the code is built, you can run it and see if it works. It’s also a good place to run complex tools that operate on the built artifacts and executables.

  • Dynamic Analysis
    • Resource usage
    • Performance
  • Tests
    • Unit tests
    • Integration tests

Pre-Deploy

Use this stage to prepare your app for deployment and prepare infrastructure where it can run.

  • Package
    • Create deployable artifact
    • Sanity tests

Deploy

In this stage you have two competing desires:

  1. Give many people access to the app.
  2. Find problems before they impact many people.

Deploy: Strategies

It is possible to give everyone access to your new changes and find the problems before they impact people if you plan out a careful deployment strategy. This typically involves using canary deployments, blue/green deployments, or other common deployment strategies. The general idea is to do a controlled rollout, starting small but growing to include more and more users as we gain confidence in the changes.

Here are some example canaries:

  1. Internal automated test environments.
    • This should be a copy of your typical production environment.
  2. Internal production environments.
    • Dogfood your own software.
  3. Small subsets of your end users.
    • Early adopters and people who opt-in to beta programs make ideal candidates, but don’t be afraid to choose any small subset of your end users.
    • The purpose of this is to run real software in a real environment with real end users in a way that won’t cause big problems if you’ve introduced a bug.
  4. Larger subsets of your end users.
    • If you’ve already deployed to the above environments and haven’t noticed any problems yet, then you’re ready to deploy to a larger group of users.
  5. All end users.
    • Eventually your software should be deployed everywhere.

Note that this can also be leveraged to provide A/B testing to determine which changes are more favorable for end users.

Deploy: Tests

For each deployment scenario, run tests and monitor the health of your app. Once you’re confident that a deployment has succeeded, you can proceed to the next broader level of deployments.

  • Health checks
  • Functional tests
  • Load tests
  • Penetration tests
  • Performance tests

Deploy: Reacting to Failure

When deployed software fails, it may be due to bugs in your app, bugs in another app, or environmental issues.

  • Compare old deployments to new deployments and see if they get the same results.
  • Roll back to a previous working version.

Use load balancers to provide high availability and facilitate automatic scaling.

  • Add or remove running instances of your app to react to changes in user load.
  • Automatically restart your app on failure.
  • Automatically redirect traffic to other healthier regions when the current region experiences severe failures.

Post-Deploy

Once your app is running, its life has really just begun. You’ll need to keep an eye on it to make sure it’s behaving well. Give it a poke once in a while to test its limits.

  • Monitoring
    • Health
    • Runtime statistics
    • Usage telemetry
  • Chaos
    • Chaos engineering will help ensure that your app and infrastructure are ready for anything at any time.