Blog/
Product

Rapid and Robust Deployment at Bond

Jonny Glazier
Contributor
Date Published
(
December 11, 2020
)
Read Time
(
min
)

Our approach to CI/CD balances three core tenets: stability, speed and simplicity. We’ve used these three principles to guide how we select our tools and processes.

The tools and processes that make deployment anything but arduous.

Continuous Integration/Continuous Delivery (CI/CD) matters. In a rudimentary form, it is the use of automated methods to bring code together in version control and subsequently build, test and deploy applications.

However, good CI/CD goes much deeper - it encompasses coding philosophy, tools selection and team procedures to allow us to speed up product iteration cycles and accelerate our path to product-market fit. It also allows us to facilitate more confident feature delivery through strong automated testing and frequently shipping smaller changes. This all works to reduce the scope of production bugs and the anxiety of engineers after any deployment. With effective CI/CD, our engineers are able to do what they’re best at: building products that delight our customers.

Our approach to CI/CD balances three core tenets: stability, speed and simplicity. We’ve used these three principles to guide how we select our tools and processes.

Stability

Stability is particularly critical for Bond since our customers rely on us to power critical financial products. A lack of vigilance and care from an engineering perspective can result in people suffering real-world financial damage or regulators imposing penalties on our customers or ourselves. Solid fundamentals and frameworks allow us to avoid extensive manual testing or code-freezes before releasing new versions of our products to our users.


You can see a rundown of our CI/CD process below. Once a Pull Request is merged into a main branch, a CircleCI pipeline is triggered to run automated tests and publish a build to our image repository, which gets picked up by Spinnaker for delivery to each environment. Many of these steps in this pipeline are designed to ensure the stability and performance of the products we’re releasing to our customers.

The entire process that elevates code to production

We employ two tools to maximize the confidence that we have in our releases. The first is  CircleCI, which automates our testing on every code commit. The second is Spinnaker, which controls a sequenced dependency tree for all of our deployment pipelines.

CircleCI

For our suite of microservices, prior to committing to our main branch we require a threshold of unit test coverage and that core workflows against other services are functional. This ensures not only that modular code functions as expected but also that the change agrees with the other services in the broader system. CircleCI is able to easily execute these tests and highlight the changes that need to be made as a result. Its simple yaml structure fits neatly into our existing repositories and authenticates easily with GitHub through OAuth 2.0.

It’s important to note that our automated testing isn’t a one-time investment. It's an ongoing effort. The unit tests that exist today would not be sufficient for the features we’d develop tomorrow, so no new feature is ready until unit tests are created and integrations tests pass.

Spinnaker

Spinnaker is a leading open-source delivery platform. One of its most useful bits of functionality is the enforcement of an ordered set of operations during a deployment pipeline. Health and readiness checks, which include smoke tests, must pass in pre production environments before the code in question is promoted to higher environments. This ensures that the code won’t break our application given the state of our current infrastructure. When paired with declarative Infrastructure-as-Code solutions, this greatly decreases the likelihood of unexpected issues manifesting in production environments.

Above is an example of one of our Spinnaker Pipelines. These pipelines follow operational protocols to elevate code in a secure + stable fashion


On top of this, we employ manual steps in our pipeline. Certain pushes into production environments require the manual approval of release managers through the Spinnaker Dashboard. These steps are simple and give internal constituents a chance to do manual tests on pre-production environments, which is especially valuable while we’re rapidly iterating early in our product lifecycle. The more we invest in our automated testing pipelines and the stability of our code, the less we'll need to rely on manual workflows. In the future, we’ll strive to employ automated deployment strategies like canary deployments.

Speed

Speed is foundational to almost everything in a startup, especially the delivery of new features. A fast CI/CD process allows us to iterate quickly on features, many times a day if necessary, empowering us to rapidly refine our products based on customer feedback. We continuously build customer trust by delivering on promises quickly.

Each step in the CI/CD pipelines is designed to optimize for speed. For instance, Spinnaker runs pipelines that connect to every Kubernetes environment and update the deployment manifest with the new application images. These automated builds allow us to immediately deploy any new code into all of the required environments without the need for manual oversight.

Each of our services runs on separate delivery pipelines so that we can deliver small changes to specific applications independently. Bond offers a suite of microservices to create financial products, as well as a web application which serves as a central hub for program management. These pipelines manage the delivery of all the software that comprise these offerings and the separation between the deployments allow us to make changes to one application without fear of breaking the system.

On average, it will take under 7 minutes for a freshly merged line of code to make it to our pre-production environments, after which a manual approval can immediately promote that change to production. Any bugs that may manifest are quickly stomped out, and we’re more quickly able to iterate through our products and ideas. This agile infrastructure helps any early-stage tech company on their march towards market validation.

Simplicity

When we have infrastructure that is consistent and easy-to-understand, pipelines are easy to maintain and it becomes easier to onboard new engineers to our rapidly growing team. One of the best ways to enforce simplicity is through repeatable processes. Terraform, our Infrastructure-as-Code tool, and Kustomize, our Kubernetes manifest generation tool, allow us to do exactly this.

Terraform

Terraform enables us to immediately deploy pipelines that connect to all of our environments, quickly destroy them and quickly rebuild them. In fact, all of our infrastructure is created through Terraform. This gives us the assurance that the infrastructure on production environments will behave the same as development environments.

Kustomize

Kustomize makes our Kubernetes manifests far more understandable. Manifests have a tendency to explode in size as applications become more complex, with blocks of repeated code across services. When deployed in Kubernetes, the additional metadata and annotations make the manifests even larger and harder to understand. Kustomize allows us to modularize and call on certain snippets of manifests, making edits to them to facilitate the creation of unique resources. This decomposition, just like decomposition in application code, allows us to know that our changes are applied in the right places and allows our manifests to be more accessible to all our engineers.


We use Kustomize to alter names and add annotations to a set base manifest for our services, deployments, and ingress objects


These three tenets don’t just apply to Bond’s DevOps processes, they apply to how we build products. Reference our APIs to learn how you can create products in a stable, speedy and simple manner. Also, we’re hiring!

Icon