Skip to main contentIBM Garage Event-Driven Reference Architecture - Reefer Container Shipment reference implementation


The implemented DevOps pipelines for the Reefer Container Shipment solution reference implementation of the Event-Driven Reference Architecture. This chapter focuses on the GitOps capability implemented in the reference implementation.


Our Continuous Integration (CI) approach is one of “zero-infrastructure overhead”. To accomplish this goal, we utilize GitHub Actions to build and push a microservice’s associated container images to Docker Hub for public consumption. The GitHub Actions workflows are defined in the owning repository’s .github/workflows/dockerbuild.yaml file.

Our GitOps approach, the topic of this chapter, focuses on a single-repository, environment-per-subdirectory model which can be forked and cloned to replicate deployments to other clusters and environments. The reference implementation utilizes Kustomize as its templating technology to utilize the app-deploy.yaml files, provided by each individual microservice, as a base and then layer in environment-specific configuration and credentials as needed.

Our Continuous Delivery (CD) approach focuses on a GitOps-based delivery model, using a GitHub repository as a single source of truth for the deployment, management, and operations of our running application components. In this model, we have the flexibility to use multiple open-source technologies to apply the single source of truth from a given GitHub repository onto a desired cluster environment.


If you are unfamiliar with GitOps as a practice, additional details around the background of GitOps and how it differs from traditional deployment models can be found in this blog post from WeaveWorks.

Reference Implementation environments

We have provided a few sample environments to get started.

  • dev environment: This environment is deployable to any Kubernetes or OCP cluster and provides its own dedicated backing services.

  • example-credentials environment: This example configures the microservices to connect to a Postgres database, using the necessary credentials & the Postgres service’s self-signed certificate, and to Event Streams, using only an API key and no additional certificates (as the remote service utilizes a certificate that is signed by a public CA). Both of these backing services are consumed from off-cluster service providers.

  • example-es-truststore environment: This example configures the microservices to connect to a Postgres database, using the necessary credentials & the Postgres service’s self-signed certificate, and to Event Streams, using an API key and the Event Streams server’s self-signed certificate. The Event Streams certificate is provided as a Java truststore to the Java-based microservices and a PEM file to the non-Java-based microservices.

  • eda-integration-XXXX.YY environment: This environment is our squad’s integration environment for a fully-automated end-to-end build pipeline, with the target environment being a Red Hat OpenShift Container Platform cluster running on IBM Cloud.

Environment overview

The GitOps environments are defined in the environments sub-directory, allowing for multiple environments (phases, stages, teams, etc) to be deployed from a single GitHub repository. This approach is the simplest for a single team to maintain.

There are several approaches to maintaining multiple GitOps environments, which offer tradeoffs between simplicity and access control:

  • Single GitOps repository, single branch, one subdirectory per environment: This is our chosen approach and is simple to manage. All collaborators have the same level of control over each environment.
  • Single GitOps repository, one branch per environment: In this approach, the environments/ directory in each branch contains a single environment. This allows some level of access control to be set per environment, for example via branch protection rules.
  • Multiple GitOps repositories: In this approach, each repository represents a single environment. This allows access control per environment at the repository level, for example to protect a production environment from accidental modification.

Each environment has the following default directory structure:

├── dev
│   ├── apps
│   │   └── refarch-kc
│   │   ├── base
│   │   │   └── kustomization.yaml
│   │   ├── kustomization.yaml
│   │   └── overlays
│   │   └── kustomization.yaml

The apps/refarch-kc subdirectory

This subdirectory aggregates together all the microservices that comprise the refarch-kc reference implementation. It also applies the common label of refarch-kc to all the deployed components for easier management.

In the dev environment, this subdirectory also contains the KafkaTopics CRDs to automatically create the required Kafka Topics inside the Kafka cluster (as the dev environments utilizes the Strimzi Operator for cluster management).

The env subdirectory

This subdirectory provides all the necessary ServiceAccount, ConfigsMap and Secret configuration. Depending upon your environment, most of your customization of a new environment will be done in this subdirectory.

The infrastructure subdirectory

This optional subdirectory provides the ability to create a Kubernetes Namespace / OpenShift Project and any necessary backing services ahead of application microservice deployment. This is only used in the dev environment, as a way to demonstrate the ability to create a fully-managed environment from scratch.

The services subdirectory

This subdirectory provides all the necessary deployment configurations for the AppsodyApplication and OpenLiberty applications that make up the reference implementation. The /base/config/app-deploy.yaml file in each services subdirectory is automatically maintained by our GitOps pipelines.

The main area of customization for individual environments will be done in each service’s overlays subdirectory, as this subdirectory provides the ability to modify and update the base app-deploy.yaml file at deploy time.

Service overview

In this section, we will walk through the Fleet microservice’s /overlays subdirectory to describe what operations can be performed on the base app-deploy.yaml through the Kustomize framework.

Adding Kafka credentials

The majority of our Kafka-based microservices look for configuration and credential information defined as environment variables upon startup. This makes it easy to configure, update, and manage running application instances when configuration and credential information changes.

To add the necessary credential information to a given microservice at deploy time, we use the /services/fleetms/overlays/appsody-env-patch.yaml overlay to append KAFKA_USER and KAFKA_PASSWORD environment variables to list of environment variables already defined in the base app-deploy.yaml file.

Adding Kafka certificates

To add the required SSL/TLS certificates necessary for connecting to secured Apache Kafka / IBM Event Streams systems, you must inject the certificate file dynamically at deploy time. This process can be completed in three steps:

  1. Create a Kubernetes Secret containing the Certificate file
  2. Create a Kubernetes Volume in the Kubernetes Deployment, AppsodyApplication, or OpenLibertyApplication specification, based on the Secret
  3. Create a Kubernetes VolumeMount in the Kubernetes Deployment, AppsodyApplication, or OpenLibertyApplication specification, mounting the previously-defined Volume to a known location inside of a Pod.

To perform these steps in our GitOps implementation:

  1. Step 1 is achieved through the artifacts defined in the /env/base/credentials subdirectory of either of the example environments.
  2. Step 2 is achieved through the /services/fleetms/overlays/appsody-volumes.yaml overlay.
  3. Step 3 is achieved through the /services/fleetms/overlays/appsody-volume-mounts.yaml overlay and the inclusion of the well-known location of the injected Certificate via the /services/fleetms/overlays/appsody-env-patch.yaml overlay.

Creating a new environment

To create a new environment, you will want to use one of the environments described above as a template and refine it as needed.

  1. Fork this repository.
  2. Copy the desired environment to a new subdirectory in the /environments directory.
  3. Update the necessary overlays/kustomization.yaml files and referenced files to suit your environment, removing unneccessary modifications and adding in any new requirements.
  4. Manually validate the environment via a kubectl kustomize environments/__your-new-environment__
  5. Commit and push your changes to your forked repository
  6. Follow along with our Continuous Delivery chapter to configure ArgoCD to deploy the environment.

Next steps

For details on how the Continuous Delivery implementation leverages the GitOps artifacts, reference the peer Continuous Delivery (CD) chapter of this manual.