One scenario that bears mentioning is if the CI toolchain was deployed to ICP, typically using the community Helm chart. For example, In the BlueCompute case, this was done using a containerized Jenkins instance. All of the stages in the pipelines run as containers in Kubernetes using the Kubernetes plugin.

https://github.com/ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes

The best practice around Jenkins is for the pipelines themselves to be written using a declarative Jenkinsfile stored with the code. The pipeline logic is stored with the application source code and itself treated as part of the application. In BlueCompute, Jenkinsfiles are stored with each microservice, so we only needed to create an instance of Jenkins in Openshift and import the pipelines to get our builds to work.

Openshift has both ephemeral and persistent Jenkins in the catalog which is very comparable to the Jenkins Helm chart. This instance of Jenkins will automatically install the Kubernetes and Openshift client plugins, so Jenkinsfile that uses podTemplates that spin up containers to run stages in the pipeline should "just work".

As pipeline stages are run in containers, there are security issues when a particular stage attempts to run a container that requires root access, mounts a hostpath, etc, just like application containers. Most runtime build images don’t need to run as root (e.g. maven, gradle, etc), which means they should run just fine in Openshift.

One security problem when using Kubernetes plugin is how to build the container image itself, which must run in a container. In Openshift this is further complicated that the default restricted SCC disallows host mounting the docker socket, running as root, or running in privileged mode without changing the SCC. The community is still attempting to resolve this problem, since traditionally the Docker tool requires root and many of the functions involved in building a container image requires various elevated privileges.

There are a few projects at various stages in development as of this writing that can build container images without docker, which run fine outside of a container, but none are perfect inside of a container.

When using these tools, we can relax the security constraints on them by adding the SCC to the jenkins service account in the namespace where the pod runs. For example, kaniko only requires root access, so we can simply add the anyuid scc to enable kaniko:

Note the security implications of the above. While containers are running with elevated privileges, any other workload running on the same worker node may be vulnerable, as Jenkins build stages may run as root. Additionally since the developer controls the commands used in the pipeline stage, this essentially gives developers root access on the worker node the build runs on. In shared clusters, it may make sense to use a nodeSelector on all projects where jenkins will run to isolate jenkins workloads to just a few nodes: https://docs.openshift.com/container-platform/3.11/admin_guide/managing_projects.html#setting-the-project-wide-node-selector

If we are running the Jenkins pipeline stages in Jenkins on Openshift, we can leverage the Openshift Jenkins client plugin and the Openshift BuildConfig to build the container image, which provides a controlled environment for producing the container image without exposing root access on any worker nodes. By providing just a Dockerfile and a build context, Openshift will build and push the resulting image to the Openshift private registry and track it using an ImageStream, and we can then provide an additional stage using skopeo to push this to an external registry. BuildConfig limits the amount of damage the developer can do because they are not explicitly running commands as root.

We have posted an example pipeline we used at a customer that leverages this at the following git repository:

https://github.com/jkwong888/liberty-hello-world-openshift/blob/master/Jenkinsfile