ICP to OpenShift migration guide

ICP can be deployed on (Linux) x86_64, Power (ppc64le) and IBM Z and LinuxOne. OpenShift now can run x86_64, Power and IBM Z hardware. Each has its own sizing recommendation in terms of CPU, memory and disk space. You can reference the system requirement for both below:

ICP (3.2) hardware requirement guide - https://www.ibm.com/support/knowledgecenter/SSBS6K_3.2.0/supported_system_config/hardware_reqs.html

OpenShift (3.11) hardware requirement - https://docs.OpenShift.com/container-platform/3.11/install/prerequisites.html#hardware

OpenShift 4.2 system requirement - https://docs.openshift.com/container-platform/4.2/welcome/oce_about.html

Both ICP and OpenShift can run on Hypervisors like VMware, OpenStack and Hyper-V in a private cloud environment. ICP is also supported on IBM PowerVC.

Both ICP and OpenShift can run on public or private IaaS. In public. We have tested ICP on IBM Cloud, Azure, AWS, GCP, and Huawei Cloud. On the other hand, we have tested OpenShift on IBM Cloud, Azure, AWS.

For OpenShift on public cloud, there are potentially 3 offering:

ICP doesn’t have a managed edition.

This is where you should pay the most attention when migrating from ICP.

Both platforms can only run on top of Linux OS. ICP supports Red Hat Enterprise Linux (RHEL) 7.3, 7.4 and 7.5, Ubuntu 18.04 LTS and 16.04 LTS, SUSE Linux Enterprise (SLES) 12. While OpenShift supports only RHEL 7.4 or later in 3.x, or Red Hat Enterprise Linux CoreOS (RHCOS) in release 4.x. In OpenShift Container Platform 4.1, you must use RHCOS for all masters, but you can use Red Hat Enterprise Linux (RHEL) as the operating system for compute, or worker, machines. If you choose to use RHEL workers, you must perform more system maintenance than if you use RHCOS for all of the cluster machines.

What does this mean is that you need to switch RHEL or RHCOS when migrating ICP running on Ubuntu or Suse Linux. Most of this is infrastructure related Ops activity.

OpenShift requires SELinux to be enforcing'' and targeted'' mode. When containers are run, the container image’s filesystem is labeled using a random label and the container processes are labeled the same way, so that only the container processes can access its own filesystem and no other processes. Any mounted filesystems (secrets, configmaps, or volumes) will have an SELinux policy applied to them to allow the container to read and write to them.

OpenShift SecurityContextContsraints (SCC) is the pre-cursor to the PodSecurityPolicy (PSP) in upstream community Kubernetes. As such, a lot of the properties of the PSP come directly from the SCC. These objects are cluster-scoped policies designed to limit the access of containers to the host kernel. Most containers do not need to privileged access to the host and should as a best practice not depend on the uid of the user owning the container process. However, many containers on DockerHub and even some IBM middleware require running as root or some other capabilities in order to function.

One important thing to note is that while the PodSecurityPolicy objects can be created in OpenShift, the platform will ignore these objects and only enforces the SecurityContextConstraints objects. OpenShift ships with some out of the box SCCs, the default restricted'' policy is the most restrictive, and the privileged'' policy is the most open.

One very large difference is that the default policy in OpenShift will generate random a uid/gid from a range for the container process to run as (the restricted'' policy), and if your container depends on a specific uid/gid being set, the container may not run. One common example is if container requires reads or writes to the local filesystem as a specific user. In this case, the nonroot'' SCC seems to match the ``ibm-restricted-psp'' default policy that ICP ships with.

Here is a comparison of the out-of-box SCCs to those shipped with ICP, as well as some brief comments:

In order for a pod to be able to run with additional access to the host system, it’s necessary to apply the SCC to the service account the pod executes as. One subtle difference between SCC and PSP is the RBAC around it; SCCs have a users'' property that lists the entities allowed to use the SCC while PSPs are controlled with roles and rolebindings. You can use the following command to apply the SCC to a service account, which under the covers adds the service accounts to the users'' property of the SCC.

OpenShift supports one or more Identity Providers as user directory sources for authentication. As OpenShift is a development platform, the default behavior is that any user that can authenticate to OpenShift is able to create a project (mappingMethod claim''). This behavior can be changed during installation or after installation by using mappingMethod lookup'', the downside is that the administrator must manually add user resources to OpenShift before they will be authorized to use the platform. https://docs.OpenShift.com/container-platform/3.11/install_config/configuring_authentication.html#LookupMappingMethod for more information.

As Kubernetes RBAC was submitted upstream by Red Hat from OpenShift features, much of the RBAC in ICP is largely the same in ICP and OpenShift. Roles and ClusterRoles are groups of permissions on objects in the Kubernetes API. RoleBindings and ClusterRoleBindings are objects that bind roles to identities to access those permissions. Users, groups, and service accounts may have multiple role bindings which aggregated together gives them an access list of parts of the platform they may access.

One shortcut around assigning roles/cluster roles to users exists in the oc CLI, which under the covers creates a RoleBinding or ClusterRoleBinding, instead of the awkward kubectl create rolebinding'' and kubectl create clusterrolebinding'' commands:

The default networking implementation in OpenShift is the OpenShift SDN.

https://docs.OpenShift.com/container-platform/3.11/architecture/networking/sdn.html

OpenShift SDN has with three different plugins that provide different levels of network isolation between projects:

When installing OpenShift, Red Hat recommends always installing using the ovs-networkpolicy plugin which provides near parity with ICP feature with Calico. To use this, add the following parameter to the ansible hosts file before installation:

os_sdn_network_plugin_name='redhat/OpenShift-ovs-multitenant'

Note that it’s possible to run Calico on OpenShift instead of Openshfit SDN; however Red Hat does not support this directly and the client will need to purchase support directly from Tigera. The list of additional vendor-supported network plugins are available here:

https://docs.OpenShift.com/container-platform/3.11/install_config/configuring_sdn.html#admin-guide-configuring-sdn-available-sdn-providers

OpenShift SDN networking components live in the openshift-sdn project in OpenShift, and consist of two daemonsets, ovs and sdn.

ovs is a containerized version of Open vSwitch which is an open source SDN software used most commonly in OpenStack. This will manage a bridge device, vxlan tunnel device for the pod network, and all of the virtual ethernet devices (veths) for each pod as they are created and destroyed.

sdn is a component used to program openvswitch by synchronizing routes to the other worker nodes and any cluster IP services created in the cluster. The routes are programmed as open vswitch flows and the cluster IPs are configured using netfilter (iptables) rules.

To dump the flows for debugging or informational purposes, you may install the `openvswitch'' package on any cluster node, and use `ovs-ofctl to view the flow table. See https://docs.OpenShift.com/enterprise/3.1/admin_guide/sdn_troubleshooting.html#debugging-local-networking for more information. This output is helpful to understand how pod traffic is forwarded.

In contrast to ICP/Calico, which uses a single controller pod running on the master nodes to orchestrate subnet selection, routes and network policy rules, and a daemonset `calico-node'' running across each cluster node to program iptables rules and do route propagation. In ICP/Calico, the `kube-proxy container running on every node programs the cluster IPs in iptables rules instead of the calico-node pod.

In both ICP and Calico cases, the daemonset runs as a privileged container on each host in order to have access to the host network.

As in standard Kubernetes, both OpenShift and ICP have a pod overlay network where address space is defined for pods, and pod IP addresses are drawn from subnets selected from this address space. In ICP this was defined using the `network_cidr'' property in the installation config.yaml. OpenShift also has the same concept, where the cluster network CIDR defined in `osm_cluster_network_cidr in the ansible hosts file, the default is 10.128.0.0/14. You can view the subnet in the clusternetwork custom resource in OpenShift (oc get clusternetwork).

Every node in the cluster will receive a slice'' of this address space. One additional parameter in OpenShift is the osm_host_subnet_length, which defines the size of the subnets assigned to each node in the cluster where pods running on them will be assigned IP addresses from. In ICP, Calico automatically selected this size based on the number of nodes in the cluster and the size of the pod network, and was able to resize and steal'' subnets from other nodes when particular worker nodes exhausted their pool. In OpenShift this is a static length. The default value of this is 9, which indicates that every worker node will get 32-9=23 bits of subnet space (i.e. a /23 subnet, or 512 IP addresses). The assigned host subnets are stored in the hostsubnets Kubernetes custom resource (oc get hostsubnets). It’s important to select a subnet length that will satisfy both the number of worker nodes and the expected number of pods on each worker node in the cluster.

Like in ICP, there is an additional `service network'' overlay network, which is a non-overlapping address space with the pod network that ClusterIP services are defined on. In OpenShift the installation parameter for this is `openshift_portal_net.

In ICP, Calico propagated routes using a node-to-node mesh where every worker node became a ``router'' for its assigned subnet on the pod network and the routes were communicated using border gateway protocol (BGP). Since BGP is a standard protocol used on the internet, it was possible for non-cluster nodes to join the peer-to-peer mesh and the routes to be propagated outside of the cluster and potentially gain some visibility into the pod network with external tools. However, because of the node-to-node mesh there can be scalability issues when the cluster becomes very large, BGP route reflectors could be used to propagate routes instead.

In OpenShift, the routes are stored in Kubernetes resources and the ``sdn'' DaemonSet programs the routes on each cluster node as flows in the local openvswitch tables. There is a bridge interface on each node that all pods receive a port on, and a tunnel interface where all outbound pod network traffic is sent when the destination pod is not running on the local node.

The following documentation helps to understand the network flows:

https://docs.OpenShift.com/container-platform/3.11/architecture/networking/sdn.html#sdn-packet-flow

In contrast to ICP and Calico’s usage of iptables rules, OpenShift SDN uses VXLAN to perform project-level isolation. Every project is assigned a Virtual Network Identifier (VNID), and as traffic leaves the Open vSwitch tunnel, the VNID is added to the outgoing packet. When traffic reaches the destination, if the worker node does not have a policy (either the same VNID, or an explicit Open vSwitch flow from a Network Policy) that allows the traffic, it is dropped. As mentioned earlier the default'' namespace runs the router and registry and as such, every project is allowed to access this project, which is given the special VNID 0. It’s important for administrators not to expose default'' to users to deploy pods in general as all projects in the cluster will have network access to it.

You can read more details here:

https://docs.OpenShift.com/container-platform/3.11/architecture/networking/sdn.html#network-isolation-multitenant

In some environments, OpenShift may run on top of infrastructure that already uses VXLAN for isolation (such as VMware and NSX) and the VXLAN port used must be changed due to conflicts. This can be done by following the steps documented here:

https://docs.OpenShift.com/container-platform/3.11/install_config/configuring_sdn.html#config-changing-vxlan-port-for-cluster-network

NetworkPolicy is largely the same in OpenShift as it is in ICP. There is one difference in that OpenShift only supports ingress NetworkPolicy, so network policies with egress rules do not work and egress network policy is controlled using a separate EgressNetworkPolicy object.

NetworkPolicy objects in OpenShift result in flow rules in Open vSwitch, and if using a podSelector to match pods, the more pods that match the rule, the more rules are created, which may cause some scalability issues. See documentation for an explanation:

https://docs.OpenShift.com/container-platform/3.11/admin_guide/managing_networking.html#admin-guide-networking-using-networkpolicy-efficiently

As mentioned in previous section, the OpenShift EgressNetworkPolicy is a separate object used to control egress traffic from pods to external subnets. These are implemented at Layer 3 in openflow table rules. The destinations may also be DNS names, but these are implemented using a DNS lookup of the name and the subsequent rules on the resolved IP address for the DNS record’s TTL. You can see more information in the documentation here: https://docs.OpenShift.com/container-platform/3.11/admin_guide/managing_networking.html#admin-guide-limit-pod-access-egress

OpenShift has an object that allows all egress to a particular external service go through a single node, called EgressRouter. This allows traffic coming from the cluster to an external service appear from a static IP and allows operations to whitelist that router. See: https://docs.OpenShift.com/container-platform/3.11/admin_guide/managing_networking.html#admin-guide-limit-pod-access-egress-router

ICP runs a DaemonSet across the masters containing CoreDNS for cluster DNS lookup and name resolution. DNS was only available inside of pods, as the kubelet would set each pod’s /etc/resolv.conf to point at the service IP address of the CoreDNS pod, and the host’s /etc/resolv.conf is used for upstream name resolution.

OpenShift 3.11 implements DNS slightly differently: SkyDNS runs on every node and is embedded within the atomic-OpenShift-node service listening on port 53. This node will sync service names and endpoints retrieved from etcd to the local SkyDNS. Every node in the cluster will have its /etc/resolv.conf rewritten to point at the local copy of SkyDNS. All pods will also have their /etc/resolv.conf rewritten to point at the IP address of the local host. This means that service names (using FQDN of the cluster internal domain) are resolvable even from cluster nodes.

OpenShift will not start if NetworkManager is not enabled on all nodes. Make sure that NetworkManager is managing all interfaces (NM_CONTROLLED=yes in /etc/sysconfig/network-scripts/ifcfg-eth*). A script that runs when NetworkManager brings up the interface will rewrite the local /etc/resolv.conf to point at SkyDNS; the upstream DNS servers are stored in /etc/origin/node/resolv.conf.

See the documentation for more information:

https://docs.OpenShift.com/container-platform/3.11/architecture/networking/networking.html#architecture-additional-concepts-OpenShift-dns

Note that OpenShift 4.x implements this differently and has moved to the more familiar CoreDNS.

In order to get external cluster traffic into the cluster, ICP used the Proxy Nodes which run an nginx-based ingress controller. Ingress resources stored in Kubernetes were used to program the nginx configuration to accept Layer-7 traffic based on specific rules, and could leverage certain nginx features like path-based rewrites and TLS termination using annotations on the ingress resource.

In OpenShift, there is a similar component running on the infra'' nodes called the Router. This is an HAProxy container, and runs in the special default'' project that all projects should have access to. OpenShift uses a special Route'' object that pre-dates Ingress'' resources in Kubernetes, which can be used to expose Layer 7 traffic, terminate TLS. There are a few more options that are exposed as first-class properties of Routes such as being able to passthrough TLS connections or re-encrypt them.

In later versions of OpenShift (3.10+), the router is able to translate Ingress'' objects to Routes''. However, HAProxy is not as feature-rich as nginx and as such some features in the ICP ingress controller are not available using OpenShift routes, most notably path-based rewrites. A workaround is to run a standalone nginx controller that can perform these rewrites as needed in each project, and expose that using through the OpenShift router.

When OpenShift is installed, it requires a wildcard domain pointing at the IP address or load balancer in front of the nodes where the router is installed (OpenShift_hosted_registry_routehost). All routes will by default be given a DNS name like <route-name>-<project-name>.<app-subdomain>.

More documentation about the default HAProxy router, including some advanced use cases like router sharding (which is similar to the ICP isolated proxy use case) is here: https://docs.OpenShift.com/container-platform/3.11/install_config/router/default_haproxy_router.html#install-config-router-default-haproxy

Note that like ICP, there is an F5 BIGIP controller for OpenShift where a controller is able to program an F5 appliance through the API in response to Kubernetes resources. See: https://clouddocs.f5.com/containers/v2/OpenShift/

We mentioned the different options to access ICP and OpenShift in early chapters. From operation perspective either manual or automated, the command line tools (cli) might be the most relevant tool. The good news is that both platform support kubectl'' to operate your cluster. The not so good news is that both have their own flavor of cli (ICP has the cloudctl while OpenShift has oc). Most of the standard kubernetes tasks can be carried out by sticking to kubectl''. That puts migration as small effort to migrate any cloudctl'' command to either kubectl'' or ``oc'' or sunset them.

One area you need to pay attention is that OpenShift runs only on RHEL or RHCOS operating system. That may introduce some migration work when your ICP is running on non-RedHat OS. For example, if you have operation scripts handles the patches update on OS, service restart etc.

Both platforms are adopting the CNCF projects as de-facto standard when comes to monitoring. They are Grafana and Prometheus. ICP has fairly decent integration with both technologies and OpenShift 3.11 installs them by default. But this doesn’t mean the migration is that straightforward.

First, Prometheus may collect different set of metrics. It will be at least a medium level of effort to adjust the Prometheus Query Language and tested in new OpenShift platform.

Then, you might need to migrate the Grafana dashboards that purposely built for ICP. OpenShift comes with some sample dashboard like Docker or Kubernetes monitoring via Prometheus.

Alerting is another area you need to consider. In theory, OpenShift Prometheus supports AlertManager (can be installed as optional component). But ensuring the existing ICP alerts fully function in OpenShift including Notification by email, webhooks, Slack, PagerDuty and alert Silencing, aggregation, inhibiting can take quite bit of effort.

ICP deploys an ELK (ElasticSearch, Logstash, Kibana) stack, referred to as the management logging service, to collect and store all Docker-captured logs.

OpenShift uses the EFK (ElasticSearch, fluentd, Kibana) stack as a logging solution. The main difference comparing to ICP is how the logs are shipped out of the cluster with Fluentd. But most of that is implementation detail and relatively transparent to the application and end user.

To migrate from ICP to Openshift, the recommended approach is a blue-green deployment where an Openshift cluster is created alongside an existing ICP cluster, the workload is migrated from ICP to Openshift, load balancers or DNS entries are updated to point clients at the new cluster, and the ICP cluster is retired.

We highly recommend infrastructure automation to create new clusters for Openshift. This provides the quickest path for new cluster creation. We have published infrastructure automation templates for ICP on various cloud platforms, for example:

https://github.com/ibm-cloud-architecture/terraform-icp-vmware

Terraform Examples for OpenShift 4.x: https://github.com/ibm-cloud-architecture/terraform-openshift4-aws https://github.com/ibm-cloud-architecture/terraform-openshift4-azure https://github.com/ibm-cloud-architecture/terraform-openshift4-gcp

In scenarios where resources are limited, the approach involves draining and removing under-utilized ICP worker nodes and using the capacity to build a small Openshift control plane with a few workers. Depending on available capacity, an Openshift cluster can start as a single master and scale up to three masters as capacity is made available.

In this section we used a case study of migrating our cloud native reference application BlueCompute that was running on ICP to Openshift.

https://github.com/ibm-cloud-architecture/refarch-cloudnative-kubernetes

For detailed steps on how to migrate Bluecompute from ICP to Openshift, refer this doc - BlueCompute on OpenShift.

Generally, ICP was not opinionated on DevOps, and if the toolchain is outside of the platform there is no reason for this to change when migrating to Openshift.

Openshift’s value proposition does include some developer productivity tools such as Source-to-Image (S2I) and containerized Jenkins, as well as tech preview and support for Openshift Pipelines and CodeReady Workspaces, but in a migration scenario where we are migrating existing workload from ICP these are not required to change. We can simply treat Openshift as another Kubernetes platform we are deploying to.