2.6 Control plane nodes
Note
While many occurrences of “master” have been replaced throughout OpenShift and Kubernetes, this is unfortunately not yet the case forMachine resources. This is why you are going to encounter a mixture of “control plane” and “master” terms in this lab.During this lab, we will learn how to replace a failed control plane node. As long as we do not lose the majority of our control plane nodes, we can simply replace those that failed. If we lose the majority (e.g., 2 out of 3), we need to restore the state of etcd from a previously created snapshot, which will not be covered in this lab.
Since the control plane nodes are hosting etcd, the key-value store that holds all the cluster’s information and state, it is not as straightforward as replacing a worker node backed by a machine set.
Note
Above statement is not completely correct anymore. OpenShift 4.12 introduced the control plane machine sets for specific infrastructure providers . Because the control plane machine set on AWS is supported and active by default, we are going to delete it in the first step of this lab.This enables you to do this lab and experience the replacement of a control plane node anyway, in case you need to do it on infrastructure that is not supported by or has no active control plane machine set.
Task 2.6.1: Replacing the machine
As mentioned in above note, the first task is to delete the control plane machine set. The control plane machine set will immediately be recreated, however, it will not be active anymore.
oc -n openshift-machine-api delete controlplanemachinesets.machine.openshift.io cluster
Now, save the resource definition of the master machine you want to replace.
Hints
oc -n openshift-machine-api get machine <failed-master-machine> -o yaml > machine_<failed-master-machine>.yaml
Edit the file to reflect the following changes:
- Remove:
metadata.annotationsmetadata.creationTimestampmetadata.generationsmetadata.uidmetadata.resourceVersionspec.providerID- all of
status
Hints
The machine resource should look similar to this:
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
finalizers:
- machine.machine.openshift.io
labels:
machine.openshift.io/cluster-api-cluster: user01-ops-training-75gjz
machine.openshift.io/cluster-api-machine-role: master
machine.openshift.io/cluster-api-machine-type: master
machine.openshift.io/instance-type: m5.xlarge
machine.openshift.io/region: eu-north-1
machine.openshift.io/zone: eu-north-1c
name: user01-ops-training-75gjz-master-2
namespace: openshift-machine-api
spec:
metadata: {}
providerSpec:
value:
ami:
id: ami-0080eb90a48d9655e
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 0
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp2
credentialsSecret:
name: aws-cloud-credentials
deviceIndex: 0
iamInstanceProfile:
id: user01-ops-training-75gjz-master-profile
instanceType: m5.xlarge
kind: AWSMachineProviderConfig
loadBalancers:
- name: user01-ops-training-75gjz-int
type: network
- name: user01-ops-training-75gjz-ext
type: network
metadata:
creationTimestamp: null
placement:
availabilityZone: eu-north-1c
region: eu-north-1
securityGroups:
- filters:
- name: tag:Name
values:
- user01-ops-training-75gjz-master-sg
subnet:
filters:
- name: tag:Name
values:
- user01-ops-training-75gjz-private-eu-north-1c
tags:
- name: kubernetes.io/cluster/user01-ops-training-75gjz
value: owned
- name: customer
value: acend
- name: username
value: user01
- name: acend-training
value: ocp4-ops
userDataSecret:
name: master-user-data
Delete the master machine you want to replace.
Warning
Make sure you delete the master machine you just saved the resource definition from in the previous step.Hints
oc -n openshift-machine-api delete machine <failed-master-machine>
This will queue the machine for deletion. You can verify this by checking the machines’ state.
Hints
oc -n openshift-machine-api get machines -o wide
Example output:
NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE
user01-ops-training-75gjz-infra-eu-north-1a-9mzd9 Running m5.2xlarge eu-north-1 eu-north-1a 46h ip-10-0-130-69.eu-north-1.compute.internal aws:///eu-north-1a/i-0a7b134c9b0d07d4d running
user01-ops-training-75gjz-infra-eu-north-1a-h88ln Running m5.2xlarge eu-north-1 eu-north-1a 46h ip-10-0-154-3.eu-north-1.compute.internal aws:///eu-north-1a/i-0b98e80b4cd46d367 running
user01-ops-training-75gjz-infra-eu-north-1a-smgzs Running m5.2xlarge eu-north-1 eu-north-1a 46h ip-10-0-154-185.eu-north-1.compute.internal aws:///eu-north-1a/i-0750413e1557c439f running
user01-ops-training-75gjz-master-0 Running m5.xlarge eu-north-1 eu-north-1a 3d3h ip-10-0-155-28.eu-north-1.compute.internal aws:///eu-north-1a/i-0736635506a9eb2cc running
user01-ops-training-75gjz-master-1 Running m5.xlarge eu-north-1 eu-north-1b 3d3h ip-10-0-184-171.eu-north-1.compute.internal aws:///eu-north-1b/i-067f7bac5768ab117 running
user01-ops-training-75gjz-master-2 Deleting m5.xlarge eu-north-1 eu-north-1c 3d3h ip-10-0-212-27.eu-north-1.compute.internal aws:///eu-north-1c/i-02210ddbae879539e running
user01-ops-training-75gjz-worker-eu-north-1a-vv59v Running m5.2xlarge eu-north-1 eu-north-1a 3d3h ip-10-0-139-81.eu-north-1.compute.internal aws:///eu-north-1a/i-0d243001e9ba4cb9b running
user01-ops-training-75gjz-worker-eu-north-1b-zddgj Running m5.2xlarge eu-north-1 eu-north-1b 3d3h ip-10-0-173-170.eu-north-1.compute.internal aws:///eu-north-1b/i-028cec3fbf26c0f13 running
user01-ops-training-75gjz-worker-eu-north-1c-6mqls Running m5.2xlarge eu-north-1 eu-north-1c 3d3h ip-10-0-195-93.eu-north-1.compute.internal aws:///eu-north-1c/i-0d7f8ed2c8b9c0932 running
This machine will not be deleted right away. You can see in the resource definition that its lifecycle contains a pre-drain-hook, executed before draining the machine. This step must run before the draining of the machine, before the deletion itself:
spec:
lifecycleHooks:
preDrain:
- name: EtcdQuorumOperator
owner: clusteroperator/etcd
The hook ensures that the collection of etcd nodes (the “quorum”) remains in a healthy state. This is no longer granted if we remove one node out of the three expected ones. If we manually decrease the number of required nodes in the healthy state, then the third node can be deleted.
Task 2.6.2: Deleting the etcd member
For the master machine to be fully removed, we need to also remove the etcd member, since it still has the configuration of the old control plane node.
Connect to an etcd pod that is not running on the control plane node you just removed:
Hints
oc -n openshift-etcd rsh <etcd-pod>
View the member list and take note of the ID and name of the etcd member you want to remove.
Note
To identify the member you need to remove, compare the list with the existing pod names. In the example, the list shows a member with the nameip-10-0-212-27.eu-north-1.compute.internal which does not correspond with any of the control plane node names.All relevant information can be found in OpenShift’s documentation .
Hints
etcdctl member list -w table
Example output:
+------------------+---------+---------------------------------------------+---------------------------+---------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+---------------------------------------------+---------------------------+---------------------------+------------+
| ce90578971689b5 | started | ip-10-0-155-28.eu-north-1.compute.internal | https://10.0.155.28:2380 | https://10.0.155.28:2379 | false |
| 2c53f4526118a2eb | started | ip-10-0-184-171.eu-north-1.compute.internal | https://10.0.184.171:2380 | https://10.0.184.171:2379 | false |
| 43997d8a75197361 | started | ip-10-0-212-27.eu-north-1.compute.internal | https://10.0.212.27:2380 | https://10.0.212.27:2379 | false |
+------------------+---------+---------------------------------------------+---------------------------+---------------------------+------------+
Remove the old peer entry.
Hints
etcdctl member remove <id>
Verify the member was removed.
Hints
etcdctl member list -w table
This is it! You can now disconnect from the etcd pod.
The machine we marked for deletion can now finally proceed. We can check the machines’ state again.
Note
The machine deletion might take several minutes to complete.Hints
oc -n openshift-machine-api get machines -o wide
Task 2.6.3: Recreate the control plane node
The cluster is again in a properly running state, of course however with only two instead of three control plane nodes and etcd members.
Recreate the master machine using the file definition you created before.
Hints
oc -n openshift-machine-api apply -f machine_<failed-master-machine>.yaml
Verify the machine was created successfully:
Hints
oc -n openshift-machine-api get machines -o wide
The etcd operator should automatically scale up a new member and add it to the cluster.
Verify that three etcd pods are up and running.
Hints
oc -n openshift-etcd get pods -l etcd
Note
If only two pods are running, you can force a redeployment:
oc patch etcd cluster \
-p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' \
--type=merge
All that is left to do is clean up the old secrets that are not used anymore.
List the secrets of the old peer that can be safely deleted:
oc -n openshift-etcd get secrets | grep <removed-peer-name>
Example output:
etcd-peer-ip-10-0-212-27.eu-north-1.compute.internal kubernetes.io/tls 2 3d4h
etcd-serving-ip-10-0-212-27.eu-north-1.compute.internal kubernetes.io/tls 2 3d4h
etcd-serving-metrics-ip-10-0-212-27.eu-north-1.compute.internal kubernetes.io/tls 2 3d4h
Delete those secrets.
Hints
oc -n openshift-etcd delete secrets \
etcd-peer-<removed-peer-name> \
etcd-serving-<removed-peer-name> \
etcd-serving-metrics-<removed-peer-name>