4.1 Troubleshooting basics
In chapter 3 you got to know the monitoring components, its rules and more. Now, imagine you get an alert. “Where to begin?”, you ask. How do you find out what’s amiss and led to the alert?
That’s what we are going to look at in this first lab.
Task 4.1.1: Cluster operators
As you already know, OpenShift 4 usually consists of about 30 different operators. Each operator is responsible that the actual state of the cluster matches the desired, configured state. If this is not the case, the operator tells us.
Check the cluster operators’ state.
Hints
oc get clusteroperators
Note
For most of the Kubernetes resource types, there’s a short version.
In the case of the clusteroperators resources, it’s co, so you could also check the state with oc get co.
This gives you something along the lines of:
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.7.6 True False False 14h
baremetal 4.7.6 True False False 10d
cloud-credential 4.7.6 True False False 10d
cluster-autoscaler 4.7.6 True False False 10d
config-operator 4.7.6 True False False 10d
console 4.7.6 True False False 6d23h
csi-snapshot-controller 4.7.6 True False False 6d23h
dns 4.7.6 True False False 6d23h
etcd 4.7.6 True False False 10d
image-registry 4.7.6 True False False 6d23h
ingress 4.7.6 True False False 10d
insights 4.7.6 True False False 10d
kube-apiserver 4.7.6 True False False 10d
kube-controller-manager 4.7.6 True False False 10d
kube-scheduler 4.7.6 True False False 10d
kube-storage-version-migrator 4.7.6 True False False 6d19h
machine-api 4.7.6 True False False 10d
machine-approver 4.7.6 True False False 10d
machine-config 4.7.6 True False False 6d23h
marketplace 4.7.6 True False False 6d23h
monitoring 4.7.6 True False False 6d20h
network 4.7.6 True False False 10d
node-tuning 4.7.6 True False False 7d
openshift-apiserver 4.7.6 True False False 6d20h
openshift-controller-manager 4.7.6 True False False 10d
openshift-samples 4.7.6 True False False 7d
operator-lifecycle-manager 4.7.6 True False False 10d
operator-lifecycle-manager-catalog 4.7.6 True False False 10d
operator-lifecycle-manager-packageserver 4.7.6 True False False 6d20h
service-ca 4.7.6 True False False 10d
storage 4.7.6 True False False 6d23h
The important part to see here is that all of the operators are available, that they are not progressing and that they are not degraded.
If you make a change to one of the operator’s configuration, the operator usually changes its progressing state to true as long as the configuration change has not finished.
Degraded means that the operator cannot function properly because there is, e.g., a configuration error.
Task 4.1.2: Configuration error
In order to test this behaviour, we are going to introduce a configuration change. Specifically, we want to add an authentication provider to our cluster:
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
name: cluster
spec:
identityProviders:
- name: github
mappingMethod: claim
type: GitHub
github:
clientID: ef60e25a4f8e2816a8f9
clientSecret:
name: github-secret
organizations:
- myorganization
Apply the authentication provider configuration to the cluster.
Hints
oc apply -f https://raw.githubusercontent.com/acend/openshift-operations-training/main/content/en/docs/04/resources/oauth_cluster.yaml
It might take a while, but after a moment, the authentication operator will change its “degraded” state to true. Obviously something seems to be wrong with our introduced configuration; it’d be quite the coincidence something else happened at the same time. But we need to know what exactly, so let’s try to find out.
To do that, let’s have a closer look at the cluster operator responsible for the oauth configuration, the authentication operator. Execute a describe on it and look for a hint on what could be wrong.
Hints
oc describe clusteroperator authentication
The output will be rather long.
Look for for the Status part to see the relevant message:
Status:
Conditions:
Last Transition Time: 2021-04-23T09:39:40Z
Message: OAuthServerConfigObservationDegraded: error validating secret openshift-config/github-secret: secret "github-secret" not found
Reason: OAuthServerConfigObservation_Error
Status: True
Type: Degraded
Last Transition Time: 2021-04-22T18:54:38Z
Message: All is well
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2021-04-22T18:54:50Z
Message: OAuthServerDeploymentAvailable: availableReplicas==2
Reason: AsExpected
Status: True
Type: Available
You can see that the operator’s status first changed from “Available” (Type field) to “Progressing” and then to “Degraded”.
This means that it tried to apply our configuration change but then failed to do so.
Looking at the topmost “Message” reveals the problem:
The configuration defines a secret in clientSecret which contains the GitHub client secret.
This secret doesn’t exist (on purpose).
Now that you’ve found out what the problem is and how you can get the relevant troubleshooting information, revert the change so the authentication operator goes back to a functioning state.
Hints
oc patch oauth cluster --type merge -p '{"spec": {"identityProviders": []}}'
Task 4.1.3: Node debugging
OpenShift 4 introduces a new capability to debug cluster nodes.
Using oc debug node/<node name>, a debug pod is started on the specified node and you get a terminal session in that debug pod.
Let’s try this out. Pick a node of your choosing.
Hints
oc get nodes
Then, using the node’s name, start the debug session.
Hints
oc debug node/<node name>
Warning
Note the / between the resource type (node) and the name.
The command will error out if you don’t write it.
After a brief moment you are presented with a root shell on the desired node. However, you’re not yet really on the node, you’re limited to what the container sees of it. You need to chroot onto the node before you’ll be able to use host binaries. The message that appeared when you opened the debug shell conveniently provides the chroot command for us.
Hints
chroot /host
If you’re not sure whether you’re on the node, there are different possibilities to find out, one of which is looking at the release file.
Hints
cat /etc/redhat-release
If the output says Red Hat Enterprise Linux release [...], you’re still contained in the container.
As we know, OpenShift nodes use CoreOS as operating system, so the output should read Red Hat Enterprise Linux CoreOS release [...].
We can now do all the things as if we had connected via ssh. Which is still possible but might be less convenient.
One of these things you might want to do is have a look at the running containers on that node.
We do this using crictl.
Hints
crictl ps
As you probably know from the good old days when you still used Docker on your machines, ps lists all running containers.
crictl is also aware of pods, so as a next task, list all pods.
Hints
crictl pods
To exit the node debug pod, either simply type exit or press ctrl+d until you’re back on your own shell.
For more information on crictl, check out this debugging how-to
or its documentation
.
Task 4.1.4: Node logs
You might be tempted now to use the oc debug node feature to use it for all kinds of debugging tasks.
However, there is one particularly useful oc adm subcommand that comes in especially handy when you want to look at a node’s logs.
oc adm node-logs allows you to retrieve node logs.
By default, the command queries the systemd journal.
Using --path, you can override this behaviour and use it to show logs within the node’s /var/logs directory.
Let’s use this to query all masters’ (--role master) kubelet (--unit) logs.
Hints
oc adm node-logs --role master --unit kubelet --tail 10
Note
As with the journalctl command you can use –tail, –since and –until to limit the amount of logs or to focus on a certain period of time.
Of course above command can be used for all the other systemd units that are running on a node.
As already mentioned, the --path parameter can be used to show logs in the /var/logs directory.
The following command lists all available log directories:
oc adm node-logs --role master --path /
This way, we can find a specific log file and finally show its content, e.g. the audit log’s:
oc adm node-logs --role master --path /audit/audit.log
Note
You might have to interrupt thenode-logs command using --path with ctrl+c depending on the number of logs.
The --tail, --since and --until parameters cannot be used to with --path.Task 4.1.5: Node resource usage
Belonging to the same category of particularly useful commands is oc adm top.
It can be used to display usage statistics of images, imagestreams, nodes and pods.
Want to know which image uses the most space on your nodes?
oc adm top images is your friend.
It works very similarly for ImageStreams objects.
A command providing you with a nice overview of resource usage on your nodes (without using Prometheus) is oc adm top node, which also works with the --selector parameter to list all nodes that match the selected label:
oc adm top node --selector node-role.kubernetes.io/worker
Finally, let’s check the uptime app’s resource consumption:
oc adm top pods --namespace uptime-app-prod
Task 4.1.6: SSH
You might be wondering why all these new subcommands were introduced with OpenShift 4. Why not simply ssh into a node you want to analyze?
The reason is the introduction of CoreOS as the default underlying operating system of OpenShift 4. CoreOS changes the way we interact with an operating system because it follows the core principles of containers:
- immutability
- statelessness
This means that it is discouraged to ssh into a node because this might introduce manual, undocumented changes to the operating system that could lead to unexpected behaviour.
Always apply configuration changes via MachineConfig objects instead.
However, there are cases where ssh represents the only viable means of analyzing a malfunctioning node.
Imagine the OpenShift API is down or the node’s kubelet is not responding.
oc will not work in these cases.
So it still is a good idea to configure ssh keys in those installation configuration files.
In order to connect to any node, first find out the node’s hostname. Fortunately, and this should always be the case, AWS uses the fully-qualified hostnames as node names. Get a hostname:
oc get nodes
Note
Note that the ssh command will not work! This is due to this training’s specific network segmentation. We added the command for reference nonetheless.The only remaining piece left to know about is that you always have to use username core when connecting to a CoreOS OpenShift node:
ssh core@<node name>
Troubleshooting references
The OpenShift documentation has multiple troubleshooting documentation pages such as this one which are worth checking out.