Zombie Processes Back in K8S

After my previous article on zombie processes I was curious to see if and how they can affect containers running in a Kubernetes (K8S) cluster.

First try

A simple descriptor to deploy the container.

# zombie-init.yml
apiVersion: v1
kind: Pod
metadata:
  name: zombie-init
spec:
  containers:
  - name: zombie-init
    image: zombie-init:latest
    imagePullPolicy: Never

$ k create -f zombie-init.yml

# pod/zombie-init created

$ k logs zombie-init         

# The zombie pid will be: 7
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# root         1  0.0  0.5  13164 10676 ?        Ss   20:09   0:00 python3 /root/test.py
# root         7  0.0  0.0      0     0 ?        Z    20:09   0:00 [python3] <defunct>
# root         8  0.0  0.1   9392  2952 ?        R    20:09   0:00 ps xawuf

We can observe the same behavior a already observed in the previous article. A zombie process (defunct) has not been reaped since the PID 1 process (it’s parent) has not played this role.

Reaping zombies

To benefit from a proper init process that will take care of reaping zombies, we have to tell Kubernetes to enable PID sharing between the containers that are running in the POD. This is done by specifying shareProcessNamespace: true. So the deployment looks now like this.

apiVersion: v1
kind: Pod
metadata:
  name: zombie-init
spec:
  shareProcessNamespace: true
  containers:
  - name: zombie-init
    image: zombie-init:latest
    imagePullPolicy: Never

Here is the result obtained with this new configuration.

$ k logs zombie-init

# The zombie pid will be: 21
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# root        15  0.0  0.5  13156 10596 ?        Ss   20:11   0:00 python3 /root/test.py
# root        22  0.0  0.1   9392  3080 ?        R    20:11   0:00  \_ ps xawuf
# root         1  1.0  0.0   1024     4 ?        Ss   20:11   0:00 /pause

In the log, we can see that the zombie process has been reaped without specifying an init process or adding one in the container. In this case the init process, which plays the parent role, is the pause process (in fact it’s a container). We can observe that it holds the PID 1.

In Kubernetes, the pause container serves as the “parent container” for all of the containers in your pod. The pause container has two core responsibilities. First, it serves as the basis of Linux namespace sharing in the pod. And second, with PID (process ID) namespace sharing enabled, it serves as PID 1 for each pod and reaps zombie processes.

—The Almighty Pause Container

If namespace sharing is not enabled each container has to do it’s own housekeeping—it could not be a problem at all if the process running in the container does not spawn other processes. The problem can also be solved by using an init process inside each container like it has been demonstrated in the previous article.

References / Further reading

• The Almighty Pause Container
• POD-PID-NAMESPACE / SHARED PID NAMESPACE