Kubernetes – Fix Pod status stuck at Terminating or ContainerCreating

A Pod stuck at Terminating or ContainerCreating can happen for lots of reasons. One of the most common is ETCD holding a stale list of leases. When that happens, your pods randomly get “connection refused” while trying to reach the Kubernetes API server.

To fix it, you’ll need kubectl and permission to exec into the ETCD Manager pod. Let’s clean up those leases.

Find your cluster’s ETCD version

kops get cluster --full -o yaml

Look for the etcdCluster section — that’s where the ETCD version lives. Write it down for the next steps. Mine was 3.5.9.

Find the ETCD Manager Main pod

kubectl -n kube-system get pod | grep etcd-manager-main

Note the pod name, for example etcd-manager-main-i-0054d377e2464e22f.

Exec into that pod

kubectl exec -it -n kube-system etcd-manager-main-i-0054d377e2464e22f -- sh

Set up etcdctl

Once you’re inside the pod, run these to get etcdctl ready:

ETCD_VERSION=3.5.9
ETCDDIR=/opt/etcd-v$ETCD_VERSION
CERTDIR=/rootfs/srv/kubernetes/kube-apiserver/
alias etcdctl="ETCDCTL_API=3 $ETCDDIR/etcdctl --cacert=$CERTDIR/etcd-ca.crt --cert=$CERTDIR/etcd-client.crt --key=$CERTDIR/etcd-client.key --endpoints=https://127.0.0.1:4001"

Check the members

etcdctl member list

Look at the current master leases

etcdctl get --prefix --keys-only /registry/masterleases

Delete all the master leases

etcdctl del --prefix /registry/masterleases/

This is the fastest way to clear out the stale leases.

Don’t worry — every active control-plane (master) node will reconnect and register its lease again on its own.

Check the leases one more time

etcdctl get --prefix --keys-only /registry/masterleases

You should now see the active control-plane nodes back in the list.