Kubernetes error “context deadline exceeded”

The error “context deadline exceeded” implies that we ran into a situation where a given action was not finished in an anticipated timeframe.

Generally, the issue occurs when Pods are stuck in Init status.

The normal error looks as shown below:

Warning FailedCreatePodSandBox 93s (x8 over 29m) kubelet, 97011e0a-f47c-4673-ace7-d6f74cde9934 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded Normal SandboxChanged 92s (x8 over 29m) kubelet, 97011e0a-f47c-4673-ace7-d6f74cde9934 Pod sandbox changed, it will be killed and re-created.

You can also check the status of the node-agent-hyperbus by running the following nsxcli command from the node (as root user):

sudo -i
/var/vcap/jobs/nsx-node-agent/bin/nsxcli
“at the nsx-cli prompt, enter”: get node-agent-hyperbus status

Anticipated output:

HyperBus status: Healthy

In this situation you would see the following error instead:

% An internal error occurred

This causes a loop of DEL (delete) requests to the nsx-node-agent process

Restarting the nsx-node-agent process will resolve this error:

Use the “bosh ssh” command to access the worker node

Then, run the following commands.

sudo -i
monit restart nsx-node-agent

Sit tight and wait for nsx-node-agent to restart: watch monit summary

This should resolve the issue.