Kubernetes
initcontainer check a service for alive before continuing
initContainers:
- name: check-mongo-status
image: busybox:1.28
command:
- "/bin/sh"
- "-c"
- |
i=0
while true; do
echo -n "Attempt $((++i))... "
echo 'quit' | nc -w 3 mongo-service 27017 && echo "OK" && exit 0
sleep 10
done
Delete all calico networkpolicy in a namespace
NAMESPACE=resource-sandbox
calicoctl get networkpolicies -n ${NAMESPACE} -o json | jq -r '.items[].metadata.name' | xargs -I {} calicoctl delete networkpolicy {} -n ${NAMESPACE}
In case of diskpressure
Most like docker overlay is using too many space. Prune it.
sudo docker system prune -a -f
Docker too many pull requests
@ref: https://devops.stackexchange.com/questions/16140/how-do-i-get-k3s-to-authenticate-with-docker-hub
Update your /etc/rancher/k3s/registries.yaml
to
configs:
registry-1.docker.io:
auth:
username: <YOUR_DOCKERHUB_USERNAME>
password: <YOUR_DOCKERHUB_PASSWORD>
Restart k3s
sudo systemctl force-reload k3s
You can confirm the changes were accepted by checking that your key exists in /var/lib/rancher/k3s/agent/etc/containerd/config.toml
.
Content of config.toml
...
[plugins.cri.registry.configs."registry-1.docker.io".auth]
username = <YOUR_DOCKERHUB_USERNAME>
password = <YOUR_DOCKERHUB_PASSWORD>
Good container image to try
netshoot - debug k8s networking
Force delete namespace / stuck
Run this if namespace stuck in terminating.
python3 -c "namespace='postgres-ha';import atexit,subprocess,json,requests,sys;proxy_process = subprocess.Popen(['kubectl', 'proxy']);atexit.register(proxy_process.kill);p = subprocess.Popen(['kubectl', 'get', 'namespace', namespace, '-o', 'json'], stdout=subprocess.PIPE);p.wait();data = json.load(p.stdout);data['spec']['finalizers'] = [];requests.put('http://127.0.0.1:8001/api/v1/namespaces/{}/finalize'.format(namespace), json=data).raise_for_status()"
TIP
If the namespace has been marked for deletion without deleting the cert-manager installation first, the namespace may become stuck in a terminating state. This is typically due to the fact that the APIService resource still exists however the webhook is no longer running so is no longer reachable.
Label node
@ref: https://medium.com/kubernetes-tutorials/learn-how-to-assign-pods-to-nodes-in-kubernetes-using-nodeselector-and-affinity-features-e62c437f3cf8 @ref: https://tachingchen.com/blog/kubernetes-assigning-pod-to-nodes/
kubectl label nodes host02 disktype=ssd
Tolerate
add to deployment
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 10
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 10
One Pod per Node
# made sure one replica is run in each node
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- traefik
topologyKey: kubernetes.io/hostname
Resource Limit
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Wait for Deployment/Daemonset
@ref: https://www.starkandwayne.com/blog/silly-kubectl-trick-5-waiting-for-things-to-finish-up-2/https://linuxhint.com/wait-for-condition-kubectl/
kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller -w --timeout 60s
or use -n
to not specify timeout and wait till success
kubectl rollout status daemonset -n kube-system rke2-ingress-nginx-controller -n -w
You can also use kubectl wait
for deployments:
$ kubectl wait deploy/slow --for condition=available
deployment.apps/slow condition met
Note that the condition for a deployment is available, not ready.
Delete Evicted Pods from One Namespace
NAMESPACE=YOUR_NAMESPACE
kubectl get pod -n ${NAMESPACE} | grep Evicted | awk '{print $1}' | xargs kubectl delete pod -n ${NAMESPACE}
Delete Evicted Pods from All Namespaces
Method 1
#!/bin/bash
# -t remove trailing new line
readarray -t ns <<< $(kubectl get ns | awk '{print $1}' | tail -n +2)
for i in "${ns[@]}"
do
kubectl get pod -n $i | grep Evicted | awk '{print $1}' | xargs kubectl delete pod -n $i
done
Method 2
#!/bin/bash
# Get a list of all evicted pods
evicted_pods=$(kubectl get pods --all-namespaces | grep Evicted | awk '{print $2}' | awk -F/ '{print $1}' | uniq)
# Loop through the evicted pods and delete them
for pod in $evicted_pods; do
kubectl delete pods --all-namespaces --field-selector=status.phase=Failed
done
Configure Node Nviction Hard Pressure
@ref: https://github.com/kubernetes/kubeadm/issues/1464
For k8s bootstrap using kubeadm
:
Edit kubelet config map
Add
evictionHard
map. See https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ for all fields supported.
...
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
evictionHard:
nodefs.available: "5%"
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
...
Get Pod by Label
LABEL_KEY_VALUE="app.kubernetes.io/name=api-gateway" \
NAMESPACE="ota" \
kubectl exec -it $(kubectl get pods --selector=$LABEL_KEY_VALUE -n $NAMESPACE --no-headers=true | awk '{print $1}') -n $NAMESPACE -- cat /var/log/traefik.log
LABEL_KEY_VALUE="app.kubernetes.io/name=deployments" \
NAMESPACE="ota" \
kubectl logs -f $(kubectl get pods --selector=$LABEL_KEY_VALUE -n $NAMESPACE --no-headers=true | awk '{print $1}') -n $NAMESPACE
Get pods from a specific node
NAMESPACE=namespace
HOSTNAME=hostname
kubectl -n ${NAMESPACE} get pod --field-selector spec.nodeName=${HOSTNAME}
Get the logs for a pod running on a specific node
NAMESPACE=namespace
POD_ID=$(kubectl -n ${NAMESPACE} get pod --field-selector spec.nodeName=$HOSTNAME | tail -n 1 | awk '{print $1}')
kubectl -n ${NAMESPACE} logs -f ${POD_ID}
Modify disk pressure
TIP
The default disk pressure in k3s is 85%. Once the disk quota use reaches this limit, no more pods can be scheduled to this node.
Edit kubelet arguments in /etc/systemd/system/k3s-agent.service
ExecStart=/usr/local/bin/k3s \
agent \
'--data-dir=/opt/rancher/k3s' \
'--server=https://192.168.10.236:6443' \
'--token=K109bdf5e62167d4b4c51ab9b2163d87d7244c81bbabe016498444fe6dca338794a::server:21bb7a3fd2a408f647ec59e13fba09b4' \
'--node-ip=192.168.10.237' \
'--kubelet-arg=eviction-hard=imagefs.available<5%,memory.available<100Mi,nodefs.available<2%' \
Route nodeport port to only specific node
How to preserve the source IP in Kubernetes: https://blog.cptsai.com/2020/11/15/k8s-external-traffic-policy/https://blog.getambassador.io/externaltrafficpolicy-local-on-kubernetes-e66e498212f9
Use externalTrafficPolicy: Local
when defining spec
even if nodeport
e.g.
apiVersion: v1
kind: Service
metadata:
name: fecb-device-api
namespace: fecb-system
spec:
externalTrafficPolicy: Local
type: NodePort
selector:
app: fecb-device-api
ports:
- name: http
protocol: TCP
port: 9017
targetPort: 46490
nodePort: 30894
Get helm
manifest for release
helm get manifest <release-name> -n <namespace>