First real blog post, and what better way to start it with some kube-prometheus-stacking.
Version 39.4.1 added some new rules and after upgrading I immediately started getting alerts for etcdDatabaseHighFragmentationRatio, oh no! What shall one do? Off to Google to find an answer!
I came across this gist which provided some insights into what steps to take.
On one of your master nodes create etcdctl-install.sh and chmod +x it.
#!/usr/bin/env bash
set -eux
etcd_version=v3.5.3
case "$(uname -m)" in
aarch64) arch="arm64" ;;
x86_64) arch="amd64" ;;
esac;
etcd_name="etcd-${etcd_version}-linux-${arch}"
curl -sSfL "https://github.com/etcd-io/etcd/releases/download/${etcd_version}/${etcd_name}.tar.gz"
| tar xzvf - -C /usr/local/bin --strip-components=1 "${etcd_name}/etcdctl"
Run the script
./etcdctl-install.sh
Export etcd variables that etcdctl needs to talk to the cluster
export ETCDCTL_ENDPOINTS="https://127.0.0.1:2379"
export ETCDCTL_CACERT="/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt"
export ETCDCTL_CERT="/var/lib/rancher/k3s/server/tls/etcd/server-client.crt"
export ETCDCTL_KEY="/var/lib/rancher/k3s/server/tls/etcd/server-client.key"
export ETCDCTL_API=3
defrag
etcdctl defrag --cluster
Confirm output
Finished defragmenting etcd member[https://192.168.42.10:2379]
Finished defragmenting etcd member[https://192.168.42.12:2379]
Finished defragmenting etcd member[https://192.168.42.11:2379]
After that the alerts should clear up!
