97 lines
2.5 KiB
Markdown
97 lines
2.5 KiB
Markdown
|
# Restore etcd
|
||
|
|
|
||
|
|
## Prerequisites
|
||
|
|
- SSH access to `k8s-master` node
|
||
|
|
- etcd snapshot available on NFS at `/mnt/main/etcd-backup/`
|
||
|
|
- etcd PKI certs at `/etc/kubernetes/pki/etcd/` on master node
|
||
|
|
|
||
|
|
## Backup Location
|
||
|
|
- NFS: `/mnt/main/etcd-backup/etcd-snapshot-YYYYMMDD-HHMMSS.db`
|
||
|
|
- Replicated to Synology NAS (192.168.1.13) via TrueNAS ZFS replication
|
||
|
|
- Retention: 30 days
|
||
|
|
- Schedule: Daily at 00:00
|
||
|
|
|
||
|
|
## CRITICAL: etcd is the foundation of the cluster
|
||
|
|
Restoring etcd will reset the entire Kubernetes state to the snapshot time. All objects created after the snapshot will be lost. This is a last-resort operation.
|
||
|
|
|
||
|
|
**Only restore etcd if the control plane is completely broken.**
|
||
|
|
|
||
|
|
## Restore Procedure
|
||
|
|
|
||
|
|
### 1. SSH to the master node
|
||
|
|
```bash
|
||
|
|
ssh k8s-master
|
||
|
|
```
|
||
|
|
|
||
|
|
### 2. Identify the snapshot to restore
|
||
|
|
```bash
|
||
|
|
ls -lt /mnt/main/etcd-backup/etcd-snapshot-*.db | head -10
|
||
|
|
```
|
||
|
|
|
||
|
|
### 3. Stop the API server and etcd
|
||
|
|
```bash
|
||
|
|
# Move static pod manifests to stop them
|
||
|
|
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/
|
||
|
|
sudo mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/
|
||
|
|
|
||
|
|
# Wait for pods to stop
|
||
|
|
sudo crictl ps | grep -E "etcd|apiserver"
|
||
|
|
```
|
||
|
|
|
||
|
|
### 4. Back up current etcd data
|
||
|
|
```bash
|
||
|
|
sudo mv /var/lib/etcd /var/lib/etcd.bak.$(date +%Y%m%d-%H%M%S)
|
||
|
|
```
|
||
|
|
|
||
|
|
### 5. Restore the snapshot
|
||
|
|
```bash
|
||
|
|
sudo ETCDCTL_API=3 etcdctl snapshot restore /mnt/main/etcd-backup/etcd-snapshot-YYYYMMDD-HHMMSS.db \
|
||
|
|
--data-dir=/var/lib/etcd \
|
||
|
|
--name=k8s-master \
|
||
|
|
--initial-cluster=k8s-master=https://127.0.0.1:2380 \
|
||
|
|
--initial-advertise-peer-urls=https://127.0.0.1:2380
|
||
|
|
```
|
||
|
|
|
||
|
|
### 6. Fix permissions
|
||
|
|
```bash
|
||
|
|
sudo chown -R root:root /var/lib/etcd
|
||
|
|
```
|
||
|
|
|
||
|
|
### 7. Restart etcd and API server
|
||
|
|
```bash
|
||
|
|
sudo mv /etc/kubernetes/etcd.yaml /etc/kubernetes/manifests/
|
||
|
|
# Wait for etcd to be ready
|
||
|
|
sleep 30
|
||
|
|
sudo mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/
|
||
|
|
```
|
||
|
|
|
||
|
|
### 8. Verify restoration
|
||
|
|
```bash
|
||
|
|
# Check etcd health
|
||
|
|
sudo ETCDCTL_API=3 etcdctl \
|
||
|
|
--endpoints=https://127.0.0.1:2379 \
|
||
|
|
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
|
||
|
|
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
|
||
|
|
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
|
||
|
|
endpoint health
|
||
|
|
|
||
|
|
# Check cluster status
|
||
|
|
kubectl get nodes
|
||
|
|
kubectl get pods -A | head -20
|
||
|
|
```
|
||
|
|
|
||
|
|
### 9. Reconcile state
|
||
|
|
After etcd restore, some objects may be stale:
|
||
|
|
```bash
|
||
|
|
# Re-apply critical infrastructure
|
||
|
|
cd /path/to/infra
|
||
|
|
scripts/tg apply stacks/platform
|
||
|
|
|
||
|
|
# Check for orphaned resources
|
||
|
|
kubectl get pods -A | grep -E "Terminating|Error|Unknown"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Estimated Time
|
||
|
|
- Snapshot restore: ~10-15 minutes
|
||
|
|
- Full reconciliation: ~30-60 minutes (depends on drift)
|