> For the complete documentation index, see [llms.txt](https://atomoh.gitbook.io/kubernetes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://atomoh.gitbook.io/kubernetes/es/yun-wei-zhi-nan/ops/11-upgrade-operations.md).

# 升级运维

> **支持版本**：EKS 1.28+、Terraform 1.5+、Karpenter 1.0+ **最后更新**：July 10, 2026

< [上一节：资源优化](/kubernetes/es/yun-wei-zhi-nan/ops/10-resource-optimization.md) | [目录](/kubernetes/es/yun-wei-zhi-nan/ops.md) | [下一节：活动容量规划](/kubernetes/es/yun-wei-zhi-nan/ops/12-event-capacity-planning.md) >

***

## 概述

本文档为零停机升级 EKS Auto Mode 集群提供了全面指南。内容涵盖版本规划、升级前验证、升级流程本身以及升级后验证。文中还通过实用代码示例详细说明了原地升级和 Blue/Green 策略。

***

## 1. 升级规划

### Kubernetes 版本支持策略

Kubernetes 遵循 **N-3 支持策略**，这意味着项目会维护最新三个次要版本以及当前版本的发布分支。

| 版本   | 发布日期     | 支持结束     | 状态               |
| ---- | -------- | -------- | ---------------- |
| 1.32 | Dec 2024 | Dec 2025 | 当前               |
| 1.31 | Aug 2024 | Aug 2025 | 受支持              |
| 1.30 | Apr 2024 | Apr 2025 | 受支持              |
| 1.29 | Dec 2023 | Feb 2025 | Extended Support |
| 1.28 | Aug 2023 | Nov 2024 | Extended Support |

### EKS 生命周期：Standard 与 Extended Support

AWS EKS 提供两种支持层级：

**Standard Support（14 个月）**

* 包含在基础 EKS 定价中
* 安全补丁和 bug 修复
* 完整 AWS 支持

**Extended Support（额外 12 个月）**

* 额外费用：每个集群每小时 $0.60
* 仅提供关键安全补丁
* 为升级规划留出更多时间

```hcl
# Terraform: Enable extended support
resource "aws_eks_cluster" "main" {
  name    = "prod-cluster"
  version = "1.29"

  upgrade_policy {
    support_type = "EXTENDED"  # or "STANDARD"
  }
}
```

### 版本兼容性矩阵

升级前，请验证所有组件之间的兼容性：

| 组件                | EKS 1.29 | EKS 1.30 | EKS 1.31 | EKS 1.32 |
| ----------------- | -------- | -------- | -------- | -------- |
| VPC CNI           | 1.15+    | 1.16+    | 1.18+    | 1.19+    |
| CoreDNS           | 1.10.1+  | 1.11.1+  | 1.11.1+  | 1.11.3+  |
| kube-proxy        | 1.29.x   | 1.30.x   | 1.31.x   | 1.32.x   |
| EBS CSI           | 1.25+    | 1.28+    | 1.31+    | 1.33+    |
| Karpenter         | 0.33+    | 0.35+    | 0.37+    | 0.39+    |
| AWS LB Controller | 2.6+     | 2.7+     | 2.8+     | 2.9+     |
| Cert Manager      | 1.13+    | 1.14+    | 1.15+    | 1.16+    |
| ArgoCD            | 2.9+     | 2.10+    | 2.11+    | 2.12+    |

### Add-on 版本要求

根据目标 Kubernetes 版本检查当前 Add-on 版本：

```bash
#!/bin/bash
# check-addon-compatibility.sh

CLUSTER_NAME="prod-cluster"
TARGET_VERSION="1.31"

echo "=== Current Add-on Versions ==="
aws eks list-addons --cluster-name $CLUSTER_NAME --query 'addons[]' --output text | while read addon; do
  version=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.addonVersion' --output text)
  echo "$addon: $version"
done

echo ""
echo "=== Compatible Versions for EKS $TARGET_VERSION ==="
for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  echo "--- $addon ---"
  aws eks describe-addon-versions \
    --addon-name $addon \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[*].addonVersion' \
    --output text | head -5
done
```

### Kubernetes 弃用策略

Kubernetes 提供结构化的弃用时间线：

1. **弃用公告**：API 在发布说明中被标记为 deprecated
2. **警告期**：使用 deprecated API 时，kubectl 会发出警告（至少 2 个版本）
3. **移除**：API 从代码库中移除

**按版本列出的关键弃用项：**

| 版本   | Deprecated/Removed APIs                      |
| ---- | -------------------------------------------- |
| 1.29 | flowcontrol.apiserver.k8s.io/v1beta2 removed |
| 1.30 | CSIStorageCapacity v1beta1 removed           |
| 1.31 | PodSecurityPolicy completely removed         |
| 1.32 | Several beta APIs promoted to stable         |

### 时间线建议

**生产升级时间线：**

| 阶段         | 持续时间 | 活动               |
| ---------- | ---- | ---------------- |
| 规划         | 2 周  | 兼容性审查、弃用审计       |
| Dev/Test   | 2 周  | 升级非生产环境、验证工作负载   |
| Staging    | 1 周  | 完整生产模拟           |
| Production | 1 周  | 带监控的滚动升级         |
| 稳定期        | 2 周  | 监控、记录、更新 runbook |

**推荐升级节奏：**

* 保持在当前稳定版本的 N-1 范围内
* 每 4-6 个月升级一次
* 不要跳过超过一个次要版本

***

## 2. 升级前检查清单

### 使用 Pluto 检测 Deprecated API

[Pluto](https://github.com/FairwindsOps/pluto) 会扫描 deprecated 和 removed API：

```bash
#!/bin/bash
# detect-deprecated-apis.sh

# Install pluto
curl -L -o pluto.tar.gz https://github.com/FairwindsOps/pluto/releases/download/v5.19.0/pluto_5.19.0_linux_amd64.tar.gz
tar -xzf pluto.tar.gz
sudo mv pluto /usr/local/bin/

# Scan live cluster
echo "=== Scanning Live Cluster ==="
pluto detect-all-in-cluster --target-versions k8s=v1.31.0 -o wide

# Scan Helm releases
echo ""
echo "=== Scanning Helm Releases ==="
pluto detect-helm --target-versions k8s=v1.31.0 -o wide

# Scan local manifests
echo ""
echo "=== Scanning Local Manifests ==="
pluto detect-files -d ./k8s-manifests/ --target-versions k8s=v1.31.0

# Generate report
pluto detect-all-in-cluster --target-versions k8s=v1.31.0 -o json > deprecated-apis-report.json
```

**示例输出：**

```
NAME                           KIND                VERSION              REPLACEMENT                    REMOVED   DEPRECATED   REPL AVAIL
my-ingress                     Ingress             extensions/v1beta1   networking.k8s.io/v1           true      true         true
my-pdb                         PodDisruptionBudget policy/v1beta1       policy/v1                      false     true         true
```

### PodDisruptionBudget 审计

PDB 可能会在升级期间阻塞 Node drain：

```bash
#!/bin/bash
# audit-pdbs.sh

echo "=== PodDisruptionBudget Audit ==="

# List all PDBs with their configuration
kubectl get pdb -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'MIN-AVAILABLE:.spec.minAvailable,'\
'MAX-UNAVAILABLE:.spec.maxUnavailable,'\
'CURRENT:.status.currentHealthy,'\
'DESIRED:.status.desiredHealthy,'\
'DISRUPTIONS-ALLOWED:.status.disruptionsAllowed'

echo ""
echo "=== Blocking PDBs (disruptionsAllowed=0) ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.status.disruptionsAllowed == 0) |
  "\(.metadata.namespace)/\(.metadata.name): currentHealthy=\(.status.currentHealthy), desiredHealthy=\(.status.desiredHealthy)"
'

echo ""
echo "=== PDBs with minAvailable=100% (potentially blocking) ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.spec.minAvailable == "100%" or .spec.maxUnavailable == 0 or .spec.maxUnavailable == "0%") |
  "\(.metadata.namespace)/\(.metadata.name)"
'
```

**升级期间 PDB 的最佳实践：**

* 使用 `maxUnavailable: 1`，而不是 `minAvailable: 100%`
* 确保 replicas > PDB 最小值
* 在维护窗口期间临时放宽 PDB

### ETCD 备份策略

EKS 管理 Control Plane，但应用状态备份至关重要：

```yaml
# velero-schedule.yaml
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: pre-upgrade-backup
  namespace: velero
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  template:
    includedNamespaces:
      - "*"
    excludedNamespaces:
      - kube-system
      - velero
    includedResources:
      - "*"
    excludedResources:
      - events
      - events.events.k8s.io
    storageLocation: aws-s3
    volumeSnapshotLocations:
      - aws-ebs
    ttl: 168h  # 7 days retention
    snapshotVolumes: true
    defaultVolumesToFsBackup: false
```

### Velero Snapshot 和 Restore 验证

```bash
#!/bin/bash
# velero-backup-verify.sh

BACKUP_NAME="pre-upgrade-$(date +%Y%m%d-%H%M%S)"

echo "=== Creating Pre-Upgrade Backup ==="
velero backup create $BACKUP_NAME \
  --include-namespaces=app-prod,app-staging \
  --snapshot-volumes \
  --wait

echo ""
echo "=== Verifying Backup ==="
velero backup describe $BACKUP_NAME --details

echo ""
echo "=== Backup Logs ==="
velero backup logs $BACKUP_NAME | tail -50

echo ""
echo "=== Test Restore (dry-run equivalent) ==="
# Create restore to different namespace for verification
velero restore create test-restore-$BACKUP_NAME \
  --from-backup $BACKUP_NAME \
  --namespace-mappings app-prod:restore-test \
  --include-namespaces app-prod \
  --wait

echo ""
echo "=== Verify Restored Resources ==="
kubectl get all -n restore-test

echo ""
echo "=== Cleanup Test Restore ==="
kubectl delete namespace restore-test
velero restore delete test-restore-$BACKUP_NAME --confirm
```

### Add-on 兼容性检查脚本

```bash
#!/bin/bash
# addon-compatibility-check.sh

CLUSTER_NAME="${1:-prod-cluster}"
TARGET_VERSION="${2:-1.31}"

echo "=============================================="
echo "Add-on Compatibility Check"
echo "Cluster: $CLUSTER_NAME"
echo "Target Version: $TARGET_VERSION"
echo "=============================================="

# Get current cluster version
CURRENT_VERSION=$(aws eks describe-cluster --name $CLUSTER_NAME \
  --query 'cluster.version' --output text)
echo "Current Version: $CURRENT_VERSION"
echo ""

# Check each add-on
check_addon() {
  local addon_name=$1

  # Get current version
  current=$(aws eks describe-addon --cluster-name $CLUSTER_NAME \
    --addon-name $addon_name --query 'addon.addonVersion' --output text 2>/dev/null)

  if [ "$current" == "None" ] || [ -z "$current" ]; then
    echo "[$addon_name] Not installed"
    return
  fi

  # Get recommended version for target
  recommended=$(aws eks describe-addon-versions \
    --addon-name $addon_name \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[?compatibilities[0].defaultVersion==`true`].addonVersion' \
    --output text 2>/dev/null)

  # Get all compatible versions
  compatible=$(aws eks describe-addon-versions \
    --addon-name $addon_name \
    --kubernetes-version $TARGET_VERSION \
    --query 'addons[0].addonVersions[*].addonVersion' \
    --output text 2>/dev/null | head -1)

  if echo "$compatible" | grep -q "$current"; then
    echo "[$addon_name] OK - Current: $current, Recommended: $recommended"
  else
    echo "[$addon_name] UPGRADE REQUIRED - Current: $current, Recommended: $recommended"
  fi
}

ADDONS=("vpc-cni" "coredns" "kube-proxy" "aws-ebs-csi-driver" "aws-efs-csi-driver" "snapshot-controller")

for addon in "${ADDONS[@]}"; do
  check_addon "$addon"
done

echo ""
echo "=== Helm Release Versions ==="
helm list -A -o json | jq -r '.[] | "\(.name) (\(.namespace)): \(.chart)"' | sort
```

### Node 和应用健康验证

```bash
#!/bin/bash
# health-verification.sh

echo "=== Node Health Check ==="
kubectl get nodes -o wide
echo ""

# Check for NotReady nodes
NOT_READY=$(kubectl get nodes --no-headers | grep -v " Ready " | wc -l)
if [ "$NOT_READY" -gt 0 ]; then
  echo "WARNING: $NOT_READY nodes are not Ready"
  kubectl get nodes --no-headers | grep -v " Ready "
fi

echo ""
echo "=== Pod Health Check ==="
# Pods not in Running/Completed state
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  --no-headers 2>/dev/null | head -20

echo ""
echo "=== Deployment Health ==="
kubectl get deployments -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'READY:.status.readyReplicas,'\
'DESIRED:.spec.replicas,'\
'AVAILABLE:.status.availableReplicas' | \
awk 'NR==1 || $3!=$4 {print}'

echo ""
echo "=== StatefulSet Health ==="
kubectl get statefulsets -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'READY:.status.readyReplicas,'\
'DESIRED:.spec.replicas' | \
awk 'NR==1 || $3!=$4 {print}'

echo ""
echo "=== Recent Events (Warnings) ==="
kubectl get events -A --field-selector type=Warning \
  --sort-by='.lastTimestamp' | tail -20

echo ""
echo "=== Resource Pressure ==="
kubectl top nodes 2>/dev/null || echo "Metrics server not available"
```

### 综合升级前脚本

```bash
#!/bin/bash
# pre-upgrade-checklist.sh

set -e

CLUSTER_NAME="${1:-prod-cluster}"
TARGET_VERSION="${2:-1.31}"
REPORT_DIR="./upgrade-reports/$(date +%Y%m%d-%H%M%S)"

mkdir -p $REPORT_DIR

echo "=============================================="
echo "EKS Pre-Upgrade Checklist"
echo "Cluster: $CLUSTER_NAME"
echo "Target: $TARGET_VERSION"
echo "Report: $REPORT_DIR"
echo "=============================================="

# 1. Current state snapshot
echo ""
echo "[1/8] Capturing current state..."
kubectl cluster-info > $REPORT_DIR/cluster-info.txt
kubectl get nodes -o wide > $REPORT_DIR/nodes.txt
kubectl get pods -A -o wide > $REPORT_DIR/pods.txt
kubectl get pv,pvc -A > $REPORT_DIR/storage.txt
aws eks describe-cluster --name $CLUSTER_NAME > $REPORT_DIR/eks-cluster.json

# 2. Deprecated API check
echo "[2/8] Checking deprecated APIs..."
pluto detect-all-in-cluster --target-versions k8s=v${TARGET_VERSION}.0 -o json \
  > $REPORT_DIR/deprecated-apis.json 2>/dev/null || echo "Pluto not installed"

DEPRECATED_COUNT=$(cat $REPORT_DIR/deprecated-apis.json | jq 'length' 2>/dev/null || echo "0")
echo "  Found $DEPRECATED_COUNT deprecated API usages"

# 3. PDB audit
echo "[3/8] Auditing PodDisruptionBudgets..."
kubectl get pdb -A -o json > $REPORT_DIR/pdbs.json
BLOCKING_PDBS=$(cat $REPORT_DIR/pdbs.json | jq '[.items[] | select(.status.disruptionsAllowed == 0)] | length')
echo "  Found $BLOCKING_PDBS blocking PDBs"

# 4. Add-on compatibility
echo "[4/8] Checking add-on compatibility..."
aws eks list-addons --cluster-name $CLUSTER_NAME --output json > $REPORT_DIR/addons.json

# 5. Helm releases
echo "[5/8] Documenting Helm releases..."
helm list -A -o json > $REPORT_DIR/helm-releases.json

# 6. Custom resources
echo "[6/8] Documenting custom resources..."
kubectl api-resources --verbs=list -o name | while read resource; do
  count=$(kubectl get $resource -A --no-headers 2>/dev/null | wc -l)
  if [ "$count" -gt 0 ]; then
    echo "$resource: $count"
  fi
done > $REPORT_DIR/resource-counts.txt

# 7. Backup verification
echo "[7/8] Verifying backups..."
if command -v velero &> /dev/null; then
  velero backup get -o json > $REPORT_DIR/velero-backups.json
  RECENT_BACKUP=$(velero backup get --selector='velero.io/schedule-name' -o json | \
    jq -r '.items | sort_by(.status.completionTimestamp) | last | .metadata.name')
  echo "  Most recent backup: $RECENT_BACKUP"
else
  echo "  Velero not installed"
fi

# 8. Health summary
echo "[8/8] Generating health summary..."

cat > $REPORT_DIR/summary.md << EOF
# Pre-Upgrade Summary

**Cluster**: $CLUSTER_NAME
**Current Version**: $(aws eks describe-cluster --name $CLUSTER_NAME --query 'cluster.version' --output text)
**Target Version**: $TARGET_VERSION
**Generated**: $(date -Iseconds)

## Checklist

| Item | Status | Details |
|------|--------|---------|
| Deprecated APIs | $([ "$DEPRECATED_COUNT" == "0" ] && echo "PASS" || echo "REVIEW") | $DEPRECATED_COUNT found |
| Blocking PDBs | $([ "$BLOCKING_PDBS" == "0" ] && echo "PASS" || echo "REVIEW") | $BLOCKING_PDBS found |
| Node Health | $(kubectl get nodes --no-headers | grep -v " Ready " | wc -l | xargs -I{} sh -c '[ {} -eq 0 ] && echo "PASS" || echo "FAIL"') | |
| Backup Status | $([ -n "$RECENT_BACKUP" ] && echo "PASS" || echo "REVIEW") | $RECENT_BACKUP |

## Action Items

$([ "$DEPRECATED_COUNT" != "0" ] && echo "- [ ] Update deprecated APIs (see deprecated-apis.json)")
$([ "$BLOCKING_PDBS" != "0" ] && echo "- [ ] Review blocking PDBs before maintenance window")
- [ ] Notify stakeholders of upgrade window
- [ ] Prepare rollback procedure
EOF

echo ""
echo "=============================================="
echo "Pre-upgrade checklist complete"
echo "Review report at: $REPORT_DIR/summary.md"
echo "=============================================="

cat $REPORT_DIR/summary.md
```

***

## 3. Auto Mode 升级

### Terraform 三层升级顺序

EKS Auto Mode 升级遵循严格顺序以保持稳定性：

1. **Layer 02 (Cluster)**：升级 Control Plane 版本
2. **等待**：允许 Control Plane 升级完成
3. **Layer 03 (Platform)**：将 Add-on 升级到兼容版本
4. **自动**：Node pool 通过 Karpenter drift detection 轮换

```
┌─────────────────────────────────────────────────────────────┐
│                    Upgrade Sequence                         │
├─────────────────────────────────────────────────────────────┤
│  1. Control Plane (02-cluster)                              │
│     └─> aws_eks_cluster.version = "1.31"                    │
│                                                             │
│  2. Wait for Control Plane (~15-20 minutes)                 │
│     └─> Verify: kubectl get nodes, cluster status           │
│                                                             │
│  3. Platform Add-ons (03-platform)                          │
│     └─> Update add-on versions in Terraform                 │
│     └─> Apply: CoreDNS, VPC CNI, kube-proxy, etc.           │
│                                                             │
│  4. Node Rotation (Automatic)                               │
│     └─> Karpenter detects AMI drift                         │
│     └─> Nodes cordoned, drained, replaced                   │
│     └─> PDBs respected during drain                         │
└─────────────────────────────────────────────────────────────┘
```

### Layer 02 Cluster 升级

```hcl
# 02-cluster/main.tf

variable "kubernetes_version" {
  description = "Target Kubernetes version"
  type        = string
  default     = "1.31"  # Upgrade: 1.30 -> 1.31
}

resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  version  = var.kubernetes_version
  role_arn = aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids              = var.subnet_ids
    endpoint_private_access = true
    endpoint_public_access  = true
    security_group_ids      = [aws_security_group.cluster.id]
  }

  # Auto Mode configuration
  compute_config {
    enabled       = true
    node_pools    = ["general-purpose", "system"]
    node_role_arn = aws_iam_role.node.arn
  }

  kubernetes_network_config {
    ip_family         = "ipv4"
    service_ipv4_cidr = var.service_cidr
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  upgrade_policy {
    support_type = "STANDARD"
  }

  access_config {
    authentication_mode                         = "API_AND_CONFIG_MAP"
    bootstrap_cluster_creator_admin_permissions = false
  }

  # Prevent accidental destruction
  lifecycle {
    prevent_destroy = true
    ignore_changes  = [
      access_config[0].bootstrap_cluster_creator_admin_permissions
    ]
  }

  tags = var.tags
}

# Output for dependency management
output "cluster_version" {
  value = aws_eks_cluster.main.version
}

output "cluster_status" {
  value = aws_eks_cluster.main.status
}
```

**应用升级：**

```bash
#!/bin/bash
# upgrade-control-plane.sh

cd terraform/02-cluster

echo "=== Current Version ==="
terraform output cluster_version

echo ""
echo "=== Planning Upgrade ==="
terraform plan -var="kubernetes_version=1.31" -out=upgrade.plan

echo ""
read -p "Proceed with control plane upgrade? (yes/no): " confirm
if [ "$confirm" != "yes" ]; then
  echo "Upgrade cancelled"
  exit 1
fi

echo ""
echo "=== Applying Upgrade ==="
terraform apply upgrade.plan

echo ""
echo "=== Waiting for Cluster Status ==="
aws eks wait cluster-active --name prod-cluster
echo "Control plane upgrade complete"

echo ""
echo "=== Verify ==="
kubectl version --short
kubectl get nodes
```

### Layer 03 Platform Add-on 升级

```hcl
# 03-platform/addons.tf

variable "addon_versions" {
  description = "Add-on versions compatible with target K8s version"
  type = object({
    vpc_cni       = string
    coredns       = string
    kube_proxy    = string
    ebs_csi       = string
    efs_csi       = string
  })
  default = {
    # Versions for EKS 1.31
    vpc_cni    = "v1.18.3-eksbuild.1"
    coredns    = "v1.11.1-eksbuild.9"
    kube_proxy = "v1.31.0-eksbuild.5"
    ebs_csi    = "v1.35.0-eksbuild.1"
    efs_csi    = "v2.0.7-eksbuild.1"
  }
}

resource "aws_eks_addon" "vpc_cni" {
  cluster_name                = var.cluster_name
  addon_name                  = "vpc-cni"
  addon_version               = var.addon_versions.vpc_cni
  resolve_conflicts_on_update = "OVERWRITE"

  configuration_values = jsonencode({
    enableNetworkPolicy = "true"
    env = {
      ENABLE_PREFIX_DELEGATION = "true"
      WARM_PREFIX_TARGET       = "1"
    }
  })

  tags = var.tags
}

resource "aws_eks_addon" "coredns" {
  cluster_name                = var.cluster_name
  addon_name                  = "coredns"
  addon_version               = var.addon_versions.coredns
  resolve_conflicts_on_update = "OVERWRITE"

  configuration_values = jsonencode({
    replicaCount = 3
    resources = {
      limits = {
        cpu    = "200m"
        memory = "256Mi"
      }
      requests = {
        cpu    = "100m"
        memory = "128Mi"
      }
    }
  })

  tags = var.tags
}

resource "aws_eks_addon" "kube_proxy" {
  cluster_name                = var.cluster_name
  addon_name                  = "kube-proxy"
  addon_version               = var.addon_versions.kube_proxy
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name                = var.cluster_name
  addon_name                  = "aws-ebs-csi-driver"
  addon_version               = var.addon_versions.ebs_csi
  service_account_role_arn    = var.ebs_csi_role_arn
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}

resource "aws_eks_addon" "efs_csi" {
  cluster_name                = var.cluster_name
  addon_name                  = "aws-efs-csi-driver"
  addon_version               = var.addon_versions.efs_csi
  service_account_role_arn    = var.efs_csi_role_arn
  resolve_conflicts_on_update = "OVERWRITE"

  tags = var.tags
}
```

### NodePool 自动轮换

带 Karpenter 的 EKS Auto Mode 会通过 drift detection 自动处理 Node 轮换：

```yaml
# NodePool configuration for Auto Mode
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    # Drift detection triggers node replacement when AMI changes
    budgets:
      - nodes: "10%"      # Max 10% of nodes disrupted at once
      - nodes: "1"        # Or at least 1 node
        schedule: "0 9 * * *"   # Maintenance window
        duration: 8h
```

**AMI 更新策略：**

```yaml
# EC2NodeClass with AMI selection
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    # Auto Mode uses EKS-optimized AMIs
    - alias: al2023@latest

  role: "KarpenterNodeRole-prod-cluster"

  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "prod-cluster"

  # Block device configuration
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
```

### 监控升级进度

**kubectl 命令：**

```bash
#!/bin/bash
# monitor-upgrade.sh

echo "=== Cluster Version ==="
kubectl version --short

echo ""
echo "=== Node Versions ==="
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,'\
'VERSION:.status.nodeInfo.kubeletVersion,'\
'OS-IMAGE:.status.nodeInfo.osImage,'\
'AGE:.metadata.creationTimestamp'

echo ""
echo "=== Node Conditions ==="
kubectl get nodes -o json | jq -r '
  .items[] |
  "\(.metadata.name): \([.status.conditions[] | select(.status=="True") | .type] | join(", "))"
'

echo ""
echo "=== Karpenter Drift Status ==="
kubectl get nodeclaims -o custom-columns=\
'NAME:.metadata.name,'\
'NODE:.status.nodeName,'\
'READY:.status.conditions[?(@.type=="Ready")].status,'\
'AGE:.metadata.creationTimestamp'

echo ""
echo "=== Add-on Status ==="
aws eks list-addons --cluster-name prod-cluster --query 'addons[]' --output text | while read addon; do
  status=$(aws eks describe-addon --cluster-name prod-cluster --addon-name $addon \
    --query 'addon.status' --output text)
  version=$(aws eks describe-addon --cluster-name prod-cluster --addon-name $addon \
    --query 'addon.addonVersion' --output text)
  echo "$addon: $status ($version)"
done
```

**用于监控的 Prometheus 查询：**

```promql
# Node rotation progress
count(kube_node_info) by (kubelet_version)

# Pods being rescheduled during upgrade
sum(rate(kube_pod_container_status_restarts_total[5m])) by (namespace)

# Node drain rate
rate(karpenter_nodes_terminated_total[10m])

# PDB blocking status
kube_poddisruptionbudget_status_pod_disruptions_allowed == 0

# Add-on health
kube_daemonset_status_number_ready{daemonset=~"aws-node|kube-proxy|ebs-csi-node"}
```

**Grafana dashboard panel：**

```json
{
  "title": "Upgrade Progress",
  "panels": [
    {
      "title": "Node Versions",
      "type": "piechart",
      "targets": [{
        "expr": "count(kube_node_info) by (kubelet_version)"
      }]
    },
    {
      "title": "Node Rotation Timeline",
      "type": "timeseries",
      "targets": [{
        "expr": "sum(karpenter_nodes_created_total)",
        "legendFormat": "Created"
      }, {
        "expr": "sum(karpenter_nodes_terminated_total)",
        "legendFormat": "Terminated"
      }]
    }
  ]
}
```

### 处理卡住的升级

```bash
#!/bin/bash
# unstick-upgrade.sh

echo "=== Identifying Stuck Nodes ==="
kubectl get nodes -l "karpenter.sh/lifecycle!=spot" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

echo ""
echo "=== Checking for PDB Blocks ==="
kubectl get pdb -A -o json | jq -r '
  .items[] |
  select(.status.disruptionsAllowed == 0) |
  "BLOCKED: \(.metadata.namespace)/\(.metadata.name) - expectedPods:\(.status.expectedPods), currentHealthy:\(.status.currentHealthy)"
'

echo ""
echo "=== Pods Preventing Drain ==="
# Find pods on nodes being drained
for node in $(kubectl get nodes -o jsonpath='{.items[?(@.spec.unschedulable==true)].metadata.name}'); do
  echo "Node: $node"
  kubectl get pods --all-namespaces --field-selector spec.nodeName=$node \
    -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind'
done

echo ""
echo "=== Force Drain Options ==="
echo "1. Temporarily relax PDB:"
echo "   kubectl patch pdb <name> -n <ns> -p '{\"spec\":{\"maxUnavailable\":1}}'"
echo ""
echo "2. Delete stuck pods (use with caution):"
echo "   kubectl delete pod <name> -n <ns> --grace-period=0 --force"
echo ""
echo "3. Skip PDB check (emergency only):"
echo "   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --disable-eviction"
```

### 回滚限制

EKS Auto Mode 有特定的回滚约束：

| 组件            | 是否可回滚 | 方法                               |
| ------------- | ----- | -------------------------------- |
| Control Plane | 否     | 创建新集群                            |
| Add-ons       | 是     | Terraform apply previous version |
| Node AMI      | 是     | 更新 EC2NodeClass amiSelectorTerms |
| Applications  | 是     | ArgoCD sync to previous revision |

**回滚流程（仅应用）：**

```bash
#!/bin/bash
# rollback-applications.sh

APP_NAME="$1"
TARGET_REVISION="$2"

if [ -z "$APP_NAME" ] || [ -z "$TARGET_REVISION" ]; then
  echo "Usage: $0 <app-name> <target-revision>"
  exit 1
fi

echo "=== Current Application State ==="
argocd app get $APP_NAME

echo ""
echo "=== Available Revisions ==="
argocd app history $APP_NAME

echo ""
echo "=== Rolling Back to $TARGET_REVISION ==="
argocd app rollback $APP_NAME $TARGET_REVISION

echo ""
echo "=== Verify Rollback ==="
argocd app wait $APP_NAME --health --timeout 300
```

***

## 4. Blue/Green 升级策略

### 策略概览

Blue/Green 升级通过并行运行两个完整集群，提供最安全的升级路径：

```
┌─────────────────────────────────────────────────────────────┐
│                Blue/Green Upgrade Flow                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   ┌─────────────┐     ┌─────────────┐     ┌─────────────┐   │
│   │   BLUE      │     │    NLB      │     │   GREEN     │   │
│   │  (Current)  │◄────│  Weighted   │────►│   (New)     │   │
│   │  EKS 1.30   │     │  Routing    │     │  EKS 1.31   │   │
│   └─────────────┘     └─────────────┘     └─────────────┘   │
│         │                   │                   │           │
│         ▼                   ▼                   ▼           │
│   100% → 90% → 50% → 10% → 0%            0% → 10% → ...     │
│                                                             │
│   Phase 1: Green cluster created                            │
│   Phase 2: Platform deployed                                │
│   Phase 3: ArgoCD deploys apps                              │
│   Phase 4: Traffic shifted gradually                        │
│   Phase 5: Blue decommissioned                              │
│                                                             │
└─────────────────────────────────────────────────────────────┘
```

### 分步骤 Blue/Green 升级

#### 步骤 1：创建 Green 集群

```hcl
# terraform/02-cluster-green/main.tf

variable "cluster_version" {
  default = "1.31"
}

variable "cluster_suffix" {
  default = "green"
}

resource "aws_eks_cluster" "green" {
  name     = "prod-cluster-${var.cluster_suffix}"
  version  = var.cluster_version
  role_arn = data.aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids              = data.aws_subnets.private.ids
    endpoint_private_access = true
    endpoint_public_access  = true
    security_group_ids      = [aws_security_group.cluster.id]
  }

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose", "system"]
    node_role_arn = data.aws_iam_role.node.arn
  }

  kubernetes_network_config {
    ip_family         = "ipv4"
    service_ipv4_cidr = "10.101.0.0/16"  # Different from blue
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  tags = merge(var.tags, {
    Environment = "production"
    Cluster     = "green"
    Version     = var.cluster_version
  })
}

output "cluster_name" {
  value = aws_eks_cluster.green.name
}

output "cluster_endpoint" {
  value = aws_eks_cluster.green.endpoint
}

output "cluster_ca_data" {
  value     = aws_eks_cluster.green.certificate_authority[0].data
  sensitive = true
}
```

#### 步骤 2：部署 Platform 组件

```bash
#!/bin/bash
# deploy-platform-green.sh

GREEN_CLUSTER="prod-cluster-green"

echo "=== Updating kubeconfig ==="
aws eks update-kubeconfig --name $GREEN_CLUSTER --alias green

echo ""
echo "=== Deploying Platform Add-ons ==="
cd terraform/03-platform
terraform workspace select green || terraform workspace new green
terraform apply -var="cluster_name=$GREEN_CLUSTER" -auto-approve

echo ""
echo "=== Installing Cert Manager ==="
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.15.0 \
  --set installCRDs=true

echo ""
echo "=== Installing AWS Load Balancer Controller ==="
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=$GREEN_CLUSTER \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

echo ""
echo "=== Installing External DNS ==="
helm upgrade --install external-dns bitnami/external-dns \
  --namespace external-dns --create-namespace \
  --set provider=aws \
  --set aws.zoneType=public \
  --set txtOwnerId=$GREEN_CLUSTER

echo ""
echo "=== Platform deployment complete ==="
kubectl get pods -A
```

#### 步骤 3：注册到 ArgoCD Hub

```yaml
# argocd/cluster-secret-green.yaml
apiVersion: v1
kind: Secret
metadata:
  name: prod-cluster-green
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: production
    cluster-color: green
    kubernetes-version: "1.31"
type: Opaque
stringData:
  name: prod-cluster-green
  server: https://XXXXXXXXXXXXXXXX.gr7.us-west-2.eks.amazonaws.com
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "prod-cluster-green",
        "roleARN": "arn:aws:iam::ACCOUNT_ID:role/argocd-hub-role"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "BASE64_ENCODED_CA_DATA"
      }
    }
```

```bash
#!/bin/bash
# register-argocd-cluster.sh

GREEN_CLUSTER="prod-cluster-green"
GREEN_ENDPOINT=$(aws eks describe-cluster --name $GREEN_CLUSTER \
  --query 'cluster.endpoint' --output text)
GREEN_CA=$(aws eks describe-cluster --name $GREEN_CLUSTER \
  --query 'cluster.certificateAuthority.data' --output text)

# Create cluster secret
cat <<EOF | kubectl apply -f - --context argocd-hub
apiVersion: v1
kind: Secret
metadata:
  name: $GREEN_CLUSTER
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: production
    cluster-color: green
    kubernetes-version: "1.31"
type: Opaque
stringData:
  name: $GREEN_CLUSTER
  server: $GREEN_ENDPOINT
  config: |
    {
      "awsAuthConfig": {
        "clusterName": "$GREEN_CLUSTER",
        "roleARN": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/argocd-hub-role"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "$GREEN_CA"
      }
    }
EOF

echo "Cluster registered with ArgoCD"
argocd cluster list
```

#### 步骤 4：ApplicationSet 部署应用

```yaml
# argocd/applicationset-production.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: production-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          # Generator 1: Clusters
          - clusters:
              selector:
                matchLabels:
                  environment: production
          # Generator 2: Applications
          - git:
              repoURL: https://github.com/company/k8s-apps.git
              revision: HEAD
              directories:
                - path: apps/*
  template:
    metadata:
      name: '{{.path.basename}}-{{.name}}'
      labels:
        app: '{{.path.basename}}'
        cluster: '{{.name}}'
        cluster-color: '{{index .metadata.labels "cluster-color"}}'
    spec:
      project: production
      source:
        repoURL: https://github.com/company/k8s-apps.git
        targetRevision: HEAD
        path: '{{.path.path}}'
        helm:
          valueFiles:
            - values.yaml
            - 'values-{{index .metadata.labels "cluster-color"}}.yaml'
      destination:
        server: '{{.server}}'
        namespace: '{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground
```

#### 步骤 5：Smoke Test

```bash
#!/bin/bash
# smoke-test-green.sh

GREEN_CLUSTER="prod-cluster-green"
aws eks update-kubeconfig --name $GREEN_CLUSTER --alias green

echo "=== Application Health Check ==="
APPS=("api-gateway" "user-service" "order-service" "payment-service")

for app in "${APPS[@]}"; do
  echo "Checking $app..."

  # Deployment status
  ready=$(kubectl get deployment $app -n $app -o jsonpath='{.status.readyReplicas}')
  desired=$(kubectl get deployment $app -n $app -o jsonpath='{.spec.replicas}')

  if [ "$ready" == "$desired" ]; then
    echo "  Deployment: OK ($ready/$desired)"
  else
    echo "  Deployment: FAIL ($ready/$desired)"
    exit 1
  fi

  # Pod health
  unhealthy=$(kubectl get pods -n $app -o jsonpath='{.items[?(@.status.phase!="Running")].metadata.name}')
  if [ -z "$unhealthy" ]; then
    echo "  Pods: OK"
  else
    echo "  Pods: FAIL - $unhealthy"
    exit 1
  fi

  # Service endpoint
  endpoint=$(kubectl get svc $app -n $app -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
  if [ -n "$endpoint" ]; then
    echo "  Service: OK ($endpoint)"
  else
    echo "  Service: Internal only"
  fi
done

echo ""
echo "=== HTTP Health Checks ==="
# Internal health check via port-forward
for app in "${APPS[@]}"; do
  kubectl port-forward svc/$app 8080:80 -n $app &
  PID=$!
  sleep 2

  status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)
  kill $PID 2>/dev/null

  if [ "$status" == "200" ]; then
    echo "$app health check: PASS"
  else
    echo "$app health check: FAIL ($status)"
    exit 1
  fi
done

echo ""
echo "=== All smoke tests passed ==="
```

#### 步骤 6：NLB 权重切换

```hcl
# terraform/04-routing/nlb-weights.tf

variable "blue_weight" {
  description = "Traffic weight for blue cluster (0-100)"
  type        = number
  default     = 100
}

variable "green_weight" {
  description = "Traffic weight for green cluster (0-100)"
  type        = number
  default     = 0
}

resource "aws_lb_target_group" "blue" {
  name        = "prod-api-blue"
  port        = 443
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 10
    protocol            = "TCP"
  }

  tags = {
    Cluster = "blue"
  }
}

resource "aws_lb_target_group" "green" {
  name        = "prod-api-green"
  port        = 443
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 10
    protocol            = "TCP"
  }

  tags = {
    Cluster = "green"
  }
}

resource "aws_lb_listener" "api" {
  load_balancer_arn = aws_lb.api.arn
  port              = 443
  protocol          = "TCP"

  default_action {
    type = "forward"

    forward {
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = var.blue_weight
      }

      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = var.green_weight
      }

      stickiness {
        enabled  = true
        duration = 3600
      }
    }
  }
}

output "current_weights" {
  value = {
    blue  = var.blue_weight
    green = var.green_weight
  }
}
```

**权重切换脚本：**

```bash
#!/bin/bash
# shift-traffic.sh

set -e

BLUE_WEIGHT=${1:-100}
GREEN_WEIGHT=${2:-0}

if [ $((BLUE_WEIGHT + GREEN_WEIGHT)) -ne 100 ]; then
  echo "Error: Weights must sum to 100"
  exit 1
fi

echo "=== Shifting Traffic ==="
echo "Blue: $BLUE_WEIGHT%"
echo "Green: $GREEN_WEIGHT%"

cd terraform/04-routing
terraform apply \
  -var="blue_weight=$BLUE_WEIGHT" \
  -var="green_weight=$GREEN_WEIGHT" \
  -auto-approve

echo ""
echo "=== Verifying Shift ==="
aws elbv2 describe-listeners \
  --load-balancer-arn $(terraform output -raw nlb_arn) \
  --query 'Listeners[0].DefaultActions[0].ForwardConfig.TargetGroups[*].{ARN:TargetGroupArn,Weight:Weight}' \
  --output table

echo ""
echo "=== Monitoring ==="
echo "Watch error rates: https://grafana.company.com/d/traffic-shift"
```

**推荐权重切换计划：**

| 阶段 | Blue | Green | 持续时间  | 验证                  |
| -- | ---- | ----- | ----- | ------------------- |
| 1  | 100% | 0%    | 初始    | Green smoke test 通过 |
| 2  | 90%  | 10%   | 30 分钟 | 错误率稳定               |
| 3  | 50%  | 50%   | 1 小时  | 延迟对比                |
| 4  | 10%  | 90%   | 30 分钟 | 最终验证                |
| 5  | 0%   | 100%  | -     | 迁移完成                |

### 回滚流程

```bash
#!/bin/bash
# rollback-to-blue.sh

echo "=== Emergency Rollback to Blue Cluster ==="

# Immediate traffic shift
cd terraform/04-routing
terraform apply \
  -var="blue_weight=100" \
  -var="green_weight=0" \
  -auto-approve

echo ""
echo "=== Traffic restored to blue cluster ==="

# Document the rollback
cat >> upgrade-log.md << EOF

## Rollback Event - $(date -Iseconds)

**Reason**: [Document reason]
**Action**: Traffic shifted 100% to blue cluster
**Next Steps**:
- [ ] Investigate green cluster issues
- [ ] Review application logs
- [ ] Plan retry attempt
EOF

echo ""
echo "=== Post-Rollback Verification ==="
aws eks update-kubeconfig --name prod-cluster-blue --alias blue
kubectl --context blue get pods -A | grep -v Running
```

### 数据迁移注意事项

存在有状态工作负载时，请谨慎规划数据迁移：

```bash
#!/bin/bash
# data-migration-checklist.sh

echo "=== Stateful Workload Inventory ==="
kubectl get pvc -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,'\
'NAME:.metadata.name,'\
'STORAGE-CLASS:.spec.storageClassName,'\
'SIZE:.spec.resources.requests.storage,'\
'STATUS:.status.phase'

echo ""
echo "=== Database Instances ==="
kubectl get pods -A -l 'app.kubernetes.io/component in (database,db,postgresql,mysql,mongodb)'

echo ""
echo "=== Migration Strategies ==="
cat << 'EOF'
1. AWS Native (RDS, ElastiCache, etc.)
   - No migration needed - external to cluster
   - Verify security group access from green cluster

2. In-Cluster StatefulSets
   - Option A: Velero backup/restore
   - Option B: Application-level replication
   - Option C: Shared EFS storage

3. PersistentVolumes
   - EBS: Snapshot and restore to new AZ
   - EFS: Mount same filesystem from both clusters

EOF
```

### 替代方案：带原生回滚的分区原地升级

由于 Amazon EKS 已添加原生 Kubernetes 版本回滚（2026 年 7 月），如果团队已经在 [NLB Weighted Target Groups](/kubernetes/es/yun-wei-zhi-nan/ops/02-infrastructure-advanced.md#2-nlb-weighted-target-groups) 后面按可用区运行一个集群，那么相比长期维护第二套完整集群 fleet，可以采用更轻量的选项：逐个可用区对每个 zonal cluster 原地升级，在升级窗口期间仅使用现有的加权路由来排空流量，并依赖 EKS 的原生回滚（而不是第二个集群）作为安全网。

```
┌────────────────────────────────────────────────────────────┐
│      Zonal In-Place Upgrade (rollback as safety net)       │
├────────────────────────────────────────────────────────────┤
│                                                              │
│   ┌─────────────┐     ┌─────────────┐    ┌─────────────┐   │
│   │   AZ-a      │◄────│    NLB      │───►│   AZ-c      │   │
│   │  1.30→1.31  │     │  Weighted   │    │   1.30      │   │
│   └─────────────┘     └─────────────┘    └─────────────┘   │
│                                                              │
│   1. Shift NLB weight: AZ-a -> 0%, AZ-c -> 100%             │
│   2. Upgrade AZ-a's control plane + nodes in place          │
│   3. Validate AZ-a, shift weight back to 50/50              │
│   4. Repeat for AZ-c                                        │
│   5. If AZ-a misbehaves post-upgrade: use EKS's native      │
│      rollback (control plane only, N -> N-1, within 7 days) │
│      instead of standing up a third cluster                 │
└────────────────────────────────────────────────────────────┘
```

**何时比永久 Blue/Green fleet 更适合：**

* 你已经为了可用性运行 zonal cluster，而不是专门为了升级
* 你希望避免运行两套完整集群 fleet 的稳态成本
* 你的升级窗口可以接受约 7 天的回滚资格期，而不是即时的集群级 failback

**何时应继续采用本节中的完整 Blue/Green fleet：**

* 你需要在切换前，在一个完全独立的集群上用真实生产流量验证新版本；原生回滚只回滚 Control Plane，不会回滚任何在原地完成的 Node/AMI/Add-on 变更
* 不满足回滚资格条件（集群以目标版本创建、已经超过 7 天、已经应用了另一次升级，或启用了向后不兼容功能）——请参阅 [EKS Upgrade Strategies — Rollback Procedure](/kubernetes/es/amazon-eks/08-eks-upgrades.md#rollback-procedure)

（来源：[Amazon EKS announces Kubernetes version rollback](https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-eks-version-rollback)，2026 年 7 月）

***

## 5. 升级后验证

### 综合健康检查脚本

```bash
#!/bin/bash
# post-upgrade-validation.sh

set -e

CLUSTER_NAME="${1:-prod-cluster}"
REPORT_DIR="./post-upgrade-reports/$(date +%Y%m%d-%H%M%S)"

mkdir -p $REPORT_DIR

echo "=============================================="
echo "Post-Upgrade Validation"
echo "Cluster: $CLUSTER_NAME"
echo "=============================================="

# 1. Cluster version verification
echo ""
echo "[1/7] Verifying cluster version..."
CLUSTER_VERSION=$(aws eks describe-cluster --name $CLUSTER_NAME \
  --query 'cluster.version' --output text)
echo "Cluster version: $CLUSTER_VERSION"

# 2. Node status
echo ""
echo "[2/7] Checking node status..."
kubectl get nodes -o wide | tee $REPORT_DIR/nodes.txt

NOT_READY=$(kubectl get nodes --no-headers | grep -v " Ready " | wc -l)
if [ "$NOT_READY" -gt 0 ]; then
  echo "WARNING: $NOT_READY nodes not ready"
  kubectl get nodes --no-headers | grep -v " Ready "
fi

# Verify all nodes on target version
NODE_VERSIONS=$(kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}' | tr ' ' '\n' | sort -u)
echo "Node versions: $NODE_VERSIONS"

# 3. Pod status
echo ""
echo "[3/7] Checking pod status..."
PROBLEM_PODS=$(kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  --no-headers 2>/dev/null | wc -l)

if [ "$PROBLEM_PODS" -gt 0 ]; then
  echo "WARNING: $PROBLEM_PODS pods not in Running/Succeeded state"
  kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
fi

# 4. Service connectivity
echo ""
echo "[4/7] Testing service connectivity..."
SERVICES=("api-gateway:api-gateway" "user-service:user-service")

for svc in "${SERVICES[@]}"; do
  ns="${svc%%:*}"
  name="${svc##*:}"

  endpoint=$(kubectl get endpoints $name -n $ns -o jsonpath='{.subsets[0].addresses[0].ip}' 2>/dev/null)
  if [ -n "$endpoint" ]; then
    echo "  $ns/$name: OK (endpoint: $endpoint)"
  else
    echo "  $ns/$name: WARNING - no endpoints"
  fi
done

# 5. Ingress reachability
echo ""
echo "[5/7] Testing ingress reachability..."
INGRESS_HOST=$(kubectl get ingress -n api-gateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}' 2>/dev/null)

if [ -n "$INGRESS_HOST" ]; then
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://$INGRESS_HOST/health" --connect-timeout 5 || echo "timeout")
  echo "  Ingress ($INGRESS_HOST): $HTTP_STATUS"
else
  echo "  No ingress found or not yet provisioned"
fi

# 6. Add-on status
echo ""
echo "[6/7] Verifying add-ons..."
for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  status=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.status' --output text 2>/dev/null || echo "NOT_INSTALLED")
  version=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon \
    --query 'addon.addonVersion' --output text 2>/dev/null || echo "-")
  echo "  $addon: $status ($version)"
done

# 7. Critical workload health
echo ""
echo "[7/7] Checking critical workloads..."
CRITICAL_DEPLOYMENTS=("kube-system:coredns" "kube-system:aws-load-balancer-controller" "argocd:argocd-server")

for deploy in "${CRITICAL_DEPLOYMENTS[@]}"; do
  ns="${deploy%%:*}"
  name="${deploy##*:}"

  ready=$(kubectl get deployment $name -n $ns -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
  desired=$(kubectl get deployment $name -n $ns -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")

  if [ "$ready" == "$desired" ] && [ "$ready" != "0" ]; then
    echo "  $ns/$name: OK ($ready/$desired)"
  else
    echo "  $ns/$name: WARNING ($ready/$desired)"
  fi
done

echo ""
echo "=============================================="
echo "Validation complete. Review any warnings above."
echo "=============================================="
```

### Smoke Test

```bash
#!/bin/bash
# smoke-tests.sh

echo "=== HTTP Smoke Tests ==="

# Define test endpoints
declare -A ENDPOINTS=(
  ["api-gateway"]="https://api.company.com/health"
  ["web-app"]="https://www.company.com/"
  ["admin-portal"]="https://admin.company.com/health"
)

for name in "${!ENDPOINTS[@]}"; do
  url="${ENDPOINTS[$name]}"

  start_time=$(date +%s%N)
  status=$(curl -s -o /dev/null -w "%{http_code}" "$url" --connect-timeout 10 || echo "timeout")
  end_time=$(date +%s%N)

  latency=$(( (end_time - start_time) / 1000000 ))

  if [ "$status" == "200" ]; then
    echo "  $name: PASS (${latency}ms)"
  else
    echo "  $name: FAIL (status: $status)"
  fi
done

echo ""
echo "=== Database Connectivity ==="
# Test via application pods
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $DB_HOST 5432 2>&1' || echo "DB connectivity test failed"

echo ""
echo "=== Message Queue Connectivity ==="
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $MQ_HOST 5672 2>&1' || echo "MQ connectivity test failed"

echo ""
echo "=== Cache Connectivity ==="
kubectl exec -n api-gateway deploy/api-gateway -- \
  /bin/sh -c 'nc -zv $REDIS_HOST 6379 2>&1' || echo "Cache connectivity test failed"
```

### 使用 PromQL 进行指标对比

对比升级前后的关键指标：

```promql
# Error rate comparison (5xx errors)
# Run before upgrade, save value, compare after

# Current error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# P99 latency comparison
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Resource utilization - CPU
avg(rate(container_cpu_usage_seconds_total{namespace!~"kube-system|monitoring"}[5m])) by (namespace)

# Resource utilization - Memory
avg(container_memory_working_set_bytes{namespace!~"kube-system|monitoring"}) by (namespace) / 1024 / 1024

# Pod restart rate (should be near zero post-upgrade stabilization)
sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace)
```

**Grafana 对比 dashboard：**

```json
{
  "title": "Upgrade Comparison",
  "templating": {
    "list": [{
      "name": "comparison_time",
      "type": "custom",
      "options": [
        {"value": "now-1d", "text": "Yesterday"},
        {"value": "now-7d", "text": "Last Week"}
      ]
    }]
  },
  "panels": [
    {
      "title": "Error Rate: Before vs After",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
          "legendFormat": "Current"
        },
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m] offset 1d)) / sum(rate(http_requests_total[5m] offset 1d)) * 100",
          "legendFormat": "Before Upgrade"
        }
      ]
    },
    {
      "title": "P99 Latency: Before vs After",
      "type": "timeseries",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "Current"
        },
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m] offset 1d)) by (le))",
          "legendFormat": "Before Upgrade"
        }
      ]
    }
  ]
}
```

### 监控期

**升级后监控检查清单：**

| 时间范围    | 关注领域 | 操作                       |
| ------- | ---- | ------------------------ |
| 0-1 小时  | 关键故障 | 观察错误率、Pod 重启             |
| 1-4 小时  | 性能回退 | 对比延迟、吞吐量                 |
| 4-24 小时 | 稳定性  | 内存泄漏、逐步退化                |
| 1-7 天   | 边界情况 | Batch job、scheduled task |

```bash
#!/bin/bash
# monitoring-period-alerts.sh

# Create temporary high-sensitivity alerts for post-upgrade period
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: post-upgrade-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
    - name: post-upgrade
      rules:
        - alert: PostUpgradeHighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.01
          for: 2m
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "Error rate elevated post-upgrade"

        - alert: PostUpgradeLatencyIncrease
          expr: |
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
            >
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m] offset 1d)) by (le)) * 1.5
          for: 5m
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "P99 latency increased >50% vs yesterday"

        - alert: PostUpgradePodRestarts
          expr: |
            sum(increase(kube_pod_container_status_restarts_total[30m])) by (namespace) > 5
          labels:
            severity: warning
            context: post-upgrade
          annotations:
            summary: "Elevated pod restarts in {{ \$labels.namespace }}"
EOF

echo "Post-upgrade alerts created. Remove after stabilization:"
echo "  kubectl delete prometheusrule post-upgrade-alerts -n monitoring"
```

### Runbook 更新

升级成功后，更新运维文档：

```markdown
# Post-Upgrade Runbook Update Checklist

## Version Information
- [ ] Update cluster version in documentation
- [ ] Update add-on version matrix
- [ ] Document any new features enabled

## Configuration Changes
- [ ] Document any API changes made
- [ ] Update Terraform module versions
- [ ] Update Helm chart versions

## Lessons Learned
- [ ] Document any issues encountered
- [ ] Note workarounds or fixes applied
- [ ] Update pre-upgrade checklist based on experience

## Timeline
- [ ] Record actual upgrade duration
- [ ] Note any deviations from plan
- [ ] Update time estimates for future upgrades
```

***

## 相关文档

* [EKS 版本升级](/kubernetes/es/amazon-eks/08-eks-upgrades.md) - 核心升级概念
* [Node 生命周期管理](/kubernetes/es/amazon-eks/eks-auto-mode/07-node-lifecycle.md) - Auto Mode Node 轮换
* [ArgoCD 多集群](/kubernetes/es/yun-wei-zhi-nan/ops/04-gitops-multi-cluster.md) - ApplicationSet 配置
* [可观测性 Stack](/kubernetes/es/yun-wei-zhi-nan/ops/09-observability-stack.md) - 监控设置

***

< [上一节：资源优化](/kubernetes/es/yun-wei-zhi-nan/ops/10-resource-optimization.md) | [目录](/kubernetes/es/yun-wei-zhi-nan/ops.md) | [下一节：活动容量规划](/kubernetes/es/yun-wei-zhi-nan/ops/12-event-capacity-planning.md) >