k8s多集群的流量治理（3、通过 karmada + istio + argo-rollout 进行多集群编排（2023-11-12））

Linux云计算+运维开发

已于 2023-11-15 14:48:52 修改

阅读量434

点赞数

CC 4.0 BY-SA版权

文章标签： kubernetes istio 容器

于 2023-11-15 14:47:22 首次发布

本文链接：https://blog.csdn.net/weixin_44797299/article/details/134420054

本文介绍了如何使用Karmada、Istio和ArgoRollout在四个集群中进行多集群编排，包括金丝雀发布策略，如何部署Rollout、Services、IstioVirtualService和Gateway，并实现前端和后端的流量路由控制。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

通过 karmada + istio + argo-rollout 进行多集群编排（2023-11-12）

后端发布方案：

四个集群，每个集群都部署 Rollout、Services

mkdir -p ~/helloworld-rollout-yml

cat > helloworld-stable-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: helloworld-stable
  namespace: helloworld-rollout
spec:
  replicas: 4 # 副本数
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {} # 人工卡点
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
      - setWeight: 100
      - pause: {} # 人工卡点
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:stable
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
EOF

cat > helloworld-stable-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  namespace: helloworld-rollout
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: helloworld
EOF

cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
 name: helloworld-stable-propagation
 namespace: helloworld-rollout
spec:
 resourceSelectors:
   - apiVersion: argoproj.io/v1alpha1  # 修正为 Argo Rollout 的 apiVersion
     kind: Rollout
     name: helloworld-stable
   - apiVersion: v1
     kind: Service
     name: helloworld
 placement:
   clusterAffinity:
     clusterNames:
       - k8s-master-beijing
       - k8s-master-shanghai
       - k8s-master-guangzhou
       - k8s-master-shenzhen
   replicaScheduling:
     replicaDivisionPreference: Weighted
     replicaSchedulingType: Divided
     weightPreference:
       staticWeightList:
         - targetCluster:
             clusterNames:
               - k8s-master-beijing
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shanghai
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-guangzhou
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shenzhen
           weight: 1
EOF

共部署4个副本，每个集群分配一个副本（这里以四个集群为例！！！）

前端发布方案（无金丝雀）：

四个集群，每个集群都部署 Rollout、Services、Istio VirtualService 和 Istio Gateway

mkdir -p ~/helloworld-rollout-yml

cat > helloworld-stable-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: helloworld-stable
  namespace: helloworld-rollout
spec:
  replicas: 4
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {} # 人工卡点
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
      - setWeight: 100
      - pause: {} # 人工卡点
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:stable
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
EOF

cat > helloworld-stable-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  namespace: helloworld-rollout
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: helloworld
EOF

cat > helloworld-stable-vsvc.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld-vsvc
  namespace: helloworld-rollout
spec:
  gateways:
  - helloworld-gateway
  hosts:
  - "helloworld.huanghuanhui.cloud"
  http:
  - name: primary
    route:
    - destination:
        host: prd-vue-svc-stable
      weight: 100
    - destination:
        host: prd-vue-svc-canary
      weight: 0
EOF

cat > helloworld-stable-gateway.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: helloworld-gateway
  namespace: helloworld-rollout
spec:
  selector:
    istio: ingressgateway # 默认创建的 istio ingressgateway pod 有这个 Label
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "helloworld.huanghuanhui.cloud" # 匹配所有 host
    tls:
      mode: SIMPLE
      credentialName: helloworld-rollout-tls-secret
EOF

# 所有的istio的证书都放在（istio-system）命名空间下
kubectl create secret -n istio-system \
tls prd-vue-tls-secret \
--key=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud.key \
--cert=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud_bundle.crt

cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
 name: helloworld-stable-propagation
 namespace: helloworld-rollout
spec:
 resourceSelectors:
   - apiVersion: argoproj.io/v1alpha1  # 修正为 Argo Rollout 的 apiVersion
     kind: Rollout
     name: helloworld-stable
   - apiVersion: v1
     kind: Service
     name: helloworld
 placement:
   clusterAffinity:
     clusterNames:
       - k8s-master-beijing
       - k8s-master-shanghai
       - k8s-master-guangzhou
       - k8s-master-shenzhen
   replicaScheduling:
     replicaDivisionPreference: Weighted
     replicaSchedulingType: Divided
     weightPreference:
       staticWeightList:
         - targetCluster:
             clusterNames:
               - k8s-master-beijing
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shanghai
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-guangzhou
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shenzhen
           weight: 1
EOF

共部署4个副本，每个集群分配一个副本（这里以四个集群为例！！！）

前端发布方案（金丝雀）：

四个集群，每个集群都部署 Rollout、Services、Istio VirtualService 和 Istio Gateway

mkdir -p ~/helloworld-rollout-yml

cat > helloworld-rollout.yml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: helloworld
  namespace: helloworld-rollout
spec:
  replicas: 4
  strategy:
    canary:
      canaryService: helloworld-svc-canary # 关联 canary Service
      stableService: helloworld-svc-stable # 关联 stable Service
      trafficRouting:
        managedRoutes:
          - name: "header-route-1"
        istio:
          virtualServices:
          - name: helloworld-vsvc # 关联的 Istio virtualService
            routes:
            - primary
      steps:
      - setHeaderRoute:
          name: "header-route-1"
          match:
            - headerName: "X-canary"
              headerValue:
                exact: "test-user"
      - pause: {duration: 10}
      - setCanaryScale:
          weight: 20
      - pause: {} # 人工卡点（当有新版本上线的时候，给canary版本20%流量，并且暂停新版本继续更新，当测试人员完成测试可以继续更新，从而达到前端可以一直触发构建并且直接上到金丝雀上）
      - setCanaryScale:
          weight: 40
      - pause: {duration: 10}
      - setCanaryScale:
          weight: 60
      - pause: {duration: 10}
      - setCanaryScale:
          weight: 80
      - pause: {duration: 10}
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: ccr.ccs.tencentyun.com/huanghuanhui/helloworld:canary
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
EOF

cat > helloworld-rollout-svc.yml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: helloworld-svc-stable
  namespace: helloworld-rollout
  labels:
    app: helloworld
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: helloworld

---
apiVersion: v1
kind: Service
metadata:
  name: helloworld-svc-canary
  namespace: helloworld-rollout
  labels:
    app: helloworld
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: helloworld
EOF

cat > helloworld-vsvc.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld-vsvc
  namespace: helloworld-rollout
spec:
  gateways:
  - helloworld-gateway
  hosts:
  - "helloworld.huanghuanhui.cloud"
  http:
  - name: primary
    route:
    - destination:
        host: helloworld-svc-stable
      weight: 100
    - destination:
        host: helloworld-svc-canary
      weight: 0
EOF

cat > helloworld-gateway.yml << 'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: helloworld-gateway
  namespace: helloworld-rollout
spec:
  selector:
    istio: ingressgateway # 默认创建的 istio ingressgateway pod 有这个 Label
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "helloworld.huanghuanhui.cloud" # 匹配所有 host
    tls:
      mode: SIMPLE
      credentialName: helloworld-rollout-tls-secret
EOF

# 所有的istio的证书都放在（istio-system）命名空间下
kubectl create secret -n istio-system \
tls prd-vue-tls-secret \
--key=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud.key \
--cert=/root/ssl/huanghuanhui.cloud_nginx/huanghuanhui.cloud_bundle.crt

cat > helloworld-stable-policy.yml << 'EOF'
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
 name: helloworld-stable-propagation
 namespace: helloworld-rollout
spec:
 resourceSelectors:
   - apiVersion: argoproj.io/v1alpha1  # 修正为 Argo Rollout 的 apiVersion
     kind: Rollout
     name: helloworld-stable
   - apiVersion: v1
     kind: Service
     name: helloworld
 placement:
   clusterAffinity:
     clusterNames:
       - k8s-master-beijing
       - k8s-master-shanghai
       - k8s-master-guangzhou
       - k8s-master-shenzhen
   replicaScheduling:
     replicaDivisionPreference: Weighted
     replicaSchedulingType: Divided
     weightPreference:
       staticWeightList:
         - targetCluster:
             clusterNames:
               - k8s-master-beijing
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shanghai
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-guangzhou
           weight: 1
         - targetCluster:
             clusterNames:
               - k8s-master-shenzhen
           weight: 1
EOF

共部署4个副本，每个集群分配一个副本（这里以四个集群为例！！！）

===

1、ruoyi-gateway（部署 Rollout、Services、Istio VirtualService 和 Istio Gateway）

2、ruoyi-auth（部署 Rollout）

3、ruoyi-system（部署 Rollout）

4、ruoyi-vue（部署 Rollout、Services、Istio VirtualService 和 Istio Gateway）

正常的流量走vue的stable版本，连接stable的gateway

带请求头的流量走vue的canary版本，连接canary版本的gateway

访问前端canary版本，后端也走canary版本；访问前端stable版本，后端也走stable版本；

问题1：ruoyi-auth、ruoyi-system怎样区分流量？？？

k8s多集群的流量治理（3、通过 karmada + istio + argo-rollout 进行多集群编排（2023-11-12） ）

通过 karmada + istio + argo-rollout 进行多集群编排（2023-11-12）

后端发布方案：

前端发布方案（无金丝雀）：

前端发布方案（金丝雀）：

k8s多集群的流量治理（3、通过 karmada + istio + argo-rollout 进行多集群编排（2023-11-12））