Envoy Cluster 해부 — 포트 4-layer, subset, DestinationRule 매핑 (출처: "Istio 운영 노하우" 정리 + Istio 1.30 공식 문서)

# NGINX upstream — backend 목록이 거의 전부
upstream reviews {
  server 10.0.1.10:9080;
  server 10.0.2.20:9080;
}

NGINX upstream
  ≈ backend server group

Envoy cluster
  = backend server group
  + dynamic discovery (EDS/DNS/STATIC)
  + LB policy
  + connection pool
  + circuit breaker
  + outlier detection
  + TLS/mTLS transport socket
  + HTTP/1.1·HTTP/2 protocol option
  + observability/stat metadata

DestinationRule          = Istio 고수준 정책 CRD (사람이 쓰는 것)
Envoy cluster            = istiod가 DestinationRule/Service/ServiceEntry를 보고 생성한 Envoy xDS Cluster resource
istioctl proxy-config cluster = 특정 proxy가 실제로 받은 Envoy cluster 설정을 보는 명령

15001 (capture port)
  = iptables가 outbound TCP를 Envoy로 빨아들이는 입구 포트
  = app이 어디로 가든 일단 여기로 redirect됨 (Istio 내부 convention, 표준 의미 없음)

listener 9080 (virtual listener)
  = Envoy 내부에서 "원래 목적지 포트가 9080이었네" 하고 고른 가상 listener
  = downstream 요청을 분류하는 수신 쪽 포트

cluster 의 9080 (service port)
  = outbound|9080||svc 의 가운데 숫자
  = Envoy가 upstream으로 보낼 대상 Kubernetes Service port

endpoint :8080 (targetPort)
  = 실제 Pod의 targetPort/containerPort
  = EDS가 내려주는 실제 backend IP의 포트

cluster:  outbound|80||istio-egressgateway.istio-system.svc.cluster.local
endpoint: 10.255.126.47:8080 HEALTHY outbound|80||istio-egressgateway.istio-system.svc.cluster.local

apiVersion: v1
kind: Service
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  ports:
  - name: http2
    port: 80          # cluster 이름에 들어가는 service port
    targetPort: 8080  # EDS endpoint에 나오는 실제 Pod port

outbound | 9080 | v1 | reviews.default.svc.cluster.local
 방향       포트   subset   서비스 FQDN

outbound|9080||reviews.default.svc.cluster.local      ← 전체 endpoint pool
outbound|9080|v1|reviews.default.svc.cluster.local    ← v1 subset만
outbound|9080|v2|reviews.default.svc.cluster.local    ← v2 subset만

reviews-v1 Pod labels: app=reviews, version=v1
reviews-v2 Pod labels: app=reviews, version=v2
reviews-v3 Pod labels: app=reviews, version=v3

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

subset은 Kubernetes Service가 아니다.
subset은 Envoy의 순수 개념도 아니다.
subset은 Istio DestinationRule 개념이고, Istio가 이를 Envoy cluster로 컴파일한다.

apiVersion: v1
kind: Service
metadata:
  name: reviews
spec:
  selector:
    app: reviews
  ports:
  - name: http
    port: 9080

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
spec:
  hosts:
  - api.vendor.com
  ports:
  - number: 443
    name: tls
    protocol: TLS
  resolution: DNS

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: legacy-db
spec:
  hosts:
  - legacy-db.internal
  ports:
  - number: 5432
    name: tcp-postgres
    protocol: TCP
  resolution: STATIC
  endpoints:
  - address: 10.10.1.10
  - address: 10.10.1.11

trafficPolicy:
  loadBalancer:
    simple: LEAST_REQUEST

trafficPolicy:
  loadBalancer:
    consistentHash:
      httpHeaderName: x-user-id   # 또는 httpCookie: {name: user, ttl: 0s}

connectionPool:
  tcp:
    maxConnections: 200
    connectTimeout: 500ms
    tcpKeepalive:
      time: 7200s
      interval: 75s
  http:
    http1MaxPendingRequests: 2000
    http2MaxRequests: 20000
    maxRequestsPerConnection: 0
    maxRetries: 1
    idleTimeout: 30m
    h2UpgradePolicy: UPGRADE

outlierDetection:
  consecutive5xxErrors: 7
  consecutiveGatewayErrors: 5
  interval: 30s
  baseEjectionTime: 5m
  maxEjectionPercent: 50
  minHealthPercent: 50

trafficPolicy:
  tls:
    mode: ISTIO_MUTUAL   # client sidecar → server sidecar, Istio workload cert로 mTLS

h2UpgradePolicy: UPGRADE
  → upstream을 HTTP/2로 올려서 보낼 수 있음

useClientProtocol: true
  → client가 HTTP/1.1이면 upstream도 HTTP/1.1, client가 HTTP/2면 upstream도 HTTP/2
  → 이 경우 h2UpgradePolicy는 효과가 없다

apiVersion: v1
kind: Namespace
metadata:
  name: mesh-demo
  labels:
    istio-injection: enabled

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-server
  namespace: mesh-demo

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server-v1
  namespace: mesh-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-server
      version: v1
  template:
    metadata:
      labels:
        app: llm-server
        version: v1
    spec:
      serviceAccountName: llm-server
      containers:
      - name: app
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - name: http
          containerPort: 8080

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server-v2
  namespace: mesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-server
      version: v2
  template:
    metadata:
      labels:
        app: llm-server
        version: v2
    spec:
      serviceAccountName: llm-server
      containers:
      - name: app
        image: gcr.io/google-samples/hello-app:2.0
        ports:
        - name: http
          containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: llm-server
  namespace: mesh-demo
spec:
  selector:
    app: llm-server
  ports:
  - name: http
    port: 80
    targetPort: 8080

---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: llm-server
  namespace: mesh-demo
spec:
  host: llm-server.mesh-demo.svc.cluster.local

  # 이 trafficPolicy는 기본적으로 이 destination host의 모든 port에 적용된다.
  # 다만 아래 portLevelSettings가 같은 port를 override하면,
  # destination-level 설정이 자동 상속되지 않는다.
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST

    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 300ms
        tcpKeepalive:
          time: 7200s
          interval: 75s
      http:
        http1MaxPendingRequests: 1000
        http2MaxRequests: 10000
        maxRequestsPerConnection: 0
        maxRetries: 2
        idleTimeout: 30m
        h2UpgradePolicy: UPGRADE

    outlierDetection:
      consecutive5xxErrors: 7
      consecutiveGatewayErrors: 5
      interval: 30s
      baseEjectionTime: 5m
      maxEjectionPercent: 50
      minHealthPercent: 50

    tls:
      mode: ISTIO_MUTUAL

    # 이 port.number는 Service port 80이다.
    # targetPort/containerPort 8080이 아니다.
    portLevelSettings:
    - port:
        number: 80
      loadBalancer:
        simple: LEAST_REQUEST
      connectionPool:
        tcp:
          maxConnections: 200
          connectTimeout: 500ms
        http:
          http1MaxPendingRequests: 2000
          http2MaxRequests: 20000
          maxRequestsPerConnection: 0
          maxRetries: 1
          idleTimeout: 30m
          h2UpgradePolicy: UPGRADE
      outlierDetection:
        consecutive5xxErrors: 7
        consecutiveGatewayErrors: 5
        interval: 30s
        baseEjectionTime: 5m
        maxEjectionPercent: 50
        minHealthPercent: 50
      tls:
        mode: ISTIO_MUTUAL

  # subset은 Service 뒤 endpoint들을 label 기준으로 나눈 이름 있는 그룹이다.
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      loadBalancer:
        simple: LEAST_REQUEST

  - name: v2
    labels:
      version: v2
    trafficPolicy:
      loadBalancer:
        simple: LEAST_REQUEST
      connectionPool:
        http:
          http2MaxRequests: 5000
          maxRetries: 0

---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: llm-server
  namespace: mesh-demo
spec:
  hosts:
  - llm-server.mesh-demo.svc.cluster.local
  http:
  - name: canary-by-header
    match:
    - headers:
        x-canary:
          exact: v2
    route:
    - destination:
        host: llm-server.mesh-demo.svc.cluster.local
        subset: v2
        port:
          number: 80

  - name: default-90-10
    route:
    - destination:
        host: llm-server.mesh-demo.svc.cluster.local
        subset: v1
        port:
          number: 80
      weight: 90
    - destination:
        host: llm-server.mesh-demo.svc.cluster.local
        subset: v2
        port:
          number: 80
      weight: 10

Service.port 80  ==  cluster 이름의 PORT  ==  portLevelSettings.port.number  ==  VS destination.port.number
Service.targetPort 8080  ==  containerPort  ==  EDS endpoint port
subset 이름 v1/v2  ==  DR.subsets[].name  ==  VS destination.subset  ==  cluster 이름의 SUBSET

Envoy cluster (CDS):
  outbound|80||llm-server.mesh-demo.svc.cluster.local       ← 전체
  outbound|80|v1|llm-server.mesh-demo.svc.cluster.local     ← v1 subset
  outbound|80|v2|llm-server.mesh-demo.svc.cluster.local     ← v2 subset

Endpoint (EDS) — 실제 Pod targetPort 8080:
  10.244.1.11:8080  HEALTHY  outbound|80|v1|llm-server...
  10.244.2.18:8080  HEALTHY  outbound|80|v1|llm-server...
  10.244.3.22:8080  HEALTHY  outbound|80|v2|llm-server...

# 1) 원래 목적지 포트의 virtual listener 확인
istioctl proxy-config listener <client-pod> -n <ns> --port 9080

# 2) route가 어떤 cluster를 가리키는지 확인
istioctl proxy-config route <client-pod> -n <ns> --name 9080 -o json

# 3) service port 기준 cluster 확인 (FQDN + port)
istioctl proxy-config cluster <client-pod> -n <ns> \
  --fqdn reviews.default.svc.cluster.local --port 9080

# 4) cluster 안의 실제 endpoint port 확인 (여기서 targetPort가 드러남)
istioctl proxy-config endpoint <client-pod> -n <ns> \
  --cluster 'outbound|9080||reviews.default.svc.cluster.local'

# 5) transport socket(TLS mode)이 잘 들어갔는지
istioctl proxy-config cluster <client-pod> -n <ns> \
  --fqdn reviews.default.svc.cluster.local -o json | jq '.[].transportSocket'

SERVICE FQDN                          PORT  SUBSET  DIRECTION  TYPE
reviews.default.svc.cluster.local     9080  -       outbound   EDS
reviews.default.svc.cluster.local     9080  v1      outbound   EDS
reviews.default.svc.cluster.local     9080  v2      outbound   EDS

istioctl proxy-config route    <pod> -n <ns>   # route가 원하는 cluster를 가리키나? (NR 의심)
istioctl proxy-config cluster  <pod> -n <ns>   # 그 cluster가 존재하나? (NC 의심)
istioctl proxy-config endpoint <pod> -n <ns>   # healthy endpoint가 있나? (UH 의심)
istioctl proxy-config secret   <pod> -n <ns>   # mTLS cert가 있나? (UF/handshake 의심)

포트 4-layer:
  15001       = outbound capture port (iptables 입구)
  listener N  = 원래 목적지 포트 라벨 (virtual listener)
  cluster N   = destination Service port
  endpoint N  = 실제 Pod targetPort/containerPort
  → Service port 80 ↔ targetPort 8080 분리는 정상

Envoy cluster = upstream backend pool
  + discovery(EDS/DNS/STATIC/ORIGINAL_DST=Passthrough) + LB + connectionPool + circuit breaker
  + outlier detection + TLS transport socket + HTTP protocol option

cluster 이름 = outbound|PORT|SUBSET|FQDN   ('||'면 subset 없음 = 전체 pool)
subset       = DestinationRule의 label 기반 named 그룹 (Service도, Envoy 순수개념도 아님)

DestinationRule → Envoy cluster 매핑:
  loadBalancer      → lb_policy
  connectionPool    → circuit_breakers / connection pool (초과 시 UO)
  outlierDetection  → outlier_detection (과한 maxEjectionPercent → UH)
  tls               → transport_socket (ISTIO_MUTUAL = workload cert mTLS)

portLevelSettings.port.number = Service port (targetPort 아님)
  + port-level override는 destination-level을 부분 상속하지 않는다 (생략 필드는 기본값)

진단 순서 = listener → route → cluster → endpoint → secret  (flag NR/NC/UH/UF가 layer를 지목)