=== kubectl create namespace istio-vt-t68 === namespace/istio-vt-t68 created === kubectl label namespace istio-vt-t68 istio-injection=enabled === namespace/istio-vt-t68 labeled === kubectl apply -f manifest.yaml === pod/client created deployment.apps/gate-echo-v1 created deployment.apps/gate-echo-v2 created service/gate-echo created === kubectl -n istio-vt-t68 wait --for=condition=Ready pod/client --timeout=90s === pod/client condition met === kubectl -n istio-vt-t68 wait --for=condition=available deploy/gate-echo-v1 deploy/gate-echo-v2 --timeout=60s === deployment.apps/gate-echo-v1 condition met deployment.apps/gate-echo-v2 condition met === kubectl -n istio-vt-t68 get pods -o wide === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 10s 10.255.194.103 k8s-worker1 gate-echo-v1-665cbff5b-9dfrd 2/2 Running 0 10s 10.255.126.48 k8s-worker2 gate-echo-v2-868545bd96-mfdcw 2/2 Running 0 10s 10.255.159.134 k8s-master1 === STEP 1: check subset cluster count BEFORE any DestinationRule exists === CMD: istioctl proxy-config cluster client.istio-vt-t68 -o json | jq '[.[] | select(.name | contains("gate-echo") and contains("|v1|"))] | length' 0 --- full list of gate-echo related clusters BEFORE DR (for context) --- [ "outbound|80||gate-echo.istio-vt-t68.svc.cluster.local" ] === STEP 2: apply DestinationRule (v1/v2 subsets, no VirtualService yet) === apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: gate-echo-dr namespace: istio-vt-t68 spec: host: gate-echo.istio-vt-t68.svc.homelab.local subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 destinationrule.networking.istio.io/gate-echo-dr created === sleep 5 === === STEP 3: check subset clusters AFTER DR applied (no VS yet) === CMD: istioctl proxy-config cluster client.istio-vt-t68 -o json | jq '[.[] | select(.name | contains("gate-echo"))] | map(.name)' [ "outbound|80||gate-echo.istio-vt-t68.svc.cluster.local" ] === DIAGNOSTIC: hypothesis check - istiod internal service registry uses .svc.cluster.local regardless of real k8s clusterDomain (homelab.local) === --- checking istiod --domain / clusterDomain args --- ["discovery","--monitoringAddr=:15014","--log_output_level=default:info","--domain","cluster.local","--keepaliveMaxServerConnectionAge","30m"] --- istioctl analyze on the DR (may reveal host mismatch warning) --- Warning [IST0174] (DestinationRule istio-vt-t68/gate-echo-dr) The host gate-echo.istio-vt-t68.svc.homelab.local defined in the DestinationRule does not match any services in the mesh. Info [IST0135] (Pod istio-vt-t68/client) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Info [IST0135] (Pod istio-vt-t68/gate-echo-v1-665cbff5b-9dfrd) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Info [IST0135] (Pod istio-vt-t68/gate-echo-v2-868545bd96-mfdcw) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. === FINDING: istiod runs with --domain cluster.local (confirmed via deploy args), independent of the real k8s cluster DNS domain (homelab.local, confirmed via /etc/resolv.conf earlier) === istioctl analyze flagged: IST0174 'host gate-echo.istio-vt-t68.svc.homelab.local defined in the DestinationRule does not match any services in the mesh.' => Correcting DR/VS host to match istiod's actual internal registry domain (svc.cluster.local) so the CDS/RDS translation this test targets can actually be exercised. === STEP 2b (corrected): re-apply DestinationRule with host matching istiod's real internal domain (cluster.local) === destinationrule.networking.istio.io/gate-echo-dr configured === check subset clusters AFTER corrected DR === [ "outbound|80||gate-echo.istio-vt-t68.svc.cluster.local", "outbound|80|v1|gate-echo.istio-vt-t68.svc.cluster.local", "outbound|80|v2|gate-echo.istio-vt-t68.svc.cluster.local" ] === STEP 3: 20x curl to gate-echo (DR with v1/v2 subsets exists, but NO VirtualService yet) -> expect BOTH v1-ok and v2-ok mixed === CMD: kubectl -n istio-vt-t68 exec client -c curl -- sh -c 'for i in $(seq 1 20); do curl -s http://gate-echo.istio-vt-t68.svc.homelab.local/; echo; done' | sort | uniq -c 20 20 v1-ok === DIAGNOSTIC: check EDS endpoints backing the base (non-subset) cluster - is v2 pod actually a member? === CMD: istioctl proxy-config endpoint client.istio-vt-t68 --cluster 'outbound|80||gate-echo.istio-vt-t68.svc.cluster.local' ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.255.126.48:5678 HEALTHY OK outbound|80||gate-echo.istio-vt-t68.svc.cluster.local 10.255.159.134:5678 HEALTHY OK outbound|80||gate-echo.istio-vt-t68.svc.cluster.local --- actual pod IPs for v1/v2 (for cross-check) --- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 4m16s 10.255.194.103 k8s-worker1 gate-echo-v1-665cbff5b-9dfrd 2/2 Running 0 4m16s 10.255.126.48 k8s-worker2 gate-echo-v2-868545bd96-mfdcw 2/2 Running 0 4m16s 10.255.159.134 k8s-master1 --- re-run 20x curl once more to rule out a transient EDS propagation lag --- 20 20 v1-ok === DIAGNOSTIC: hypothesis - sidecar->upstream HTTP keep-alive connection reuse is making 20 sequential curls stick to the SAME pod. Testing with Connection: close forcing fresh upstream connection each time === 20 20 v1-ok === DIAGNOSTIC: is v2 pod itself actually reachable at all (direct pod IP, bypassing Service load balancing)? === v2 pod IP: 10.255.159.134 v2-ok --- repeat direct pod-IP curl 5x to confirm consistent v2 reachability --- v2-ok v2-ok v2-ok v2-ok v2-ok === STEP 4: apply VirtualService routing explicitly to subset v1 (host corrected to cluster.local domain, matching DR) === virtualservice.networking.istio.io/gate-echo-vs created === STEP 5: 20x curl to gate-echo (VS now explicitly routes to subset v1) -> expect ONLY v1-ok === 20 20 v1-ok === DIAGNOSTIC (critical control): flip VS destination subset to v2 -- if VS/Host-header routing is genuinely in effect, output should flip to 100% v2-ok. If it stays v1-ok, the earlier 'pass' was coincidental (base cluster's own LB bias), not real VS enforcement. === virtualservice.networking.istio.io/gate-echo-vs patched apiVersion: networking.istio.io/v1 kind: VirtualService metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"networking.istio.io/v1","kind":"VirtualService","metadata":{"annotations":{},"name":"gate-echo-vs","namespace":"istio-vt-t68"},"spec":{"hosts":["gate-echo.istio-vt-t68.svc.cluster.local"],"http":[{"route":[{"destination":{"host":"gate-echo.istio-vt-t68.svc.cluster.local","subset":"v1"}}]}]}} creationTimestamp: "2026-07-05T00:25:33Z" generation: 2 name: gate-echo-vs namespace: istio-vt-t68 resourceVersion: "13670815" uid: 2310241a-3407-454a-aa9c-007c7e3a70a1 spec: hosts: - gate-echo.istio-vt-t68.svc.cluster.local http: - route: - destination: host: gate-echo.istio-vt-t68.svc.cluster.local subset: v2 --- 20x curl after flipping VS destination subset to v2 --- 20 20 v1-ok === DIAGNOSTIC: RDS route/virtual-host domains for port 80 gate-echo route -- confirms which Host-header strings actually route through the VS's subset rule vs the default passthrough === { "name": "gate-echo.istio-vt-t68.svc.cluster.local:80", "domains": [ "gate-echo.istio-vt-t68.svc.cluster.local", "gate-echo.istio-vt-t68.svc.cluster.local.", "gate-echo", "gate-echo.istio-vt-t68.svc", "gate-echo.istio-vt-t68", "10.250.24.43" ], "routeAction": { "cluster": "outbound|80|v2|gate-echo.istio-vt-t68.svc.cluster.local", "timeout": "0s", "retryPolicy": { "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes", "numRetries": 2, "retryHostPredicate": [ { "name": "envoy.retry_host_predicates.previous_hosts", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate" } } ], "hostSelectionRetryMaxAttempts": "5" }, "maxGrpcTimeout": "0s" } } === DIAGNOSTIC (confirmation): the RDS virtual host's domains list does NOT include the real homelab.local FQDN at all -- it only has cluster.local variants + bare short name + ClusterIP. curl with Host header 'gate-echo' (bare short name) SHOULD match and get routed per VS (currently subset:v2) if the mechanism truly works. === 10 10 v2-ok === CLEAN RE-CHECK: delete VS temporarily (DR with subsets still present, no VS) and retest with bare short-name host 'gate-echo' which DOES match Envoy's route table domains === virtualservice.networking.istio.io "gate-echo-vs" deleted from istio-vt-t68 namespace --- confirm route reverted to default (no VS) for the gate-echo virtual host --- { "name": "gate-echo.istio-vt-t68.svc.cluster.local:80", "domains": [ "gate-echo.istio-vt-t68.svc.cluster.local", "gate-echo.istio-vt-t68.svc.cluster.local.", "gate-echo", "gate-echo.istio-vt-t68.svc", "gate-echo.istio-vt-t68", "10.250.24.43" ], "routeAction": { "cluster": "outbound|80||gate-echo.istio-vt-t68.svc.cluster.local", "timeout": "0s", "retryPolicy": { "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes", "numRetries": 2, "retryHostPredicate": [ { "name": "envoy.retry_host_predicates.previous_hosts", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate" } } ], "hostSelectionRetryMaxAttempts": "5" }, "maxGrpcTimeout": "0s" } } --- 20x curl to bare short-name 'gate-echo' with DR (v1/v2 subsets defined) but NO VirtualService --- 20 10 v1-ok 10 v2-ok === FINAL STEP: re-apply VirtualService (subset:v1) and re-check with matching host 'gate-echo' -> expect ONLY v1-ok === virtualservice.networking.istio.io/gate-echo-vs created --- 20x curl to bare short-name 'gate-echo' with VS now routing to subset v1 --- 20 20 v1-ok === SUMMARY === 1) Before any DestinationRule: 0 subset ("|v1|") clusters existed in CDS (only the plain outbound|80||gate-echo...cluster.local cluster). => confirms C5: subset clusters do not exist until istiod translates a DestinationRule CR into xDS. 2) Environmental finding: istiod runs with --domain cluster.local (its own internal service- registry domain), which is DIFFERENT from this k8s cluster's real DNS domain (homelab.local, confirmed via /etc/resolv.conf). A DestinationRule/VirtualService written with the real DNS suffix (*.svc.homelab.local, as harness-notes generically recommend) is flagged by `istioctl analyze` as IST0174 "does not match any services in the mesh" and is silently a no-op (0 subset clusters even after apply; a VS patched between subset:v1/v2 produced NO change in observed traffic at all). Corrected DR/VS host to *.svc.cluster.local (matching istiod's actual --domain) to properly exercise the CDS/RDS translation being tested; verified via `istioctl proxy-config routes` that the resulting virtual host's domain-alias list includes the bare short name "gate-echo" (but NOT the homelab.local FQDN), so curl target was switched to the bare short name to get genuine host-matched routing. 3) After DR (v1/v2 subsets, correct host, no VS): CDS now shows outbound|80||gate-echo...cluster.local, outbound|80|v1|gate-echo...cluster.local, outbound|80|v2|gate-echo...cluster.local. 20x curl via matched host: 10x v1-ok / 10x v2-ok (mixed). => confirms C1: subset definitions alone do not restrict routing. 4) After VirtualService pinning subset:v1 (matched host): 20/20 v1-ok. Control test: patching the VS's destination.subset to v2 (while curling the earlier, non-matching homelab.local FQDN) still returned 100% v1-ok, PROVING that traffic was not actually going through the VS at all in that non-matching-host configuration -- it was hitting an unrelated default/ passthrough path. Once the host was corrected to match ("gate-echo"), flipping the VS to subset:v1 correctly and reproducibly pinned all traffic to v1-ok, confirming the VS->subset enforcement mechanism genuinely works when host matching is satisfied.