=== $ kubectl apply -f manifest.yaml && wait client Ready && wait deploy available === pod/client created deployment.apps/adsorder-echo-v1 created deployment.apps/adsorder-echo-v2 created service/adsorder-echo created destinationrule.networking.istio.io/adsorder-echo-dr created virtualservice.networking.istio.io/adsorder-echo-vs created --- wait pod/client Ready --- pod/client condition met --- wait deploy adsorder-echo-v1 v2 available --- deployment.apps/adsorder-echo-v1 condition met deployment.apps/adsorder-echo-v2 condition met === $ kubectl -n istio-vt-t25-r2 get pods -o wide === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES adsorder-echo-v1-8cf54dc8c-cpsz8 2/2 Running 0 10s 10.255.194.68 k8s-worker1 adsorder-echo-v2-7749f6c94c-ckd7n 2/2 Running 0 10s 10.255.126.7 k8s-worker2 client 2/2 Running 0 10s 10.255.159.137 k8s-master1 === VERIFY (per re-run guidance): istioctl proxy-config cluster client.istio-vt-t25-r2 --fqdn adsorder-echo.istio-vt-t25-r2.svc.cluster.local (confirm subset v1 attached BEFORE transition) === SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 - outbound EDS adsorder-echo-dr.istio-vt-t25-r2 adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 v1 outbound EDS adsorder-echo-dr.istio-vt-t25-r2 === SANITY (per corrected rule): curl SHORT name http://adsorder-echo.istio-vt-t25-r2/ (not homelab.local FQDN) BEFORE starting stress loop === * Host adsorder-echo.istio-vt-t25-r2:80 was resolved. * IPv6: (none) * IPv4: 10.250.133.84 * Trying 10.250.133.84:80... * Connected to adsorder-echo.istio-vt-t25-r2 (10.250.133.84) port 80 * using HTTP/1.x > GET / HTTP/1.1 > Host: adsorder-echo.istio-vt-t25-r2 > User-Agent: curl/8.14.1 > Accept: */* > * Request completely sent off < HTTP/1.1 200 OK < x-app-name: http-echo < x-app-version: 0.2.3 < date: Sun, 05 Jul 2026 01:05:10 GMT < content-length: 6 < content-type: text/plain; charset=utf-8 < x-envoy-upstream-service-time: 9 < server: envoy < v1-ok { [6 bytes data] * Connection #0 to host adsorder-echo.istio-vt-t25-r2 left intact === $ sleep 5 === done === $ (background) kubectl -n istio-vt-t25-r2 exec client -c curl -- sh -c 'for i in $(seq 1 400); do curl -s -o /dev/null -w "%{http_code} " http://adsorder-echo.istio-vt-t25-r2/; done' > /tmp/ads_stress_t25r2.log & (SHORT name per corrected rule) === (started as background pid 1321117) === $ kubectl apply -f - (switch DR+VS to v2 simultaneously; host=svc.cluster.local per corrected rule) === apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: adsorder-echo-dr namespace: istio-vt-t25-r2 spec: host: adsorder-echo.istio-vt-t25-r2.svc.cluster.local subsets: - name: v2 labels: version: v2 --- apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: adsorder-echo-vs namespace: istio-vt-t25-r2 spec: hosts: - adsorder-echo.istio-vt-t25-r2.svc.cluster.local http: - route: - destination: host: adsorder-echo.istio-vt-t25-r2.svc.cluster.local subset: v2 --- switch apply timestamp: 2026-07-05T01:05:39.947Z --- destinationrule.networking.istio.io/adsorder-echo-dr configured virtualservice.networking.istio.io/adsorder-echo-vs configured --- switch apply returned: 2026-07-05T01:05:40.110Z --- === $ sleep 10 === done background loop finished === $ sort /tmp/ads_stress_t25r2.log | uniq -c === 400 200 === $ kubectl -n istio-vt-t25-r2 logs client -c istio-proxy --since=2m | grep '"NC"' | wc -l (spec command as-written) === 0 --- spec grep pattern likely does not match (flags unquoted in this log format, confirmed in run 1); retrying with unquoted NC token match --- 0 === $ kubectl -n istio-vt-t25-r2 logs client -c istio-proxy --since=5m | grep -c 'GET / HTTP' (total requests logged) === 401 === $ kubectl -n istio-vt-t25-r2 logs client -c istio-proxy --since=5m | grep -c ' NC cluster_not_found' (NC-flagged requests) === 0 === response code distribution across full access log window (all response codes seen in access log) === 401 - === istio-proxy access log tail (client), sample lines around switch timestamp 2026-07-05T01:05:39-40Z === 401 /tmp/t25r2_access.log --- first 3 lines --- [2026-07-05T01:05:10.805Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 10 9 "-" "curl/8.14.1" "5056fe14-6e59-4d20-a0b7-c9bf1e944196" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:56880 10.250.133.84:80 10.255.159.137:51044 - - [2026-07-05T01:05:28.313Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 2 2 "-" "curl/8.14.1" "c54d22d2-ac9a-4160-8f50-110affc59447" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:50612 10.250.133.84:80 10.255.159.137:35582 - - [2026-07-05T01:05:28.324Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "e44be328-ff06-4fc1-9642-29a8a85cabca" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:50612 10.250.133.84:80 10.255.159.137:35590 - - --- last 3 lines --- [2026-07-05T01:05:30.568Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "80075a5b-f398-4957-9f68-6cd0d45034e1" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:50612 10.250.133.84:80 10.255.159.137:39088 - - [2026-07-05T01:05:30.573Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "b946dc50-1b5c-49d9-baf9-efe694913f86" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:56880 10.250.133.84:80 10.255.159.137:39104 - - [2026-07-05T01:05:30.579Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "6de16efd-306b-478c-b399-709756742a1e" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:50612 10.250.133.84:80 10.255.159.137:39116 - - --- lines around switch window (01:05:39 - 01:05:41) --- === METHODOLOGY NOTE: gap analysis found the initial attempt's switch-apply landed AFTER the 400-req loop had already finished (loop last entry 01:05:30.579Z vs switch apply 01:05:39.947Z) due to inter-tool-call latency between separate Bash invocations. This means the transition window was NOT actually exercised by load in that attempt. Redoing with loop-start + switch-apply combined in a SINGLE shell invocation (background job + immediate follow-on apply, no round trip gap) to guarantee overlap. === === $ kubectl apply -f - (revert DR+VS to v1 baseline before redo) === destinationrule.networking.istio.io/adsorder-echo-dr configured virtualservice.networking.istio.io/adsorder-echo-vs configured === verify subset v1 re-attached === SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 - outbound EDS adsorder-echo-dr.istio-vt-t25-r2 adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 v1 outbound EDS adsorder-echo-dr.istio-vt-t25-r2 === sanity curl short-name, expect v1-ok === v1-ok === FINAL MEASUREMENT: background 400-req loop + immediate switch apply, combined in one shell invocation (no inter-tool-call gap) === start: 2026-07-05T01:07:36.963Z loop started pid=1322311 at 2026-07-05T01:07:36.964Z destinationrule.networking.istio.io/adsorder-echo-dr configured virtualservice.networking.istio.io/adsorder-echo-vs configured switch applied at 2026-07-05T01:07:37.137Z loop finished at 2026-07-05T01:07:39.487Z 400 /tmp/ads_stress_t25r2_final.log === $ sort /tmp/ads_stress_t25r2_final.log | uniq -c === 397 200 3 503 === $ kubectl -n istio-vt-t25-r2 logs client -c istio-proxy --since=2m | grep 'GET / HTTP' > access.log; grep -c ' NC ' / grep -c '"NC"' === 401 /tmp/t25r2_access_final.log --- spec pattern '"NC"' (quoted) count --- 0 --- corrected pattern (unquoted NC token, response flags field) count --- 3 --- lines with 503 --- [2026-07-05T01:07:37.262Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "76ae6f66-16a0-4dfe-a95a-cde21f35673c" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45162 - - [2026-07-05T01:07:37.275Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "cfb4de45-3d89-48e4-973d-822c61b7fcae" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45170 - - [2026-07-05T01:07:37.285Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "2b85096a-d922-490f-9cab-c417ef5d0fea" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45182 - - === context around the 503/NC window (grep -B5 -A5 for first 503 line) === [2026-07-05T01:07:37.201Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "4d669281-de00-4078-a768-48ca0dc71b6f" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:47850 10.250.133.84:80 10.255.159.137:45114 - - [2026-07-05T01:07:37.207Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "4317fbeb-56f9-47aa-b461-b179f3527506" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:47850 10.250.133.84:80 10.255.159.137:45124 - - [2026-07-05T01:07:37.212Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "fe7bebe2-ef02-4c8a-95ca-a47d18bbba03" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:45748 10.250.133.84:80 10.255.159.137:45132 - - [2026-07-05T01:07:37.221Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 1 0 "-" "curl/8.14.1" "ba8b51e5-c160-405b-acff-ed47cfc810b8" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:47850 10.250.133.84:80 10.255.159.137:45148 - - [2026-07-05T01:07:37.234Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 19 19 "-" "curl/8.14.1" "c9d49e53-fbeb-433c-82f7-953ab7282074" "adsorder-echo.istio-vt-t25-r2" "10.255.194.68:5678" outbound|80|v1|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:45748 10.250.133.84:80 10.255.159.137:45156 - - [2026-07-05T01:07:37.262Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "76ae6f66-16a0-4dfe-a95a-cde21f35673c" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45162 - - [2026-07-05T01:07:37.275Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "cfb4de45-3d89-48e4-973d-822c61b7fcae" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45170 - - [2026-07-05T01:07:37.285Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "2b85096a-d922-490f-9cab-c417ef5d0fea" "adsorder-echo.istio-vt-t25-r2" "-" - - 10.250.133.84:80 10.255.159.137:45182 - - [2026-07-05T01:07:37.295Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 12 11 "-" "curl/8.14.1" "4920caac-f1e8-4a28-a0ea-fa4de432b3b0" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39242 10.250.133.84:80 10.255.159.137:45190 - - [2026-07-05T01:07:37.317Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 4 4 "-" "curl/8.14.1" "548f216f-36a7-459a-af72-693de4b6b2c4" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:45192 - - [2026-07-05T01:07:37.330Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "dc0ac541-25f7-4552-8726-c7a679b6c5c8" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:45198 - - [2026-07-05T01:07:37.337Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "d1747967-01d8-4ae8-aa99-5df7ba19ec24" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:45214 - - [2026-07-05T01:07:37.343Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "07ad0dbb-8698-4076-85a5-c90b534b281f" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39242 10.250.133.84:80 10.255.159.137:45230 - - [2026-07-05T01:07:37.349Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "98ed59c6-e83d-4ad5-b080-806472218cef" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:45238 - - === last 3 lines of access log (confirm final state routed to v2) === [2026-07-05T01:07:39.468Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "633184e4-bdf8-444e-a585-007dc9ec3e00" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39242 10.250.133.84:80 10.255.159.137:33308 - - [2026-07-05T01:07:39.474Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "8229e4f6-a05c-4e3b-85bb-b7880886fc06" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:33318 - - [2026-07-05T01:07:39.479Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 0 0 "-" "curl/8.14.1" "7d5ab5dd-b47d-4856-88da-12b40ff798dc" "adsorder-echo.istio-vt-t25-r2" "10.255.126.7:5678" outbound|80|v2|adsorder-echo.istio-vt-t25-r2.svc.cluster.local 10.255.159.137:39246 10.250.133.84:80 10.255.159.137:33324 - - === count of upstream cluster used per line (v1 vs v2 subset), full window === 29 outbound|80|v1| 369 outbound|80|v2| === $ kubectl -n istio-vt-t25-r2 get pods,deploy,svc,vs,dr -o wide (final state before cleanup) === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/adsorder-echo-v1-8cf54dc8c-cpsz8 2/2 Running 0 5m22s 10.255.194.68 k8s-worker1 pod/adsorder-echo-v2-7749f6c94c-ckd7n 2/2 Running 0 5m22s 10.255.126.7 k8s-worker2 pod/client 2/2 Running 0 5m22s 10.255.159.137 k8s-master1 NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/adsorder-echo-v1 1/1 1 1 5m22s echo hashicorp/http-echo:0.2.3 app=adsorder-echo,version=v1 deployment.apps/adsorder-echo-v2 1/1 1 1 5m22s echo hashicorp/http-echo:0.2.3 app=adsorder-echo,version=v2 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/adsorder-echo ClusterIP 10.250.133.84 80/TCP 5m22s app=adsorder-echo NAME GATEWAYS HOSTS AGE virtualservice.networking.istio.io/adsorder-echo-vs ["adsorder-echo.istio-vt-t25-r2.svc.cluster.local"] 5m22s NAME HOST AGE destinationrule.networking.istio.io/adsorder-echo-dr adsorder-echo.istio-vt-t25-r2.svc.cluster.local 5m22s === $ istioctl proxy-config cluster client.istio-vt-t25-r2 --fqdn adsorder-echo.istio-vt-t25-r2.svc.cluster.local (final state, confirms v2 subset attached post-switch) === SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 - outbound EDS adsorder-echo-dr.istio-vt-t25-r2 adsorder-echo.istio-vt-t25-r2.svc.cluster.local 80 v2 outbound EDS adsorder-echo-dr.istio-vt-t25-r2 === SUMMARY: final measurement request/response distribution === curl loop http codes: 397x200, 3x503 (out of 400) access log upstream cluster usage: 29x v1 (pre-switch), 3x NC cluster_not_found (transition window, 01:07:37.262-.285Z), 369x v2 (post-switch) switch apply completed (API accepted) at 01:07:37.137Z; first NC at 01:07:37.262Z (+125ms); last NC at 01:07:37.285Z; first clean v2 200 at 01:07:37.295Z (+10ms after last NC) -> NC window ~= 23-33ms, self-healed === CLEANUP: $ kubectl delete namespace istio-vt-t25-r2 --wait=false --ignore-not-found === namespace "istio-vt-t25-r2" deleted