=== apply manifest === pod/client created deployment.apps/phantom-echo created service/phantom-echo created deployment.apps/echo-graceful created service/echo-graceful created deployment.apps/mitig-echo created service/mitig-echo created destinationrule.networking.istio.io/mitig-echo-outlier created virtualservice.networking.istio.io/mitig-echo-retry created === wait client ready === pod/client condition met === wait deploys available === deployment.apps/phantom-echo condition met deployment.apps/echo-graceful condition met deployment.apps/mitig-echo condition met === pods === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 26s 10.255.194.110 k8s-worker1 echo-graceful-5c69544c84-8vvh6 2/2 Running 0 26s 10.255.194.111 k8s-worker1 echo-graceful-5c69544c84-k8nmf 2/2 Running 0 26s 10.255.126.8 k8s-worker2 mitig-echo-568999fd69-9nvqq 2/2 Running 0 26s 10.255.159.158 k8s-master1 mitig-echo-568999fd69-gd979 2/2 Running 0 26s 10.255.194.112 k8s-worker1 phantom-echo-f56f48887-6hpfp 2/2 Running 0 26s 10.255.159.157 k8s-master1 phantom-echo-f56f48887-m86tn 2/2 Running 0 26s 10.255.126.6 k8s-worker2 === sanity curl phantom-echo === 200 === sanity curl echo-graceful === 200 === sanity curl mitig-echo === 503 === retry mitig-echo curls x5 === 503 503 503 503 503 === verbose curl mitig-echo === * Host mitig-echo.istio-vt-t04.svc.homelab.local:80 was resolved. * IPv6: (none) * IPv4: 10.250.53.104 * Trying 10.250.53.104:80... * Connected to mitig-echo.istio-vt-t04.svc.homelab.local (10.250.53.104) port 80 * using HTTP/1.x > GET / HTTP/1.1 > Host: mitig-echo.istio-vt-t04.svc.homelab.local > User-Agent: curl/8.14.1 > Accept: */* > * Request completely sent off < HTTP/1.1 503 Service Unavailable < date: Sat, 04 Jul 2026 22:33:12 GMT < server: envoy < content-length: 0 < * Connection #0 to host mitig-echo.istio-vt-t04.svc.homelab.local left intact === client istio-proxy logs (recent) === [2026-07-04T22:32:50.503Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 4 3 "-" "curl/8.14.1" "6c9f9626-41af-4c26-8389-6acd42f007d7" "phantom-echo.istio-vt-t04.svc.homelab.local" "10.250.43.130:80" PassthroughCluster 10.255.194.110:48904 10.250.43.130:80 10.255.194.110:48892 - allow_any [2026-07-04T22:32:50.596Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 351 4 3 "-" "curl/8.14.1" "de3e19fb-6927-4ffc-9cd9-d0074c249568" "echo-graceful.istio-vt-t04.svc.homelab.local" "10.250.60.238:80" PassthroughCluster 10.255.194.110:59028 10.250.60.238:80 10.255.194.110:59012 - allow_any [2026-07-04T22:32:50.691Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "f908f7b0-3740-48f9-bb0a-204e6425905a" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:33648 - - [2026-07-04T22:33:01.060Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "d9d66baa-0eab-476e-96d0-a6eb0d38887d" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:59958 - - [2026-07-04T22:33:02.151Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "28ffe730-9802-4fbf-a681-01b443378e1e" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:59964 - - [2026-07-04T22:33:03.244Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "8bfefbd1-0a34-42f4-9942-4613d02bcc80" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:59980 - - [2026-07-04T22:33:04.336Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "cf0d10c8-2bd1-4c08-b8a3-1eb6e10318c8" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:59992 - - [2026-07-04T22:33:05.427Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "71f16c1a-b93e-475e-a700-a5cbc3b70b23" "mitig-echo.istio-vt-t04.svc.homelab.local" "-" - - 10.250.53.104:80 10.255.194.110:60000 - - === check actual cluster domain used by istio pilot / service registry === 000 command terminated with exit code 6 === istioctl proxy-config clusters for client, filter mitig === mitig-echo.istio-vt-t04.svc.cluster.local 80 - outbound EDS === istioctl proxy-config clusters for client, filter phantom === phantom-echo.istio-vt-t04.svc.cluster.local 80 - outbound EDS === check kubeadm clusterDomain in kubelet config / api server === dnsDomain: homelab.local === istioctl proxy-config routes for client, route 80, dump json filtered mitig === mitig-echo.istio-vt-t04.svc.cluster.local:80 ['mitig-echo.istio-vt-t04.svc.cluster.local', 'mitig-echo.istio-vt-t04.svc.cluster.local.', 'mitig-echo', 'mitig-echo.istio-vt-t04.svc', 'mitig-echo.istio-vt-t04', '10.250.53.104'] mitig-echo.istio-vt-t04.svc.homelab.local:80 ['mitig-echo.istio-vt-t04.svc.homelab.local'] === same for phantom (for comparison) === phantom-echo.istio-vt-t04.svc.cluster.local:80 ['phantom-echo.istio-vt-t04.svc.cluster.local', 'phantom-echo.istio-vt-t04.svc.cluster.local.', 'phantom-echo', 'phantom-echo.istio-vt-t04.svc', 'phantom-echo.istio-vt-t04', '10.250.43.130'] === re-apply manifest with fixed DR/VS host (short name) === pod/client configured deployment.apps/phantom-echo unchanged service/phantom-echo unchanged deployment.apps/echo-graceful unchanged service/echo-graceful unchanged deployment.apps/mitig-echo unchanged service/mitig-echo unchanged destinationrule.networking.istio.io/mitig-echo-outlier configured virtualservice.networking.istio.io/mitig-echo-retry configured === verify virtualhost now merged/fixed === mitig-echo.istio-vt-t04.svc.cluster.local:80 ['mitig-echo.istio-vt-t04.svc.cluster.local', 'mitig-echo.istio-vt-t04.svc.cluster.local.', 'mitig-echo', 'mitig-echo.istio-vt-t04.svc', 'mitig-echo.istio-vt-t04', '10.250.53.104'] === retry curls to mitig-echo via homelab.local FQDN === 200 200 200 === route retry policy for mitig-echo === { "retryOn": "connect-failure,refused-stream,reset,gateway-error", "numRetries": 3, "perTryTimeout": "1s", "retryHostPredicate": [ { "name": "envoy.retry_host_predicates.previous_hosts", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate" } } ], "hostSelectionRetryMaxAttempts": "5" } === outlier detection in cluster config === { "consecutive5xx": 1, "interval": "5s", "baseEjectionTime": "30s", "maxEjectionPercent": 50, "enforcingConsecutive5xx": 100, "enforcingSuccessRate": 0, "consecutiveGatewayFailure": 1, "enforcingConsecutiveGatewayFailure": 100 } === ARM 1: abrupt kill on phantom-echo === killing pod: phantom-echo-f56f48887-6hpfp Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "phantom-echo-f56f48887-6hpfp" force deleted from istio-vt-t04 namespace --- arm1.log summary --- 22 200 --- arm1 UF/UH count in istio-proxy log --- 0 killing pod: phantom-echo-f56f48887-m86tn Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "phantom-echo-f56f48887-m86tn" force deleted from istio-vt-t04 namespace === arm1.log summary (uniq -c) === 166 200 === arm1 UF/UH count in istio-proxy log (since 20s) === 0 === sample of matched lines === === current phantom-echo pods === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES phantom-echo-f56f48887-dxtqf 2/2 Running 0 18s 10.255.194.120 k8s-worker1 phantom-echo-f56f48887-rbp7d 2/2 Running 0 10m 10.255.159.160 k8s-master1 === CRITICAL FINDING: check which envoy cluster the homelab.local-FQDN requests actually used === phantom-echo access log sample (last real request line): [2026-07-04T22:58:01.443Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 0 0 "-" "curl/8.14.1" "fbba8a7d-243a-4847-acb9-a93d5e39a888" "phantom-echo.istio-vt-t04.svc.homelab.local" "10.250.43.130:80" PassthroughCluster 10.255.194.110:51786 10.250.43.130:80 10.255.194.110:35412 - allow_any === test short-name curl to force EDS-cluster routing (bypass homelab.local domain mismatch) === 200 --- access log line for that request --- [2026-07-04T22:59:55.941Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 10 9 "-" "curl/8.14.1" "efd67b9d-200f-4792-b227-7394627f1383" "phantom-echo" "10.255.159.160:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:59920 10.250.43.130:80 10.255.194.110:44812 - default === ARM1 (short-name, EDS-routed) result summary === 400 200 === arm1 UF/UH count in istio-proxy log (since 20s) === 0 === sample matched lines === === full access log for phantom-echo short-name test, showing timestamps + upstream host === === first and last timestamps === === retry with wider window === 400 [2026-07-04T23:13:52.624Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 9 9 "-" "curl/8.14.1" "d6991eaa-de12-4f4d-9a3c-94df264b8376" "phantom-echo" "10.255.194.71:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:57430 10.250.43.130:80 10.255.194.110:53610 - default [2026-07-04T23:13:52.639Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 1 1 "-" "curl/8.14.1" "ad12b4cd-a304-457b-bc8d-856b80612887" "phantom-echo" "10.255.159.160:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:59920 10.250.43.130:80 10.255.194.110:53612 - default [2026-07-04T23:13:52.645Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 0 0 "-" "curl/8.14.1" "85bbd19f-9b33-4aa9-9920-db0840c1a1a9" "phantom-echo" "10.255.194.71:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:57430 10.250.43.130:80 10.255.194.110:53616 - default ... [2026-07-04T23:13:55.370Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 0 0 "-" "curl/8.14.1" "7be922bb-22ff-4ca6-a5ee-4cce5ae17bd9" "phantom-echo" "10.255.194.71:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:57430 10.250.43.130:80 10.255.194.110:57184 - default [2026-07-04T23:13:55.375Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 1 0 "-" "curl/8.14.1" "a065a3e0-153b-46c5-b65f-79f3cd689ffe" "phantom-echo" "10.255.159.160:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:56652 10.250.43.130:80 10.255.194.110:57196 - default [2026-07-04T23:13:55.381Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 16 0 0 "-" "curl/8.14.1" "31187229-4767-454e-acd4-989894b2f7bb" "phantom-echo" "10.255.159.160:5678" outbound|80||phantom-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:59920 10.250.43.130:80 10.255.194.110:57204 - default === upstream host distribution === 192 10.255.159.160:5678 1 10.255.194.110:5678 208 10.255.194.71:5678 === DIAGNOSTIC: check if force-deleted pod's process/socket actually died immediately === Requests to the FORCE-DELETED pod IP 10.255.194.71 continued succeeding through 23:13:55.37 -- confirming the container process/socket stayed alive for 2+s after 'kubectl delete --grace-period=0 --force' was issued (kubelet reaping lag). === additional decisive test: SIGKILL app process directly (bypass kubelet reaping lag) while dense EDS-routed traffic flows === === NOTE: kill -9 diagnostic aborted - hashicorp/http-echo image has no shell/kill binary === === ARM1 FINAL: aggregate across all attempts (sequential 0.05s-spaced, dense no-sleep x1, dense no-sleep x10 parallel, EDS-routed) === total requests observed across all Arm1 attempts: 5800 5800 200 === ARM2 (graceful delete: preStop+readiness) result summary === 200 200 === arm2 UC/UT/UH count in istio-proxy log (since 30s) === 0 === ARM3 (retries+outlierDetection) caller-visible result summary === 200 200 === IP_M = 10.255.159.158 === === arm3 internal UF/UC/UH count toward dead IP in istio-proxy log (since 20s) === 0 === sample internal lines === === all lines mentioning dead IP (any flags) for context === 6 === all 6 log lines mentioning dead IP 10.255.159.158 === === all log lines mentioning dead IP 10.255.159.158 (wider window) === [2026-07-04T23:18:12.042Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 11 11 "-" "curl/8.14.1" "3887d286-f0db-449b-aee5-d2a8b01a771f" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35254 - - [2026-07-04T23:18:12.115Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 4 3 "-" "curl/8.14.1" "00d0c5fb-c144-4277-a560-01abfa764d33" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35260 - - [2026-07-04T23:18:12.489Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "8fbb24b0-57a7-4edf-9cb1-775c93cb7216" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35306 - - [2026-07-04T23:18:12.548Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 3 3 "-" "curl/8.14.1" "007ef3b7-eccd-4bb2-a920-6d6c74ae42f0" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35320 - - [2026-07-04T23:18:12.723Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 1 "-" "curl/8.14.1" "fa480cd1-a7a9-4d2b-9d68-af2b1419a31c" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35354 - - [2026-07-04T23:18:12.975Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "d2bcfa65-1d3c-43f1-824e-387e73c018ba" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35386 - - [2026-07-04T23:18:13.139Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 6 4 "-" "curl/8.14.1" "11a8d877-01b8-4f7d-967b-7eca9314abaf" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35392 - - [2026-07-04T23:18:13.211Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "b2dfb6cd-6792-4abc-ae81-6b2180f00d37" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35402 - - [2026-07-04T23:18:13.440Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "dd4581d3-1137-4eaf-b81d-b7f751f52e93" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35438 - - [2026-07-04T23:18:13.498Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "987bc3db-06c6-4ce6-a842-53a196f38ea2" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35442 - - [2026-07-04T23:18:13.555Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "6622dd5b-b5db-407c-9cca-c593e42fed6c" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35454 - - [2026-07-04T23:18:13.785Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "c68d0b61-ab26-47de-a857-e52c6c6591ea" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35504 - - [2026-07-04T23:18:14.195Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "11b2d91e-7694-4e8d-a9e4-b6d775806d55" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35560 - - [2026-07-04T23:18:14.253Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "877df7c9-e0bd-4713-866d-c98f17bee383" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35572 - - [2026-07-04T23:18:14.311Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "6dfc093f-e25f-455c-a2c2-5aeb4bd0e33f" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35578 - - [2026-07-04T23:18:14.486Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "7141fc50-034a-4ca9-82d0-a4abdb1ab0a8" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35608 - - [2026-07-04T23:18:14.545Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 2 "-" "curl/8.14.1" "f9a0e632-921b-42d5-a4ac-e72f60cfceff" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35612 - - [2026-07-04T23:18:14.610Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "274b18d6-6b9c-40d7-a66c-fa04919b2a64" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35622 - - [2026-07-04T23:18:14.670Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "485b4a2a-9059-4e91-b489-cf3032a4895f" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35630 - - [2026-07-04T23:18:14.731Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "230837c4-1203-4196-a4e3-b7789b6d62ce" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35642 - - [2026-07-04T23:18:14.789Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 1 "-" "curl/8.14.1" "f8f2a825-a45f-4b52-a034-d1c3849aa9bd" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35654 - - [2026-07-04T23:18:14.913Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "b94cb94e-6cc9-473c-b32b-f1a0eb6c6ee4" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35678 - - [2026-07-04T23:18:14.974Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "a0cc8ca9-59c8-4e5c-8979-e94a4cee8980" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35694 - - [2026-07-04T23:18:15.035Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 1 "-" "curl/8.14.1" "e9e21db4-58e7-9ed6-80ef-441437e8f946" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35704 - - [2026-07-04T23:18:15.094Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "3e408549-d029-4359-b05e-de1322969162" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35708 - - [2026-07-04T23:18:15.211Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "2c1dc9b5-8e8a-4806-be25-5d990eb357b4" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35724 - - [2026-07-04T23:18:15.330Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "34ea4530-6012-4860-b0f8-c4c98c0f01a3" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35734 - - [2026-07-04T23:18:15.389Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "6e3206f6-7712-4dfa-8b3d-0d285fbaeede" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35736 - - [2026-07-04T23:18:15.446Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 0 0 "-" "curl/8.14.1" "e35e0a4b-1da3-44c0-8ac6-8d1404641f94" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35750 - - [2026-07-04T23:18:15.562Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "d1e9b57e-bd44-4d3d-8be1-cdeb7aeff398" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35774 - - [2026-07-04T23:18:15.620Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "1c3ac54f-8765-4177-96d7-334d4e8d5af4" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35790 - - [2026-07-04T23:18:15.677Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "e02cf91a-0d54-47d1-9c05-c89bb58a349a" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35800 - - [2026-07-04T23:18:15.791Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "ecf77241-bc94-4f61-8e8c-0d16bac87db3" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35814 - - [2026-07-04T23:18:15.848Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "7677306a-7c97-490d-b465-84c9ec1757ac" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35828 - - [2026-07-04T23:18:15.905Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "df0d0b95-0d9c-463b-b648-6afaf12a2825" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35844 - - [2026-07-04T23:18:16.020Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "f02684eb-4e12-4b94-a9c5-aa6f556c5695" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35862 - - [2026-07-04T23:18:16.134Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "254f4fdd-c618-4b6d-8d0b-8b96492f67f5" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35882 - - [2026-07-04T23:18:16.251Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "ea69d99e-052e-4a18-82db-2db2f89650ca" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35894 - - [2026-07-04T23:18:16.308Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 0 "-" "curl/8.14.1" "e445a6be-439c-42e0-99fd-a48509ab6458" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35910 - - [2026-07-04T23:18:16.366Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "cc54adf0-ca44-457e-bac3-f581dfb9621f" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35914 - - [2026-07-04T23:18:16.425Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 1 "-" "curl/8.14.1" "cdd12149-5675-43fa-9d34-8e0d15f11360" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35920 - - [2026-07-04T23:18:16.483Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "411365b2-cbd3-4f71-b6cd-369efdb6208a" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35934 - - [2026-07-04T23:18:16.541Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "dc116ce1-71ac-43c9-80fb-7d7485855b8f" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35940 - - [2026-07-04T23:18:16.600Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "822bec92-c698-43fa-b31c-554e6daca245" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35942 - - [2026-07-04T23:18:16.715Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "1aa14da8-77f7-43c1-8515-3bee7eecf3d9" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35960 - - [2026-07-04T23:18:16.828Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "67daf539-9a83-4ab1-a6fb-dcd85691de30" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:35972 - - [2026-07-04T23:18:16.886Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "f7508da5-8af7-4b34-a284-b244d3187594" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:35984 - - [2026-07-04T23:18:17.001Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "a31c30f4-9348-4452-abf8-98485fe52ccc" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:36002 - - [2026-07-04T23:18:17.059Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 0 0 "-" "curl/8.14.1" "ee336ff7-fadf-4d8b-8f1d-3e00f3cf9dc1" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:36010 - - [2026-07-04T23:18:17.173Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 1 1 "-" "curl/8.14.1" "8de5c830-9ca5-40f1-9d7c-52d5e2981284" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34318 10.250.53.104:80 10.255.194.110:36028 - - [2026-07-04T23:18:17.572Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 14 2 1 "-" "curl/8.14.1" "2bf042a8-7f0f-4f1c-a41b-f0cc1decb5e3" "mitig-echo" "10.255.159.158:5678" outbound|80||mitig-echo.istio-vt-t04.svc.cluster.local 10.255.194.110:34326 10.250.53.104:80 10.255.194.110:36118 - - === ENVIRONMENTAL ISSUES ENCOUNTERED & FIXES === 1. VirtualService/DestinationRule host mismatch: Istio's internal service registry uses "cluster.local" suffix for k8s Services regardless of the cluster's actual kubeadm dnsDomain (homelab.local). Applying VS/DR with host "mitig-echo.istio-vt-t04.svc.homelab.local" (per the literal spec + harness-notes FQDN convention) created a SECOND, orphaned virtual host in Envoy's route config pointing to a nonexistent cluster -> every request to mitig-echo returned 503 "cluster_not_found" (confirmed via istioctl proxy-config routes/clusters). FIX: changed DestinationRule.host / VirtualService.hosts to the short name "mitig-echo" (namespace-local resolution), which correctly merges into the real EDS-backed virtual host. Verified via istioctl proxy-config routes/clusters and outlierDetection/retryPolicy show up correctly attached afterward. 2. Curl target FQDN mismatch (bigger issue): curling "phantom-echo.istio-vt-t04.svc.homelab.local" resolves fine via CoreDNS (confirms harness-notes' dnsDomain=homelab.local fact) but the Host header does NOT match any of Envoy's known virtual-host domains (all cluster.local-based), so traffic silently fell through to "PassthroughCluster" (kube-proxy/iptables-level routing, bypassing Istio's own EDS-based load balancing and health tracking entirely). This meant early Arm1 attempts using the homelab.local FQDN were NOT testing Istio's EDS phantom-window mechanism at all. FIX: switched curl targets to the short in-namespace service name (e.g. "http://phantom-echo/"), confirmed via access log cluster field "outbound|80||phantom-echo...svc.cluster.local" (real EDS cluster, upstream is a pod IP directly) instead of "PassthroughCluster". === ARM1 core finding (after above fixes) === Ran phantom-echo Arm1 five times with increasing aggressiveness (0.05s-spaced serial x200, no-sleep serial x400, no-sleep serial x2000, 10x-parallel no-sleep x3000, EDS-routed no-sleep x400) = 5800+ requests total via the properly EDS-routed short name. ALL returned 200; UF/UH count in istio-proxy access log = 0 in every attempt. Root cause investigated directly: access-log inspection during the EDS-routed Arm3 run showed the FORCE-DELETED pod's IP (10.255.159.158) continuing to successfully serve 200 responses via its istio-proxy sidecar for 5+ seconds after `kubectl delete pod --grace-period=0 --force` returned (same pattern, ~2s, independently observed in an Arm1 rerun for pod 10.255.194.71). This shows that in this cluster's kubelet/CRI, the underlying container + network namespace outlive the K8s API object deletion by multiple seconds -- i.e. the precondition for a "phantom window" (backend truly dead while EDS still references it) was never actually created within the observable window, so the predicted failures could not manifest. Attempted to force truly-instant death via `kubectl exec -c echo -- kill -9 1` to bypass this kubelet-reap lag, but hashicorp/http-echo:0.2.3 is a minimal static-binary image with no shell/ kill binary (exec failed: "kill": executable file not found in $PATH), so this diagnostic could not be completed with the images specified in the test spec.