===== Command 1: apply flaky-fixtures.yaml (already applied as manifest.yaml above; re-recording wait) ===== $ kubectl apply -f manifest.yaml && kubectl -n istio-vt-t05 wait --for=condition=available deploy/flaky-echo --timeout=60s pod/client configured deployment.apps/flaky-echo unchanged service/flaky-echo unchanged destinationrule.networking.istio.io/flaky-echo-dr-aggressive unchanged virtualservice.networking.istio.io/flaky-echo-vs-fault unchanged deployment.apps/flaky-echo condition met ===== Command 2: 100 requests to flaky-echo, tally status codes ===== $ kubectl -n istio-vt-t05 exec client -c curl -- sh -c 'for i in $(seq 1 100); do curl -s -o /dev/null -w "%{http_code} " http://flaky-echo.istio-vt-t05.svc.homelab.local/; done' | sort | uniq -c 100 503 ===== Command 3: check outlier ejection state via istio-proxy admin /clusters ===== $ kubectl -n istio-vt-t05 exec client -c istio-proxy -- curl -s localhost:15000/clusters | grep flaky-echo | grep -E "outlier_check::ejections_active|health_flags" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::health_flags::healthy outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::health_flags::healthy ===== Command 3b (diagnostic): full /clusters output for flaky-echo (no grep filter) ===== outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::observability_name::outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local; outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::default_priority::max_connections::4294967295 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::default_priority::max_pending_requests::4294967295 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::default_priority::max_requests::4294967295 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::default_priority::max_retries::4294967295 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::high_priority::max_connections::1024 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::high_priority::max_pending_requests::1024 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::high_priority::max_requests::1024 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::high_priority::max_retries::3 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::added_via_api::true outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::eds_service_name::outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::cx_active::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::cx_connect_fail::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::cx_total::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_active::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_error::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_success::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_timeout::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_total::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::hostname:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::health_flags::healthy outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::weight::1 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::region:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::zone:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::sub_zone:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::canary::false outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::priority::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::success_rate::-1 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::local_origin_success_rate::-1 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::cx_active::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::cx_connect_fail::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::cx_total::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_active::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_error::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_success::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_timeout::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_total::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::hostname:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::health_flags::healthy outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::weight::1 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::region:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::zone:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::sub_zone:: outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::canary::false outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::priority::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::success_rate::-1 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::local_origin_success_rate::-1 ===== Diagnostic: check the ACTUAL cluster used for homelab.local-hostname route (found via config_dump) ===== $ kubectl -n istio-vt-t05 exec client -c istio-proxy -- curl -s localhost:15000/clusters | grep "svc.homelab.local" outbound|80||mock.istio-verify-ext.svc.homelab.local::observability_name::outbound|80||mock.istio-verify-ext.svc.homelab.local; outbound|80||mock.istio-verify-ext.svc.homelab.local::default_priority::max_connections::4294967295 outbound|80||mock.istio-verify-ext.svc.homelab.local::default_priority::max_pending_requests::4294967295 outbound|80||mock.istio-verify-ext.svc.homelab.local::default_priority::max_requests::4294967295 outbound|80||mock.istio-verify-ext.svc.homelab.local::default_priority::max_retries::4294967295 outbound|80||mock.istio-verify-ext.svc.homelab.local::high_priority::max_connections::1024 outbound|80||mock.istio-verify-ext.svc.homelab.local::high_priority::max_pending_requests::1024 outbound|80||mock.istio-verify-ext.svc.homelab.local::high_priority::max_requests::1024 outbound|80||mock.istio-verify-ext.svc.homelab.local::high_priority::max_retries::3 outbound|80||mock.istio-verify-ext.svc.homelab.local::added_via_api::true outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::cx_active::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::cx_connect_fail::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::cx_total::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::rq_active::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::rq_error::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::rq_success::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::rq_timeout::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::rq_total::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::hostname::mock.istio-verify-ext.svc.homelab.local outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::health_flags::healthy outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::weight::1 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::region:: outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::zone:: outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::sub_zone:: outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::canary::false outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::priority::0 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::success_rate::-1 outbound|80||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:80::local_origin_success_rate::-1 outbound|443||mock.istio-verify-ext.svc.homelab.local::observability_name::outbound|443||mock.istio-verify-ext.svc.homelab.local; outbound|443||mock.istio-verify-ext.svc.homelab.local::default_priority::max_connections::4294967295 outbound|443||mock.istio-verify-ext.svc.homelab.local::default_priority::max_pending_requests::4294967295 outbound|443||mock.istio-verify-ext.svc.homelab.local::default_priority::max_requests::4294967295 outbound|443||mock.istio-verify-ext.svc.homelab.local::default_priority::max_retries::4294967295 outbound|443||mock.istio-verify-ext.svc.homelab.local::high_priority::max_connections::1024 outbound|443||mock.istio-verify-ext.svc.homelab.local::high_priority::max_pending_requests::1024 outbound|443||mock.istio-verify-ext.svc.homelab.local::high_priority::max_requests::1024 outbound|443||mock.istio-verify-ext.svc.homelab.local::high_priority::max_retries::3 outbound|443||mock.istio-verify-ext.svc.homelab.local::added_via_api::true outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::cx_active::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::cx_connect_fail::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::cx_total::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::rq_active::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::rq_error::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::rq_success::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::rq_timeout::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::rq_total::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::hostname::mock.istio-verify-ext.svc.homelab.local outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::health_flags::healthy outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::weight::1 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::region:: outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::zone:: outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::sub_zone:: outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::canary::false outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::priority::0 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::success_rate::-1 outbound|443||mock.istio-verify-ext.svc.homelab.local::10.250.183.220:443::local_origin_success_rate::-1 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::observability_name::outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local; outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::default_priority::max_connections::4294967295 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::default_priority::max_pending_requests::4294967295 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::default_priority::max_requests::4294967295 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::default_priority::max_retries::4294967295 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::high_priority::max_connections::1024 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::high_priority::max_pending_requests::1024 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::high_priority::max_requests::1024 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::high_priority::max_retries::3 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::added_via_api::true outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::cx_active::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::cx_connect_fail::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::cx_total::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::rq_active::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::rq_error::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::rq_success::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::rq_timeout::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::rq_total::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::hostname::multi-ip-target.istio-vt-t03.svc.homelab.local outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::health_flags::healthy outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::weight::1 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::region:: outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::zone:: outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::sub_zone:: outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::canary::false outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::priority::0 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::success_rate::-1 outbound|80||multi-ip-target.istio-vt-t03.svc.homelab.local::10.255.126.5:80::local_origin_success_rate::-1 (exit: 0, empty means the cluster has no registered endpoints in the EDS-driven /clusters admin listing) ===== Diagnostic: full clusters json for the homelab.local cluster from config_dump ===== ===== Diagnostic: confirm cluster 'outbound|80||flaky-echo.istio-vt-t05.svc.homelab.local' does NOT exist in CDS ===== flaky-echo related clusters in CDS: ['outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local'] ===== Diagnostic: raw access log lines from client istio-proxy for flaky-echo requests (response flags) ===== [2026-07-04T22:32:44.195Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "eff748b5-7e29-43b8-bbe8-3b97c29d1ad9" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33550 - - [2026-07-04T22:32:44.200Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "9cdc379f-b2ae-4513-9462-fb6759ad9790" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33554 - - [2026-07-04T22:32:44.205Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "fa90d6c5-53ce-41be-a344-d9edb023555a" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33562 - - [2026-07-04T22:32:44.211Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "830b758c-5c8c-4819-ba7e-619840579a28" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33570 - - [2026-07-04T22:32:44.216Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "d3c2131d-d426-49aa-b8a4-7f38ae8fa152" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33584 - - [2026-07-04T22:32:44.221Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "98df7801-a727-4858-ba52-4d30e4f5e1e0" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33598 - - [2026-07-04T22:32:44.227Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "d622462b-cc30-4d82-acf3-258d42eca7cd" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33602 - - [2026-07-04T22:32:44.233Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "f4283c20-4cd0-4d54-8b53-538372eda131" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33610 - - [2026-07-04T22:32:44.239Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "50bffb80-d19d-44ba-8e42-c7513d2ebc41" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33620 - - [2026-07-04T22:32:44.245Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "6e950153-9ee6-4b0e-a89e-b969c7e7784d" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33628 - - [2026-07-04T22:32:44.251Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "f49637ef-3941-45dc-afc0-1603cd165c48" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33644 - - [2026-07-04T22:32:44.257Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "7ad3f3e6-e2bc-4bb7-8bbb-1dcf79e6e0c2" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33660 - - [2026-07-04T22:32:44.263Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "77f4778e-bf3e-926c-9f29-1a5d9c5c24e7" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33662 - - [2026-07-04T22:32:44.268Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "626ee864-944c-4f4a-ae5f-84fe4305c22f" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33664 - - [2026-07-04T22:32:44.274Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "ae2920a2-9c67-4165-8e65-b1db177b9862" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33668 - - [2026-07-04T22:32:44.279Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "8789b3cd-efe3-4d92-a86b-fe83f2640fd5" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33682 - - [2026-07-04T22:32:44.285Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "607086e8-5ad4-4a86-a841-abe025f6ce3c" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33686 - - [2026-07-04T22:32:44.291Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "cdbb7015-ae32-4aae-a8d6-e3b474fe5f4b" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33702 - - [2026-07-04T22:32:44.297Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "71083f8d-1ed3-4a32-9de8-f083678f2622" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33708 - - [2026-07-04T22:32:44.302Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "3eff7840-c1e4-40ef-a6b7-0f512bab1922" "flaky-echo.istio-vt-t05.svc.homelab.local" "-" - - 10.250.168.166:80 10.255.126.9:33718 - - ===== CRITICAL FINDING: VS host 'svc.homelab.local' does not match Istio's internal registry hostname 'svc.cluster.local' ===== Access log shows ~40% FI fault_filter_abort (fault injection worked at the route match layer) and ~60% NC cluster_not_found (route pointed to a cluster that was never created by CDS, because 'flaky-echo.istio-vt-t05.svc.homelab.local' does not correspond to any registered Istio service -- Istio's internal K8s service registry hostname is 'flaky-echo.istio-vt-t05.svc.cluster.local' regardless of the cluster's actual kubeadm/CoreDNS clusterDomain (homelab.local). Confirmed the real cluster 'outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local' has rq_total=0 (real endpoints were NEVER hit), while cluster 'outbound|80||flaky-echo.istio-vt-t05.svc.homelab.local' referenced by our VS route does not exist at all in CDS (config_dump dynamic_active_clusters). => The test AS LITERALLY SPECIFIED (with the blanket svc.homelab.local substitution) never exercises outlier detection on the real pods at all; all failures are FI/NC, never UH. => Retesting with DR/VS host corrected to the mesh-internal hostname (svc.cluster.local) so that the policy actually attaches to the real, already-registered EDS cluster with the live healthy pods. ===== CORRECTED RE-RUN: DR/VS host patched to 'flaky-echo.istio-vt-t05.svc.cluster.local' (matches mesh registry); client curl target changed to short name 'flaky-echo' (resolves via search domain, Host header matches the merged vhost) ===== Confirmed via config_dump: single vhost now serves domains including cluster.local FQDN/short-name/IP, fault filter attached, route->outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local (the REAL EDS cluster, healthy endpoints). Confirmed cluster's outlier_detection block: consecutive_5xx=1, interval=2s, base_ejection_time=10s, max_ejection_percent=100. ===== Command 2 (corrected): 100 requests to flaky-echo (short name), tally status codes ===== $ kubectl -n istio-vt-t05 exec client -c curl -- sh -c 'for i in $(seq 1 100); do curl -s -o /dev/null -w "%{http_code} " http://flaky-echo/; done' | sort | uniq -c 62 200 38 503 ===== Command 3 (corrected): check outlier ejection state via istio-proxy admin /clusters ===== $ kubectl -n istio-vt-t05 exec client -c istio-proxy -- curl -s localhost:15000/clusters | grep flaky-echo | grep -E "outlier_check::ejections_active|health_flags" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::health_flags::healthy outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::health_flags::healthy ===== Command 4 (corrected): count of UH in istio-proxy access log ===== $ kubectl -n istio-vt-t05 logs client -c istio-proxy --since=2m | grep -c "\"UH\"" 0 ===== sample UH lines ===== ===== sample FI lines (fault injection, distinct from UH) ===== [2026-07-04T22:39:38.774Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "41db5679-a1e2-4c26-981a-617af0aa9d9d" "flaky-echo" "-" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local - 10.250.168.166:80 10.255.126.9:39556 - - [2026-07-04T22:39:38.789Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "a35a3f12-bad8-438d-a85a-cf7325d449a6" "flaky-echo" "-" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local - 10.250.168.166:80 10.255.126.9:39578 - - [2026-07-04T22:39:38.799Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "ff893d15-11ec-4246-9690-4c85e89041c2" "flaky-echo" "-" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local - 10.250.168.166:80 10.255.126.9:39594 - - [2026-07-04T22:39:38.804Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "7b1f6849-86a2-4814-ad2d-ef4e72a6f819" "flaky-echo" "-" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local - 10.250.168.166:80 10.255.126.9:39600 - - [2026-07-04T22:39:38.820Z] "GET / HTTP/1.1" 503 FI fault_filter_abort - "-" 0 18 0 - "-" "curl/8.14.1" "55ca3acc-0a85-480f-b7e2-08b74f0e9e98" "flaky-echo" "-" outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local - 10.250.168.166:80 10.255.126.9:39626 - - ===== Diagnostic: could 'no ejection' be a TIMING artifact (100 fast requests all completed < 1-2 outlier check intervals)? ===== Access log timestamps for the 100-request burst span ~1-2s wall clock; outlier interval=2s -> check for timing artifact, run an EXTENDED test: 150 requests spaced 200ms apart (30s total, spans ~15 outlier-check intervals of 2s each). $ kubectl -n istio-vt-t05 exec client -c curl -- sh -c 'for i in $(seq 1 150); do curl -s -o /dev/null -w "%{http_code} " http://flaky-echo/; sleep 0.2; done' | sort | uniq -c 78 200 72 503 ===== post-extended-run: cluster health/ejection state ===== outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::health_flags::healthy outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::health_flags::healthy ===== post-extended-run: UH count in access log ===== 0 ===== post-extended-run: response-flag breakdown (field 4 of the access log = response flags) ===== 140 " 200 - 109 " 503 FI ===== Diagnostic: confirm the 200 responses DID reach the real upstream hosts (rq_total/rq_success > 0), proving fault-aborted requests bypass host selection entirely ===== outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_error::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_success::76 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.194.113:5678::rq_total::76 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_error::0 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_success::64 outbound|80||flaky-echo.istio-vt-t05.svc.cluster.local::10.255.126.10:5678::rq_total::64 ===== Final confirmation: get pods (both should be Running/Ready throughout - never actually died) ===== NAME READY STATUS RESTARTS AGE flaky-echo-6bd88f97fc-k6449 2/2 Running 0 9m30s flaky-echo-6bd88f97fc-mz77b 2/2 Running 0 9m30s