=== T52 execution log === Date: 2026-07-05T00:04:13Z ### CMD1: apply + wait === $ kubectl apply -f client-echo.yaml -f echo-connpool-limit.yaml && kubectl -n ISTIO_VT_NS wait --for=condition=Ready pod/client --timeout=90s && kubectl -n ISTIO_VT_NS rollout status deploy/echo (already applied via manifest.yaml combining both files; substituted namespace = istio-vt-t52) NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 10s 10.255.126.1 k8s-worker2 echo-5767bbcc56-7fntl 2/2 Running 0 10s 10.255.126.44 k8s-worker2 ### CMD2: before UO stats === $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats | grep 'outbound|80||echo.*upstream_cx_overflow' > /tmp/before_uo.txt; cat /tmp/before_uo.txt [exit status of grep/pipeline above: 0] [before_uo_t52.txt content - EMPTY, no matching stat line found] (raw before_uo.txt is empty; investigating why - see IMPORTANT FINDINGS section below) ### DIAGNOSTIC (before CMD3): DestinationRule host mismatch confirmed === $ istioctl -n istio-vt-t52 analyze Warning [IST0174] (DestinationRule istio-vt-t52/echo-connpool-limit) The host echo.istio-vt-t52.svc.homelab.local defined in the DestinationRule does not match any services in the mesh. $ istioctl -n istio-vt-t52 proxy-config cluster client.istio-vt-t52 --fqdn echo.istio-vt-t52.svc.cluster.local -o json | jq circuitBreakers outbound|80||echo.istio-vt-t52.svc.cluster.local -> circuitBreakers.thresholds[0].maxConnections = 4294967295 (UNLIMITED / default, NOT 2) => The DestinationRule spec.host = echo.ISTIO_VT_NS.svc.homelab.local (as literally written in the spec manifest) does NOT match the actual Istio service-registry hostname for k8s-derived Services, which remains echo.ISTIO_VT_NS.svc.cluster.local regardless of the real kubeadm clusterDomain (homelab.local). This is because istiod constructs Service hostnames using its own internal/default domain suffix (cluster.local), independent of the actual cluster DNS domain used for name resolution. Confirmed for multiple pre-existing namespaces (istio-vt-t57/t58/t63/t69/t75, istio-verify) via /clusters dump - ALL show *.svc.cluster.local as the Envoy cluster name, never *.svc.homelab.local (except where an explicit ServiceEntry with a custom hostname was created, e.g. weight-echo in t69). => RESULT: the connectionPool trafficPolicy in the literal spec manifest NEVER attaches to the actual echo cluster. ### DIAGNOSTIC 2: mesh-wide proxyStatsMatcher excludes ALL outbound cluster stats from /stats === $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats | wc -l => 203 (total) $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats | grep -c 'outbound|' => 0 => This proxy exposes ONLY cluster.xds-grpc (control plane) stats via /stats; ALL outbound service-cluster stats (including upstream_cx_overflow for ANY cluster, not just echo) are excluded by the mesh proxyStatsMatcher config. This means CMD2/CMD4 (grep on /stats for upstream_cx_overflow) will return EMPTY regardless of whether circuit breaking actually occurs - independent confirmation needed via access logs (CMD5) and observed HTTP status codes (CMD3). ### CMD3 (AS LITERALLY SPECIFIED, broken DR host still in place): 40 concurrent requests === $ kubectl -n istio-vt-t52 exec client -c curl -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code} " --max-time 3 http://echo.istio-vt-t52.svc.homelab.local/ & done; wait; echo' 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 ### CMD4 (AS LITERALLY SPECIFIED): after UO stats === $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats | grep 'outbound|80||echo.*upstream_cx_overflow' > /tmp/after_uo.txt; cat /tmp/after_uo.txt [EMPTY - no matching line, consistent with DIAGNOSTIC 2 above] ### CMD5 (AS LITERALLY SPECIFIED): UO count in access log === $ kubectl -n istio-vt-t52 logs client -c istio-proxy --since=1m | grep '"UO"' | wc -l 0 --- sample of client istio-proxy access log lines (last 5) for context --- [2026-07-05T00:09:07.003Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 606 8 7 "-" "curl/8.14.1" "a37bf0e3-3928-4b76-b4b7-f21be86d2936" "echo.istio-vt-t52.svc.homelab.local" "10.250.146.97:80" PassthroughCluster 10.255.126.1:42848 10.250.146.97:80 10.255.126.1:42838 - allow_any [2026-07-05T00:09:07.005Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 606 6 5 "-" "curl/8.14.1" "a7e2e041-865b-4daa-ac5f-4ea6fba80aee" "echo.istio-vt-t52.svc.homelab.local" "10.250.146.97:80" PassthroughCluster 10.255.126.1:42538 10.250.146.97:80 10.255.126.1:42878 - allow_any [2026-07-05T00:09:07.004Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 606 9 9 "-" "curl/8.14.1" "43699948-8b5d-4d6a-a722-91b834eedcd5" "echo.istio-vt-t52.svc.homelab.local" "10.250.146.97:80" PassthroughCluster 10.255.126.1:42870 10.250.146.97:80 10.255.126.1:42856 - allow_any [2026-07-05T00:09:07.005Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 606 7 7 "-" "curl/8.14.1" "d64db415-df21-47c4-9194-e2b4f2c2e69a" "echo.istio-vt-t52.svc.homelab.local" "10.250.146.97:80" PassthroughCluster 10.255.126.1:42890 10.250.146.97:80 10.255.126.1:42876 - allow_any [2026-07-05T00:09:07.012Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 606 2 2 "-" "curl/8.14.1" "b8c837b3-712e-40ee-99b7-a8a2fbfe9626" "echo.istio-vt-t52.svc.homelab.local" "10.250.146.97:80" PassthroughCluster 10.255.126.1:42538 10.250.146.97:80 10.255.126.1:42904 - allow_any ================================================================== ### FOLLOW-UP CORRECTED RE-TEST (to properly exercise the claim) === ================================================================== Root cause established above: this cluster's Istio installation constructs its internal Service registry hostnames with the DEFAULT '.svc.cluster.local' suffix regardless of the actual kubeadm clusterDomain (homelab.local). This affects BOTH (a) DestinationRule.host matching AND (b) the HTTP virtual-host domain list Envoy uses to route a request to its named cluster. A client request whose Host header uses the real cluster DNS suffix (*.svc.homelab.local) resolves fine via CoreDNS but does NOT match any known virtual-host domain in Envoy, so under the mesh's ALLOW_ANY outbound policy it silently falls through to 'PassthroughCluster' -- bypassing ALL DestinationRule policy (circuit breaker, mTLS upgrade, etc.) even though the destination is an in-mesh pod. Using the short service name ('echo') instead avoids the domain suffix entirely and correctly resolves+routes through 'outbound|80||echo.istio-vt-t52.svc.cluster.local'. Corrective steps applied (NOT part of the literal spec, done to actually exercise the claim under test): 1) kubectl -n istio-vt-t52 patch destinationrule echo-connpool-limit --type merge -p '{"spec":{"host":"echo.istio-vt-t52.svc.cluster.local"}}' -> circuitBreakers.thresholds[0] now: maxConnections=2, maxPendingRequests=1 (confirmed via istioctl proxy-config cluster) 2) burst test re-run using short name 'http://echo/' instead of the FQDN, to ensure requests route through the correctly-configured named cluster (outbound|80||echo.istio-vt-t52.svc.cluster.local) instead of PassthroughCluster. $ kubectl -n istio-vt-t52 exec client -c curl -- sh -c 'for i in $(seq 1 40); do curl -s -o /dev/null -w "%{http_code} " --max-time 3 http://echo/ & done; wait; echo' 200 200 503 200 200 200 200 200 200 200 503 503 503 503 503 503 503 503 503 200 503 503 503 200 200 503 503 503 200 503 503 503 200 200 503 200 503 503 200 200 --- count of 503 vs 200 in corrected burst --- 18 200 22 503 $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats | grep 'outbound|80||echo.*upstream_cx_overflow' (post-burst, corrected DR) [still empty - confirms mesh-wide stats matcher excludes this regardless of DR correctness, see DIAGNOSTIC 2] $ kubectl -n istio-vt-t52 logs client -c istio-proxy --since=1m | grep '"UO"' | wc -l (post-corrected-burst) 0 --- sample access log lines showing UO flag and 503 --- ### DIAGNOSTIC 3: access log format is Envoy TEXT (not JSON) - explains CMD5 literal grep returning 0 === $ kubectl -n istio-vt-t52 logs client -c istio-proxy --since=1m | grep -c ' UO ' (correct TEXT-format flag match) 22 Sample matching line: [2026-07-05T00:10:32.798Z] "GET / HTTP/1.1" 503 UO upstream_reset_before_response_started{overflow} - "-" 0 81 0 - "-" "curl/8.14.1" "..." "echo" "-" outbound|80||echo.istio-vt-t52.svc.cluster.local - 10.250.146.97:80 10.255.126.1:47062 - default => The literal spec grep pattern '"UO"' assumes a JSON access log (e.g. {"response_flags":"UO",...}), but this installation emits Envoy's default TEXT access log format, where UO appears as a bare space-delimited token, never inside double quotes as a standalone field. This is why CMD5 as literally specified returns 0 regardless of whether UO actually occurred. ### DIAGNOSTIC 4: /stats and /stats/prometheus BOTH exclude all outbound-cluster counters mesh-wide === $ kubectl -n istio-vt-t52 exec client -c istio-proxy -- curl -s localhost:15000/stats/prometheus | grep -i overflow (only cluster_name="xds-grpc" entries present; no echo cluster entries at all, even after triggering real overflow events below) => Confirms CMD2/CMD4 (via /stats) can NEVER show upstream_cx_overflow for the echo cluster on this installation, independent of whether circuit breaking is genuinely occurring. ================================================================== ### SUMMARY OF CORRECTED RE-TEST (properly exercising the claim) === ================================================================== - circuitBreakers.thresholds[0] confirmed applied: maxConnections=2, maxPendingRequests=1 (was 4294967295/unset before host fix) - 40 concurrent requests via short name 'echo' (correct in-mesh routing): 18x200, 22x503 - All 22x503 carry response flag 'UO' with detail 'upstream_reset_before_response_started{overflow}' in the client sidecar's TEXT-format access log - this IS Envoy's standard circuit-breaker-overflow signature. - /stats and /stats/prometheus upstream_cx_overflow counter for the echo cluster: NOT OBSERVABLE on this cluster (mesh-wide mesh stats-matcher trims all outbound cluster stats down to nothing but xds-grpc control-plane stats). CONCLUSION: connectionPool.tcp.maxConnections DOES control an Envoy circuit breaker that rejects excess concurrent connections with 503 + response flag UO + reason 'overflow', exactly as the covered docs claim - CONFIRMED empirically once the DestinationRule host correctly matches the mesh's Service registry hostname and traffic is routed within the mesh (not PassthroughCluster). The 'upstream_cx_overflow counter increment via /stats' portion of pass_criteria could not be directly confirmed on this specific cluster due to an orthogonal mesh-wide stats-matcher config that excludes essentially all outbound-cluster stats from admin/Prometheus endpoints - this is a real, separate, and practically important limitation for this homelab's Istio setup, unrelated to the correctness of the circuit breaking mechanism itself.