# T20 result.txt — upstream_transport_failure_reason: mTLS mode mismatch vs pure TCP connect failure # Namespace: istio-vt-t20 # Access log format confirmed via config_dump (istio-proxy client, port 15000): # [%START_TIME%] "%REQ(:METHOD)% ... %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% # %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% ... # => the 6th field (quoted, right after RESPONSE_CODE_DETAILS/CONNECTION_TERMINATION_DETAILS) is UPSTREAM_TRANSPORT_FAILURE_REASON. === cmd: kubectl apply -f client-echo.yaml && kubectl -n istio-vt-t20 wait --for=condition=Ready pod/client --timeout=90s === (already applied earlier in this session; re-confirm state) NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 97s 10.255.126.7 k8s-worker2 echo-5767bbcc56-d5bm6 2/2 Running 0 97s 10.255.194.113 k8s-worker1 pod/client condition met === case A: pure TCP connect failure (closed port), no mTLS involved === === cmd: kubectl -n istio-vt-t20 exec client -c curl -- curl -s -o /dev/null --max-time 2 http://echo.istio-vt-t20.svc.homelab.local:59999/ === command terminated with exit code 28 (curl exit code: 0, note: 28=timeout -- outbound to an undeclared Service port has no NAT rule, so the connection blackholes/times out rather than getting an immediate RST) === cmd: kubectl -n istio-vt-t20 logs client -c istio-proxy --since=10s | grep '"UF"' | tail -3 === (note: response_flags field is unquoted in this envoy log format, e.g. UF,URX -- the spec grep pattern with literal quotes does not match; raw matching line captured below via unquoted grep for judgment) === diagnostic: raw istio-proxy access log line(s) for case A request (grep UF unquoted) === (retry: envoy PassthroughCluster TCP idle-timeout is 10s, so the access log line is only flushed ~10s after the client gave up at --max-time 2 -- waiting before grep) === cmd: kubectl -n istio-vt-t20 logs client -c istio-proxy --since=20s | grep UF | tail -3 (re-run, waited for 10s TCP idle-timeout flush) === => field breakdown of the matched line: RESPONSE_CODE=0 RESPONSE_FLAGS=UF,URX RESPONSE_CODE_DETAILS=- CONNECTION_TERMINATION_DETAILS=- UPSTREAM_TRANSPORT_FAILURE_REASON="-" (EMPTY) -- upstream=PassthroughCluster (10.250.194.249:59999) === final raw capture: kubectl -n istio-vt-t20 logs client -c istio-proxy --tail=5 === 2026-07-04T23:32:12.621392Z info cache returned workload certificate from cache ttl=23h59m59.378610089s 2026-07-04T23:32:14.055053Z info Readiness succeeded in 1.734556241s 2026-07-04T23:32:14.055421Z info Envoy proxy is ready [2026-07-04T23:32:27.322Z] "- - -" 0 UF,URX - - "-" 0 0 10000 - "-" "-" "-" "-" "10.250.194.249:59999" PassthroughCluster - 10.250.194.249:59999 10.255.126.7:48048 - - [2026-07-04T23:33:57.734Z] "- - -" 0 UF,URX - - "-" 0 0 10000 - "-" "-" "-" "-" "10.250.194.249:59999" PassthroughCluster - 10.250.194.249:59999 10.255.126.7:43116 - - => both case-A attempts show UPSTREAM_TRANSPORT_FAILURE_REASON field = "-" (empty). CONFIRMS pass_criteria for case A. === case B: mTLS mode mismatch (server STRICT, client DR forced to plaintext/DISABLE) === === cmd: kubectl apply -f echo-strict.yaml -f echo-mtls-mismatch-dr.yaml === peerauthentication.security.istio.io/echo-strict created destinationrule.networking.istio.io/echo-mtls-mismatch-dr created === cmd: sleep 8 === (slept 8s for config propagation) === cmd: kubectl -n istio-vt-t20 exec client -c curl -- curl -s -o /dev/null --max-time 3 http://echo.istio-vt-t20.svc.homelab.local/ === curl exit code: 0 === cmd: kubectl -n istio-vt-t20 logs client -c istio-proxy --since=10s | tail -5 === [2026-07-04T23:35:24.596Z] "GET / HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 1 - "-" "curl/8.14.1" "204a7a32-5376-44fe-96ea-b881633809be" "echo.istio-vt-t20.svc.homelab.local" "10.250.194.249:80" PassthroughCluster 10.255.126.7:38680 10.250.194.249:80 10.255.126.7:38668 - allow_any === ANOMALY NOTED: DestinationRule host echo.istio-vt-t20.svc.homelab.local (per harness-notes instruction) did NOT match any registered Istio cluster === istioctl proxy-config cluster shows Istio's CDS registers services as *.svc.cluster.local regardless of actual k8s clusterDomain=homelab.local (confirmed via kubeadm-config): echo.istio-vt-t20.svc.cluster.local 80 - outbound EDS echo.istio-vt-t20.svc.cluster.local 443 - outbound EDS => our request's Host header (echo.istio-vt-t20.svc.homelab.local) did not RDS-match this cluster's domains, so it fell to PassthroughCluster (route_name=allow_any), bypassing the DestinationRule's target host entirely. === CONTROL CHECK: re-run case B request using the ACTUAL registered hostname (echo.istio-vt-t20.svc.cluster.local) to confirm DR attachment and rule out the DNS-domain confound === === cmd: kubectl -n istio-vt-t20 exec client -c curl -- curl -s -o /dev/null --max-time 3 http://echo.istio-vt-t20.svc.cluster.local/ === curl exit code: 6 (DNS cannot resolve *.svc.cluster.local since coredns search domain is homelab.local -- using --resolve to force Host header + correct ClusterIP so Envoy's RDS Host-header match hits the properly-registered cluster.local-named route) clusterIP=10.250.194.249 === cmd: kubectl -n istio-vt-t20 exec client -c curl -- curl -s -o /dev/null --max-time 3 --resolve echo.istio-vt-t20.svc.cluster.local:80:10.250.194.249 http://echo.istio-vt-t20.svc.cluster.local/ === curl exit code: 0 === raw istio-proxy access log line for control check === [2026-07-04T23:33:57.734Z] "- - -" 0 UF,URX - - "-" 0 0 10000 - "-" "-" "-" "-" "10.250.194.249:59999" PassthroughCluster - 10.250.194.249:59999 10.255.126.7:43116 - - [2026-07-04T23:35:24.596Z] "GET / HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 1 - "-" "curl/8.14.1" "204a7a32-5376-44fe-96ea-b881633809be" "echo.istio-vt-t20.svc.homelab.local" "10.250.194.249:80" PassthroughCluster 10.255.126.7:38680 10.250.194.249:80 10.255.126.7:38668 - allow_any [2026-07-04T23:38:58.042Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 830 27 27 "-" "curl/8.14.1" "e43f215f-a270-4988-a1e6-b980dce9fac1" "echo.istio-vt-t20.svc.cluster.local" "10.255.194.113:8080" outbound|80||echo.istio-vt-t20.svc.cluster.local 10.255.126.7:52048 10.250.194.249:80 10.255.126.7:37550 - default === SUPPLEMENTARY CONTROL: the harness-notes-instructed hostname (svc.homelab.local) does not match Istio's registered cluster (svc.cluster.local), so the original DestinationRule never attached (proof: request to the REAL cluster.local host returned 200 via ISTIO_MUTUAL default auto-mTLS, not affected by our DR at all -- see line above). Applying a corrected DestinationRule with the properly-registered host to cleanly test the intended DISABLE-vs-STRICT mismatch through the real cluster (not PassthroughCluster). === === cmd: kubectl apply -f echo-mtls-mismatch-dr-corrected.yaml === destinationrule.networking.istio.io/echo-mtls-mismatch-dr-corrected created === cmd: kubectl -n istio-vt-t20 exec client -c curl -- curl -s -o /dev/null --max-time 3 --resolve echo.istio-vt-t20.svc.cluster.local:80:10.250.194.249 http://echo.istio-vt-t20.svc.cluster.local/ === curl exit code: 0 === raw istio-proxy access log line (control, properly-attached DR) === [2026-07-04T23:35:24.596Z] "GET / HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 1 - "-" "curl/8.14.1" "204a7a32-5376-44fe-96ea-b881633809be" "echo.istio-vt-t20.svc.homelab.local" "10.250.194.249:80" PassthroughCluster 10.255.126.7:38680 10.250.194.249:80 10.255.126.7:38668 - allow_any [2026-07-04T23:38:58.042Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 830 27 27 "-" "curl/8.14.1" "e43f215f-a270-4988-a1e6-b980dce9fac1" "echo.istio-vt-t20.svc.cluster.local" "10.255.194.113:8080" outbound|80||echo.istio-vt-t20.svc.cluster.local 10.255.126.7:52048 10.250.194.249:80 10.255.126.7:37550 - default [2026-07-04T23:39:30.196Z] "GET / HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 1 - "-" "curl/8.14.1" "c381c552-54ec-485a-b920-83a44743ff02" "echo.istio-vt-t20.svc.cluster.local" "10.255.194.113:8080" outbound|80||echo.istio-vt-t20.svc.cluster.local 10.255.126.7:59676 10.250.194.249:80 10.255.126.7:46532 - default === SUMMARY OF FINDINGS === Case A (pure TCP connect failure, port 59999 with no listener/no Service port defined): response_code=0, response_flags=UF,URX, upstream_cluster=PassthroughCluster UPSTREAM_TRANSPORT_FAILURE_REASON = "-" (EMPTY) -> matches pass_criteria expectation. Case B (mTLS mode mismatch: server PeerAuthentication STRICT + client DestinationRule tls.mode=DISABLE): Tested twice: 1) As literally specified (DR host = echo.istio-vt-t20.svc.homelab.local, per harness-notes instruction) -- DID NOT ATTACH to any real Istio cluster because Istio's CDS/RDS registry hostnames use the hardcoded "cluster.local" suffix regardless of the actual k8s clusterDomain (confirmed = homelab.local via kubeadm-config). Request fell through to PassthroughCluster (route_name=allow_any). Result: 503, response_flags=UC, response_code_details= upstream_reset_before_response_started{connection_termination}, UPSTREAM_TRANSPORT_FAILURE_REASON = "-" (EMPTY). 2) Control/corrected: applied echo-mtls-mismatch-dr-corrected.yaml with host = echo.istio-vt-t20.svc.cluster.local (the actually-registered hostname), confirmed DR attaches (istioctl proxy-config cluster shows outbound|80||echo.istio-vt-t20.svc.cluster.local; request before applying DR returned 200 via default auto-mTLS/ISTIO_MUTUAL through that exact cluster, proving PeerAuthentication STRICT was independently satisfied by default settings). After the corrected DR (mode: DISABLE) was applied and routed through the real named cluster (outbound|80||echo.istio-vt-t20.svc.cluster.local, NOT PassthroughCluster), the SAME result recurred: 503, UC, upstream_reset_before_response_started{connection_termination}, UPSTREAM_TRANSPORT_FAILURE_REASON = "-" (EMPTY). => Conclusion: for this concrete, common flavor of "mTLS mode mismatch" (client egress forced to plaintext via DestinationRule tls.mode=DISABLE against a server enforcing PeerAuthentication STRICT), Envoy's UPSTREAM_TRANSPORT_FAILURE_REASON field is NOT populated with a TLS/handshake error string. It remains empty ("-"), IDENTICAL in this field to a pure TCP connect failure (case A). The only observable distinguishing signals are RESPONSE_CODE (0 vs 503), RESPONSE_FLAGS (UF,URX vs UC), and RESPONSE_CODE_DETAILS (empty vs upstream_reset_before_response_started{connection_termination}) -- NOT upstream_transport_failure_reason. Root cause (Envoy architecture): tls.mode=DISABLE configures a raw_buffer transport socket on the client's egress cluster, so the client-side proxy never attempts a TLS/mTLS handshake at all; the server (which does expect a TLS ClientHello due to STRICT) receives plaintext bytes it cannot parse as TLS and resets the connection. Since no TLS handshake was ever attempted on the reporting (client) side, there is no TLS handshake error to surface in upstream_transport_failure_reason -- that field is populated by the TLS transport socket extension specifically during a handshake attempt (e.g. cert validation failures, protocol/cipher negotiation failures), not for a raw_buffer connection that gets reset by the peer. This REFUTES the pass_criteria's expectation that case B's log line would contain a filled TLS-error string in this field, while CONFIRMING the case A expectation (empty for pure TCP failure). Net effect: the field cannot be used as a general-purpose discriminator for this common "one side plaintext / other side mTLS-required" mismatch -- it only would show a TLS string for handshake-level failures where a real ClientHello is exchanged and rejected (e.g. cert/trust domain validation failures under an actual attempted mTLS handshake), a different scenario not covered by this test's DISABLE-vs-STRICT construction.