### T47 RUN 2 execution log (namespace istio-vt-t47-r2) ### ### Corrected adaptation applied per harness-notes.md CORRECTION (2026-07-05) and ### TEST-SPECIFIC GUIDANCE: DestinationRule host + VS "hop-gw-route-to-mesh" tls-route ### destination host changed from hop-gw..svc.homelab.local -> hop-gw..svc.cluster.local ### (Istio's internal service-registry FQDN for a real k8s Service, independent of the ### cluster's real kubeadm clusterDomain=homelab.local). ServiceEntry hosts, Gateway hosts, ### VS sniHosts, and the at-gw VS's route-to-mock host were left as literal ### mock.istio-verify-ext.svc.homelab.local per the "ServiceEntry hosts are literal" rule. $ kubectl create namespace istio-vt-t47-r2 && kubectl label namespace istio-vt-t47-r2 istio-injection=enabled namespace/istio-vt-t47-r2 created namespace/istio-vt-t47-r2 labeled $ kubectl apply -f manifest.yaml pod/client created deployment.apps/echo created service/echo created serviceaccount/hop-gw created deployment.apps/hop-gw created service/hop-gw created serviceentry.networking.istio.io/hop-gw-mock-se created gateway.networking.istio.io/hop-gw-gateway created destinationrule.networking.istio.io/hop-gw-dr created virtualservice.networking.istio.io/hop-gw-route-to-mesh created virtualservice.networking.istio.io/hop-gw-route-at-gw created $ kubectl -n istio-vt-t47-r2 wait --for=condition=Ready pod/client --timeout=90s && \ kubectl -n istio-vt-t47-r2 rollout status deploy/echo --timeout=90s && \ kubectl -n istio-vt-t47-r2 rollout status deploy/hop-gw --timeout=90s pod/client condition met deployment "echo" successfully rolled out deployment "hop-gw" successfully rolled out NAME READY STATUS RESTARTS AGE client 2/2 Running 0 5s echo-5767bbcc56-vpjvv 2/2 Running 0 5s hop-gw-5ccf775548-mkl4g 1/1 Running 0 5s ### CMD1 (spec) $ kubectl -n istio-vt-t47-r2 exec client -c curl -- curl -sk -o /dev/null -w 'twohop_https=%{http_code}\n' \ https://mock.istio-verify-ext.svc.homelab.local/ twohop_https=200 ### CMD2 (spec) -- hop-gw access log, expect L4-only format if traffic actually transits hop-gw $ kubectl -n istio-vt-t47-r2 logs deploy/hop-gw -c istio-proxy --since=1m | tail -10 2026-07-05T01:08:15.282914Z info ads ADS: new connection for node:1 2026-07-05T01:08:15.283516Z info ads ADS: new connection for node:2 2026-07-05T01:08:15.308901Z info cache generated new workload certificate resourceName=default latency=121.35644ms ttl=23h59m59.691103027s 2026-07-05T01:08:15.308930Z info cache Root cert has changed, start rotating root cert 2026-07-05T01:08:15.308963Z info cache returned workload trust anchor from cache ttl=23h59m59.691036852s 2026-07-05T01:08:15.308982Z info cache returned workload certificate from cache ttl=23h59m59.691018818s 2026-07-05T01:08:15.309042Z info cache returned workload trust anchor from cache ttl=23h59m59.690959627s 2026-07-05T01:08:15.309296Z info cache returned workload trust anchor from cache ttl=23h59m59.690705089s 2026-07-05T01:08:16.857538Z info Readiness succeeded in 1.882349233s 2026-07-05T01:08:16.858023Z info Envoy proxy is ready (NO access-log line at all -- zero access log entries after the 200 request, same as attempt1) ### CMD3 (spec) $ istioctl proxy-config listener client.istio-vt-t47-r2 --port 443 -o json | jq '.[0].filterChains[0].filters[].name' "istio.stats" "envoy.filters.network.tcp_proxy" ### CMD4 (spec) -- expect >0 if hop-gw truly in path $ kubectl -n istio-vt-t47-r2 exec deploy/hop-gw -c istio-proxy -- curl -s localhost:15000/stats/prometheus | grep -c '^istio_tcp_connections_opened_total' 0 ### CMD5 (spec) -- expect 0 $ kubectl -n istio-vt-t47-r2 exec deploy/hop-gw -c istio-proxy -- curl -s localhost:15000/stats/prometheus | grep -c '^istio_requests_total' 0 ### CMD6 (spec) -- expect 0 $ istioctl proxy-config route client.istio-vt-t47-r2 -o json | jq '[.[] | select(.name | contains("mock.istio-verify-ext"))] | length' 0 ### CMD7 (spec) -- expect no retry counters $ kubectl -n istio-vt-t47-r2 exec client -c istio-proxy -- curl -s localhost:15000/stats | grep upstream_rq_retry | grep mock || echo 'no retry counters exist for this host (route never compiled)' no retry counters exist for this host (route never compiled) ### INVESTIGATION: even with the corrected registry-FQDN rule applied, pass_criteria is still ### contradicted -- hop-gw gets ZERO traffic. Root-caused at the LDS/listener-precedence level ### (a step further than attempt1's investigation) ### Step 1 -- confirm the DR/VS host fix actually took effect at CDS/EDS level (unlike attempt1's domain-corrected variant, which referenced a nonexistent CDS cluster): $ istioctl proxy-config cluster client.istio-vt-t47-r2 -o json | jq -r '.[] | select(.name|contains("hop-gw")) | .name' outbound|8443||hop-gw.istio-vt-t47-r2.svc.cluster.local $ istioctl proxy-config endpoints client.istio-vt-t47-r2 | grep -i hop-gw 10.255.159.138:8443 HEALTHY OK outbound|8443||hop-gw.istio-vt-t47-r2.svc.cluster.local => Confirmed: this run's CDS cluster for hop-gw exists AND has a live healthy endpoint (matching the real hop-gw pod IP). This is a real improvement over attempt1's variant, which referenced a phantom cluster. The registry-FQDN fix is itself valid and necessary. Step 2 -- despite CDS/EDS being correctly wired, hop-gw still saw 0 connections/requests (CMD4/CMD5 above). To find out why, dumped ALL of client's listeners bound on port 443 (not just filterChains[0] of the first listener, since Envoy creates one internal listener PER destination IP for every k8s Service in the registry): $ istioctl proxy-config listener client.istio-vt-t47-r2 --port 443 -o json | jq -r '.[].name' 10.250.13.92_443 10.250.140.93_443 ... (13 other per-ClusterIP listeners for unrelated pre-existing cluster services) ... 10.250.183.220_443 <-- this IS mock's real ClusterIP (kubectl get svc mock -n istio-verify-ext) 0.0.0.0_443 <-- the wildcard/catch-all virtual-outbound listener ... $ istioctl proxy-config listener client.istio-vt-t47-r2 --port 443 -o json \ | jq -r '.[] | select(.name=="10.250.183.220_443") | .filterChains[] | {serverNames:(.filterChainMatch.serverNames//[""]), filters:[.filters[].name], cluster:[.filters[].typedConfig.cluster]}' { "serverNames": [""], "filters": ["istio.stats", "envoy.filters.network.tcp_proxy"], "cluster": [null, "outbound|443||mock.istio-verify-ext.svc.cluster.local"] } => DECISIVE: the dedicated per-ClusterIP listener for mock has exactly ONE filter chain -- a plain default (no SNI match at all) TCP proxy straight to "outbound|443||mock.istio-verify-ext.svc.cluster.local". There is NO SNI-matched filter chain here referencing the "mesh" gateway's hop-gw route. The mesh-gateway VS's SNI match for "mock.istio-verify-ext.svc.homelab.local" only lives on the 0.0.0.0_443 wildcard listener -- but Envoy's original_dst internal redirection resolves to the MORE SPECIFIC per-destination-IP listener (10.250.183.220_443) whenever the dialed IP matches an existing k8s Service ClusterIP, so the wildcard/SNI-routing listener is never reached. Step 3 -- behavioral confirmation (matches attempt1's decisive test, re-run under the corrected manifest to make sure the fix genuinely doesn't change the outcome): $ kubectl -n istio-vt-t47-r2 scale deploy/hop-gw --replicas=0 $ kubectl -n istio-vt-t47-r2 wait --for=delete pod -l istio=hop-gw --timeout=60s pod/hop-gw-5ccf775548-mkl4g condition met $ kubectl -n istio-vt-t47-r2 exec client -c curl -- curl -sk -o /dev/null -w 'twohop_https_gwdown=%{http_code}\n' \ https://mock.istio-verify-ext.svc.homelab.local/ twohop_https_gwdown=200 => hop-gw pod fully deleted (0 replicas); request still succeeds. Gateway is not in the data path. $ kubectl -n istio-vt-t47-r2 scale deploy/hop-gw --replicas=1 && kubectl -n istio-vt-t47-r2 rollout status deploy/hop-gw --timeout=90s deployment "hop-gw" successfully rolled out $ kubectl -n istio-vt-t47-r2 exec client -c curl -- curl -sk -o /dev/null -w 'twohop_https_final=%{http_code}\n' \ https://mock.istio-verify-ext.svc.homelab.local/ twohop_https_final=200 $ kubectl -n istio-vt-t47-r2 exec deploy/hop-gw -c istio-proxy -- curl -s localhost:15000/stats/prometheus | grep -c '^istio_tcp_connections_opened_total' 0 => hop-gw restored, request still 200, hop-gw connection counter still 0 after the request. Confirms: not a one-off race, hop-gw structurally never receives this traffic in this manifest, with or without the registry-FQDN correction. Step 4 -- sanity: confirm the observability/logging pipeline itself is not broken cluster-wide (so the absence of signal at hop-gw reflects "no traffic", not "broken telemetry"): $ kubectl -n istio-vt-t47-r2 exec client -c curl -- curl -s -o /dev/null -w 'echo_http=%{http_code}\n' http://echo/ echo_http=200 $ kubectl -n istio-vt-t47-r2 logs deploy/echo -c istio-proxy --since=1m | tail -3 [2026-07-05T01:10:09.748Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 729 18 17 "-" "curl/8.14.1" ... echo.istio-vt-t47-r2.svc.cluster.local default => normal in-mesh L7 access log + 200, confirming telemetry/logging config is healthy in general. ### CONCLUSION ### Applying the corrected adaptation rule (DR/VS host = hop-gw..svc.cluster.local, the Istio service-registry FQDN, instead of the DNS-domain-based hop-gw..svc.homelab.local used in attempt1) DID fix the CDS/EDS-level defect found in attempt1 (a phantom cluster reference is now a real cluster with a live healthy endpoint). However, pass_criteria's central requirement ("hop-gw의 istio_tcp_connections_opened_total > 0 while E2E succeeds") is STILL violated, reproducibly, for a different and more fundamental reason found by this run: the ServiceEntry's "external" host (mock.istio-verify-ext.svc.homelab.local) resolves via kube-dns to the exact ClusterIP of a REAL, already-registered k8s Service (mock, in the mesh's own service registry under its cluster.local name). Istio/Envoy always generates a dedicated per-ClusterIP listener for every k8s Service in the registry, and that dedicated listener takes precedence over the wildcard 0.0.0.0_443 listener where the "mesh"-gateway SNI-based VirtualService routing lives. Client's sidecar therefore never evaluates the SNI match / hop-gw route at all for this destination -- it always takes the specific per-IP listener's single default filter chain straight to the real mock service, bypassing hop-gw completely, gateway pod present or not. This is precisely the failure mode the harness-notes CORRECTION warns about generally ("an in-cluster k8s Service is IN the registry and is a wrong stand-in" for exercising "external"/registry-boundary-dependent mechanisms) -- confirmed here to apply not just to REGISTRY_ONLY outboundTrafficPolicy semantics but to this Gateway(PASSTHROUGH)+ServiceEntry+VS mesh-hijack mechanism as well, whenever the ServiceEntry host happens to resolve to a real, already-registered k8s Service ClusterIP. Verdict: fail (not blocked -- this is a reproducible, root-caused mechanism finding, not infra flakiness or unresolved environment mismatch).