+ openssl req -x509 -newkey rsa:2048 -nodes -keyout /tmp/it.key -out /tmp/it.crt -days 3 -subj '/CN=echo.test.local' .......+.......+...+..+......+.......+...........+............+....+...+.....+..........+........+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+....+..................+..+...+.......+..............+.+...+......+............+..+.+...........+.+..+.+..+...................+.....+.........+....+..+..........+...+.....+.+..+...+.........+...+...+.+........+.......+.....+......+....+...........+...+......+....+..+...+....+...+...+......+...+.....+.+............+.....+..........+..+.+.....+......+..........+...+..+.+..+......+...............+..........+...............+.................+...+.+....................+.+.................+.............+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ...........+...+.........+...+.+.....+.+...+........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+......+...............+.......+...+..+....+........+.+..+....+...+......+.....+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+.....+......+.+......+..............+..........+...+..+.........+......+.+..+.....................+.+.........+.....+.+......+.......................+.+...........+....+..............+....+.....+...+............+...+.......+...+...+..+.......+...+...+......+..............+....+...+..+............+.........+......+...+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ----- + kubectl -n istio-vt-t09 create secret tls ingress-term-cert --cert=/tmp/it.crt --key=/tmp/it.key secret/ingress-term-cert created + kubectl apply -f manifest.yaml && kubectl -n istio-vt-t09 wait --for=condition=Ready pod/client --timeout=90s && kubectl -n istio-vt-t09 rollout status deploy/ingress-term-gw --timeout=90s pod/client created deployment.apps/echo created service/echo created serviceaccount/ingress-term-gw created deployment.apps/ingress-term-gw created service/ingress-term-gw created gateway.networking.istio.io/ingress-term-gw created virtualservice.networking.istio.io/ingress-term-route created pod/client condition met Waiting for deployment "ingress-term-gw" rollout to finish: 0 of 1 updated replicas are available... deployment "ingress-term-gw" successfully rolled out + kubectl -n istio-vt-t09 rollout status deploy/echo --timeout=90s deployment "echo" successfully rolled out + kubectl -n istio-vt-t09 get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 37s 10.255.194.125 k8s-worker1 echo-5767bbcc56-tflz4 2/2 Running 0 37s 10.255.126.21 k8s-worker2 ingress-term-gw-75b5cbb777-dkb2p 1/1 Running 0 37s 10.255.126.20 k8s-worker2 + GWIP=$(kubectl -n istio-vt-t09 get svc ingress-term-gw -o jsonpath='{.spec.clusterIP}'); echo GWIP=$GWIP GWIP=10.250.124.153 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=000 command terminated with exit code 35 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'sni_mismatch=%{http_code} ' --max-time 4 --connect-to nomatch.test.local:443:ingress-term-gw.istio-vt-t09.svc.homelab.local:9443 https://nomatch.test.local/ ; echo exit=$? sni_mismatch=000 command terminated with exit code 35 exit=35 === RBAC gotcha discovered: custom gateway SA lacked secrets read RBAC; added rbac-fix.yaml, restarted gw === + cat rbac-fix.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: ingress-term-gw-sds namespace: istio-vt-t09 rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "watch", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: ingress-term-gw-sds namespace: istio-vt-t09 subjects: - kind: ServiceAccount name: ingress-term-gw namespace: istio-vt-t09 roleRef: kind: Role name: ingress-term-gw-sds apiGroup: rbac.authorization.k8s.io + kubectl apply -f rbac-fix.yaml role.rbac.authorization.k8s.io/ingress-term-gw-sds unchanged rolebinding.rbac.authorization.k8s.io/ingress-term-gw-sds unchanged + kubectl -n istio-vt-t09 rollout restart deploy/ingress-term-gw deployment.apps/ingress-term-gw restarted + kubectl -n istio-vt-t09 rollout status deploy/ingress-term-gw --timeout=90s Waiting for deployment "ingress-term-gw" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "ingress-term-gw" rollout to finish: 1 old replicas are pending termination... deployment "ingress-term-gw" successfully rolled out + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=404 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'sni_mismatch=%{http_code} ' --max-time 4 --connect-to nomatch.test.local:443:ingress-term-gw.istio-vt-t09.svc.homelab.local:9443 https://nomatch.test.local/ ; echo exit=$? sni_mismatch=000 command terminated with exit code 35 exit=35 === SECOND FIX: VS destination host must use Istio's internal clusterDomain 'cluster.local' (global.proxy.clusterDomain), NOT the actual k8s clusterDomain 'homelab.local'. Confirmed via envoy cluster dump on the gateway (outbound|80||echo.istio-vt-t09.svc.cluster.local present, no *.svc.homelab.local entries). Patched VS destination.host accordingly. === + kubectl -n istio-vt-t09 patch virtualservice ingress-term-route --type=merge -p '{"spec":{"http":[{"match":[{"headers":{"host":{"exact":"echo.test.local"}}}],"route":[{"destination":{"host":"echo.istio-vt-t09.svc.cluster.local"}}]}]}}' virtualservice.networking.istio.io/ingress-term-route patched (no change) + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=404 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'sni_mismatch=%{http_code} ' --max-time 4 --connect-to nomatch.test.local:443:ingress-term-gw.istio-vt-t09.svc.homelab.local:9443 https://nomatch.test.local/ ; echo exit=$? sni_mismatch=000 command terminated with exit code 35 exit=35 === THIRD FIX: VS had a redundant http.match.headers.host.exact:"echo.test.local" sub-match. curl sends 'Host: echo.test.local:9443' (port included, since 9443 is non-default), which fails that EXACT header match (port mismatch), causing route_not_found even though the top-level VS 'hosts' domain match (which has ignore_port_in_host_matching=true) already succeeded. Removed the redundant header match, keeping only hosts-based (Gateway SNI/Host) routing, which is the standard/documented pattern. === + kubectl -n istio-vt-t09 patch virtualservice ingress-term-route --type=merge -p '{"spec":{"http":[{"route":[{"destination":{"host":"echo.istio-vt-t09.svc.cluster.local"}}]}]}}' virtualservice.networking.istio.io/ingress-term-route patched + kubectl -n istio-vt-t09 get virtualservice ingress-term-route -o jsonpath='{.spec}' {"gateways":["ingress-term-gw"],"hosts":["echo.test.local"],"http":[{"route":[{"destination":{"host":"echo.istio-vt-t09.svc.cluster.local"}}]}]} + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=503 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 === FOURTH FIX: VS route destination had no explicit port for the multi-port 'echo' Service (80,443). Istio's ambiguous-port fallback used the incoming gateway listener port (9443) as the target cluster port literally, producing a nonexistent cluster outbound|9443||echo...svc.cluster.local -> 503 (no healthy upstream). Fixed by pinning destination.port.number: 80 (plaintext HTTP backend, since TLS is terminated at the gateway). === + kubectl -n istio-vt-t09 patch virtualservice ingress-term-route --type=merge -p '{"spec":{"http":[{"route":[{"destination":{"host":"echo.istio-vt-t09.svc.cluster.local","port":{"number":80}}}]}]}}' virtualservice.networking.istio.io/ingress-term-route patched + kubectl -n istio-vt-t09 get virtualservice ingress-term-route -o jsonpath='{.spec}' {"gateways":["ingress-term-gw"],"hosts":["echo.test.local"],"http":[{"route":[{"destination":{"host":"echo.istio-vt-t09.svc.cluster.local","port":{"number":80}}}]}]} + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=200 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 === DIAGNOSTIC: spec's host_mismatch command couples SNI and Host header (both = nope.test.local via --resolve), so it actually tests SNI-mismatch-at-TLS-layer, not pure L7 Host-header mismatch. Adding a supplementary probe that keeps SNI=echo.test.local (registered, TLS terminates) but overrides only the HTTP Host header to isolate true L7 behavior. === + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_header_only_mismatch=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 -H 'Host: nope.test.local' https://echo.test.local:9443/ host_header_only_mismatch=404 === FINAL CLEAN RE-RUN (all spec commands + diagnostic, post-fixes) === GWIP=10.250.124.153 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_match=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 https://echo.test.local:9443/ host_match=200 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_mismatch=%{http_code} ' --resolve nope.test.local:9443:10.250.124.153 https://nope.test.local:9443/ host_mismatch=000 command terminated with exit code 35 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'host_header_only_mismatch=%{http_code} ' --resolve echo.test.local:9443:10.250.124.153 -H 'Host: nope.test.local' https://echo.test.local:9443/ host_header_only_mismatch=404 + kubectl -n istio-vt-t09 exec client -c curl -- curl -sk -o /dev/null -w 'sni_mismatch=%{http_code} ' --max-time 4 --connect-to nomatch.test.local:443:ingress-term-gw.istio-vt-t09.svc.homelab.local:9443 https://nomatch.test.local/ ; echo exit=$? sni_mismatch=000 command terminated with exit code 35 exit=35 === REPRODUCIBILITY VALIDATION: full clean run of run.sh from scratch === [1/6] create namespace istio-vt-t09 with istio-injection=enabled namespace/istio-vt-t09 created namespace/istio-vt-t09 labeled [2/6] generate self-signed cert for echo.test.local and create TLS secret .....+......+.+..+...+.......+..+..........+.........+......+...+...........+.+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+............+..+.........+...............................+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ..+....+.....+....+..+...+....+..+.+.........+.....+..........+...+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.+...........+....+...........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+.+..+....+......+.....+...+.......+.....+...+.........+.......+...+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ----- secret/ingress-term-cert created [3/6] apply manifest (client, echo, custom ingress gateway, Gateway/VS, SDS RBAC) pod/client created deployment.apps/echo created service/echo created serviceaccount/ingress-term-gw created deployment.apps/ingress-term-gw created service/ingress-term-gw created gateway.networking.istio.io/ingress-term-gw created virtualservice.networking.istio.io/ingress-term-route created role.rbac.authorization.k8s.io/ingress-term-gw-sds created rolebinding.rbac.authorization.k8s.io/ingress-term-gw-sds created [4/6] wait for workloads Ready pod/client condition met deployment "echo" successfully rolled out deployment "ingress-term-gw" successfully rolled out [5/6] observe: ingress SIMPLE-mode Host-header routing GWIP=10.250.88.232 -- host_match: registered Host header (SNI+Host both = echo.test.local) -- host_match=200 -- host_mismatch (spec-literal): unregistered SNI+Host both = nope.test.local -- (note: --resolve makes curl's SNI and Host header travel together, so this actually probes an SNI not present in the Gateway's hosts list, i.e. an L4/filter_chain_not_found rejection -- see host_header_only_mismatch below for the isolated pure-L7 Host-header-mismatch probe.) host_mismatch=000 command terminated with exit code 35 exit=35 -- host_header_only_mismatch: SNI=echo.test.local (registered, TLS terminates) but Host header overridden to nope.test.local -- host_header_only_mismatch=404 [6/6] observe: egress-style PASSTHROUGH SNI mismatch contrast sni_mismatch=000 command terminated with exit code 35 exit=35 done. [cleanup] deleting namespace istio-vt-t09 (background) and temp cert files