=== CMD: kubectl apply -f manifest.yaml === pod/client created deployment.apps/echo created service/echo created serviceaccount/mutual-gw created deployment.apps/mutual-gw created service/mutual-gw created virtualservice.networking.istio.io/mutual-gw-route created destinationrule.networking.istio.io/mutual-gw-dr created pod/non-mesh-caller created Error from server: error when creating "manifest.yaml": admission webhook "validation.istio.io" denied the request: configuration is invalid: server cannot have TLS settings for non HTTPS/TLS ports === CMD (retry, protocol fix TCP->TLS + tcp:->tls: match, see notes): kubectl apply -f manifest.yaml === pod/client configured deployment.apps/echo unchanged service/echo unchanged serviceaccount/mutual-gw unchanged deployment.apps/mutual-gw unchanged service/mutual-gw unchanged gateway.networking.istio.io/mutual-gw created virtualservice.networking.istio.io/mutual-gw-route configured destinationrule.networking.istio.io/mutual-gw-dr unchanged pod/non-mesh-caller unchanged === CMD: kubectl -n istio-vt-t46 wait --for=condition=Ready pod/client pod/non-mesh-caller --timeout=90s === pod/client condition met pod/non-mesh-caller condition met === CMD: kubectl -n istio-vt-t46 rollout status deploy/mutual-gw --timeout=90s === deployment "mutual-gw" successfully rolled out === CMD: kubectl -n istio-vt-t46 rollout status deploy/echo --timeout=90s === deployment "echo" successfully rolled out === kubectl -n istio-vt-t46 get pods -o wide === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 2m21s 10.255.194.115 k8s-worker1 echo-5767bbcc56-bq62d 2/2 Running 0 2m21s 10.255.126.58 k8s-worker2 mutual-gw-7856884554-f65xv 1/1 Running 0 2m21s 10.255.194.116 k8s-worker1 non-mesh-caller 1/1 Running 0 2m21s 10.255.194.109 k8s-worker1 === CMD: istioctl proxy-config listener deploy/mutual-gw.istio-vt-t46 --port 9443 -o json | jq '.[0].filterChains[0].transportSocket.typedConfig.requireClientCertificate' === null === CMD (retry #2, revert VS to tcp: match, keep Gateway protocol TLS): kubectl apply -f manifest.yaml === pod/client configured deployment.apps/echo unchanged service/echo unchanged serviceaccount/mutual-gw unchanged deployment.apps/mutual-gw unchanged service/mutual-gw unchanged gateway.networking.istio.io/mutual-gw unchanged virtualservice.networking.istio.io/mutual-gw-route configured destinationrule.networking.istio.io/mutual-gw-dr unchanged pod/non-mesh-caller unchanged === istiod logs check for gateway warning === === CMD (final, listener requireClientCertificate check): istioctl proxy-config listener deploy/mutual-gw.istio-vt-t46 --port 9443 -o json | jq '.[0].filterChains[0].transportSocket.typedConfig.requireClientCertificate' === true === supplementary: istioctl analyze -n istio-vt-t46 (FQDN domain-suffix note) === Error [IST0101] (VirtualService istio-vt-t46/mutual-gw-route) Referenced host not found: "echo.istio-vt-t46.svc.homelab.local" Warning [IST0174] (DestinationRule istio-vt-t46/mutual-gw-dr) The host mutual-gw.istio-vt-t46.svc.homelab.local defined in the DestinationRule does not match any services in the mesh. Info [IST0135] (Pod istio-vt-t46/client) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Info [IST0135] (Pod istio-vt-t46/echo-5767bbcc56-bq62d) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Info [IST0135] (Pod istio-vt-t46/mutual-gw-7856884554-f65xv) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Info [IST0135] (Pod istio-vt-t46/non-mesh-caller) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions. Error: Analyzers found issues when analyzing namespace: istio-vt-t46. See https://istio.io/v1.30/docs/reference/config/analysis for more information about causes and resolutions. === CMD: kubectl -n istio-vt-t46 exec client -c curl -- curl -sS -o /dev/null -w 'mesh_client=%{http_code} ' http://mutual-gw.istio-vt-t46.svc.homelab.local:9443/ === curl: (52) Empty reply from server mesh_client=000 command terminated with exit code 52 mesh_client_exit=52 === DIAGNOSTIC (not spec command; confirming domain-suffix hypothesis): same request but host = mutual-gw.istio-vt-t46.svc.cluster.local === curl: (52) Empty reply from servmesh_client_clusterlocal=000 er command terminated with exit code 52 diag_exit=52 === ENV ISSUE DIAGNOSED & FIX APPLIED === Root cause: this cluster's real DNS domain is homelab.local (CoreDNS/kubeadm), but istiod's own k8s-Service-registry hostname suffix is still 'cluster.local' (global.proxy.clusterDomain was not set to homelab.local at Istio install time). Confirmed via: (1) istioctl analyze reported 'Referenced host not found: echo.istio-vt-t46.svc.homelab.local' and DR host mismatch warning; (2) istioctl proxy-config clusters / CDS dump show the REAL registered cluster names use '*.svc.cluster.local' suffix, and NO cluster exists at all for the '*.svc.homelab.local' literal unless a ServiceEntry explicitly registers it (as seen from unrelated parallel tests T37/T47's ServiceEntries for mock.istio-verify-ext.svc.homelab.local, visible mesh-wide). Effect if left unfixed: DestinationRule mutual-gw-dr (host=*.svc.homelab.local) never binds to the real cluster -> client's outbound to mutual-gw never gets the ISTIO_MUTUAL trafficPolicy -> plain TCP (no TLS at all) is sent to a TLS-terminating listener -> 'Empty reply from server' (curl 52), which would misrepresent BOTH mesh-client and non-mesh-caller as identically failing -- unable to observe the actual claim under test (whether the listener distinguishes authenticated mesh clients from non-mesh callers). FIX: changed DestinationRule.host and the VirtualService's TCP route destination.host from *.svc.homelab.local to *.svc.cluster.local (Istio's real internal registry suffix in this cluster). curl target hosts in the actual test commands are left as *.svc.homelab.local, since those need to resolve via real CoreDNS (homelab.local zone), which works fine independent of istiod's registry. === CMD (retry #3, DR/VS host domain-suffix fix): kubectl apply -f manifest.yaml === pod/client configured deployment.apps/echo unchanged service/echo unchanged serviceaccount/mutual-gw unchanged deployment.apps/mutual-gw unchanged service/mutual-gw unchanged gateway.networking.istio.io/mutual-gw unchanged virtualservice.networking.istio.io/mutual-gw-route configured destinationrule.networking.istio.io/mutual-gw-dr configured pod/non-mesh-caller unchanged === CMD (retry, post domain-suffix fix): kubectl -n istio-vt-t46 exec client -c curl -- curl -sS -o /dev/null -w 'mesh_client=%{http_code} ' http://mutual-gw.istio-vt-t46.svc.homelab.local:9443/ === mesh_client=200 mesh_client_exit=0 === CMD: kubectl -n istio-vt-t46 exec non-mesh-caller -- curl -sk -o /dev/null -w 'non_mesh_https_attempt=%{http_code} ' --max-time 4 https://mutual-gw.istio-vt-t46.svc.homelab.local:9443/ ; echo non_mesh_exit=$? === non_mesh_https_attempt=000 command terminated with exit code 55 non_mesh_exit=55 === SUPPLEMENTARY (verbose, for diagnostic record only): non-mesh-caller curl -v === * [HTTP/2] [1] [:authority: mutual-gw.istio-vt-t46.svc.homelab.local:9443] * [HTTP/2] [1] [:path: /] * [HTTP/2] [1] [user-agent: curl/8.14.1] * [HTTP/2] [1] [accept: */*] } [5 bytes data] > GET / HTTP/2 > Host: mutual-gw.istio-vt-t46.svc.homelab.local:9443 > User-Agent: curl/8.14.1 > Accept: */* > * Request completely sent off { [5 bytes data] * TLSv1.3 (IN), TLS alert, unknown (628): { [2 bytes data] * OpenSSL SSL_read: OpenSSL/3.5.0: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0 * Failed receiving HTTP2 data: 56(Failure when receiving data from the peer) 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #0 to host mutual-gw.istio-vt-t46.svc.homelab.local left intact curl: (56) OpenSSL SSL_read: OpenSSL/3.5.0: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0 command terminated with exit code 56 === non-mesh-caller pod readiness/injection check (sanity: confirm NO sidecar) === curl === client pod readiness/injection check (sanity: confirm sidecar present) === curl istio-proxy