=== CMD: kubectl create namespace istio-vt-t65 === namespace/istio-vt-t65 created === CMD: kubectl label namespace istio-vt-t65 istio-injection=enabled === namespace/istio-vt-t65 labeled === CMD: kubectl apply -f manifest.yaml && wait client && wait chain-echo-v1 === pod/client created deployment.apps/chain-echo-v1 created service/chain-echo created destinationrule.networking.istio.io/chain-echo-dr created virtualservice.networking.istio.io/chain-echo-vs-nc created pod/client condition met deployment.apps/chain-echo-v1 condition met === CMD: sleep 5 (allow config propagation) === done === CMD: baseline curl v1 === baseline_v1=503 === DEBUG: baseline unexpected 503, investigating === * Host chain-echo.istio-vt-t65.svc.homelab.local:80 was resolved. * IPv6: (none) * IPv4: 10.250.174.73 * Trying 10.250.174.73:80... * Connected to chain-echo.istio-vt-t65.svc.homelab.local (10.250.174.73) port 80 * using HTTP/1.x > GET / HTTP/1.1 > Host: chain-echo.istio-vt-t65.svc.homelab.local > User-Agent: curl/8.14.1 > Accept: */* > * Request completely sent off < HTTP/1.1 503 Service Unavailable < date: Sun, 05 Jul 2026 00:07:22 GMT < server: envoy < content-length: 0 < * Connection #0 to host chain-echo.istio-vt-t65.svc.homelab.local left intact === DEBUG: client istio-proxy access log === 2026-07-05T00:06:51.673604Z info cache generated new workload certificate resourceName=default latency=87.240039ms ttl=23h59m59.32639781s 2026-07-05T00:06:51.673674Z info cache Root cert has changed, start rotating root cert 2026-07-05T00:06:51.673722Z info cache returned workload trust anchor from cache ttl=23h59m59.326279247s 2026-07-05T00:06:51.673773Z info cache returned workload trust anchor from cache ttl=23h59m59.32622743s 2026-07-05T00:06:51.673923Z info cache returned workload certificate from cache ttl=23h59m59.326078871s 2026-07-05T00:06:51.674209Z info cache returned workload trust anchor from cache ttl=23h59m59.325791552s 2026-07-05T00:06:52.863964Z info Readiness succeeded in 1.488257432s 2026-07-05T00:06:52.864327Z info Envoy proxy is ready [2026-07-05T00:07:17.337Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "a1280fd9-0c59-4c44-af77-53ed8c563fee" "chain-echo.istio-vt-t65.svc.homelab.local" "-" - - 10.250.174.73:80 10.255.126.38:57442 - - [2026-07-05T00:07:22.725Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "484b0587-fd43-44a0-8e84-19bb86acede7" "chain-echo.istio-vt-t65.svc.homelab.local" "-" - - 10.250.174.73:80 10.255.126.38:53422 - - === DEBUG: check clusters on client related to chain-echo === chain-echo.istio-vt-t65.svc.cluster.local 80 - outbound EDS --- apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"networking.istio.io/v1","kind":"DestinationRule","metadata":{"annotations":{},"name":"chain-echo-dr","namespace":"istio-vt-t65"},"spec":{"host":"chain-echo.istio-vt-t65.svc.homelab.local","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v3-does-not-exist"},"name":"v3-mismatched-selector"}]}} creationTimestamp: "2026-07-05T00:06:49Z" generation: 1 name: chain-echo-dr namespace: istio-vt-t65 resourceVersion: "13666085" uid: 87392f13-541a-46ef-be9e-a8832efb22bd spec: host: chain-echo.istio-vt-t65.svc.homelab.local subsets: - labels: version: v1 name: v1 - labels: version: v3-does-not-exist name: v3-mismatched-selector === DIAGNOSTIC: baseline_v1 unexpectedly returned 503/NC (not 200) === Root cause investigation: - istioctl proxy-config cluster showed the plain service cluster registered as "chain-echo.istio-vt-t65.svc.CLUSTER.LOCAL" (EDS), NOT svc.homelab.local. - kubectl -n istio-system get cm istio -o jsonpath='{.data.mesh}' -> trustDomain: cluster.local - kubectl -n istio-system get cm istio-sidecar-injector values -> global.proxy.clusterDomain: cluster.local - kubectl -n kube-system get cm kubelet-config -> clusterDomain: homelab.local (real k8s DNS domain) - coredns Corefile -> "kubernetes homelab.local ..." (confirms real DNS domain) - getent hosts chain-echo...svc.cluster.local from client pod -> NXDOMAIN (does not resolve) - getent hosts chain-echo...svc.homelab.local -> resolves fine to ClusterIP - Cross-check: pre-existing namespace mesh-test (untouched, real working fixture) uses DestinationRule host = *.svc.cluster.local for in-mesh services (e.g. egw-mtls-dr: host=egw-mtls.egress-mtls.svc.cluster.local) confirming this cluster's convention: Istio's registered service hostname (used for DestinationRule host / VS destination.host matching, subset attachment, and CDS cluster naming) is governed by proxy.clusterDomain=cluster.local, NOT by kubeadm's actual clusterDomain=homelab.local used for CoreDNS/DNS resolution. These are three DIFFERENT concepts: (1) k8s DNS domain=homelab.local, (2) istio trustDomain=cluster.local (SPIFFE), (3) istio proxy.clusterDomain=cluster.local (service registry hostnames) -- (2) and (3) happen to share the same default value but are independent settings; this install never reconfigured (3) to match the cluster's actual (1). CONCLUSION: harness-notes.md's blanket guidance ("all manifests must use *.svc.homelab.local FQDNs") is correct for literal client-facing request URLs/Host headers (which resolve via real DNS), but does NOT apply to DestinationRule.spec.host / VirtualService destination.host fields, which must match Istio's internal service registry hostname (svc.cluster.local on this install) for subset attachment / CDS cluster compilation to occur at all. Using svc.homelab.local in DestinationRule host + VS destination.host caused ISTIO TO NEVER COMPILE ANY SUBSET CLUSTER (v1 included) -- hence baseline_v1 itself returned NC, which is a manifest/environment artifact unrelated to the doc claims under test (NC vs UH distinguishing route->CDS-miss vs cluster->EDS-empty). FIX APPLIED: keep VirtualService `hosts:` field = *.svc.homelab.local (must match real curl Host header for RDS domain match to succeed), but change DestinationRule `host:` and all VirtualService route destination.host fields to *.svc.cluster.local (matching Istio's actual registered service hostname on this install), matching the proven-working convention already used in namespace mesh-test. Manifest updated accordingly; re-applying and re-running below. === CMD (re-run, corrected manifest): kubectl apply -f manifest.yaml === pod/client configured deployment.apps/chain-echo-v1 unchanged service/chain-echo unchanged destinationrule.networking.istio.io/chain-echo-dr configured virtualservice.networking.istio.io/chain-echo-vs-nc configured pod/client condition met deployment.apps/chain-echo-v1 condition met === CMD: sleep 5 (allow config propagation to sidecar after DR/VS update) === done === CMD: baseline curl v1 (corrected) === baseline_v1=200 === CMD: nc_case curl (x-target: nc) === nc_case=503 === CMD: check client istio-proxy access log for x-target: nc request === (note: access log format does not echo request headers; using tail of recent log lines instead below) [2026-07-05T00:11:13.314Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 11 11 "-" "curl/8.14.1" "a04db173-79ef-40f3-b5fe-7be8a93d69f7" "chain-echo.istio-vt-t65.svc.homelab.local" "10.255.159.132:5678" outbound|80|v1|chain-echo.istio-vt-t65.svc.cluster.local 10.255.126.38:59712 10.250.174.73:80 10.255.126.38:45788 - - === full recent istio-proxy log (last 30s) to capture nc_case entry === [2026-07-05T00:11:13.314Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 6 11 11 "-" "curl/8.14.1" "a04db173-79ef-40f3-b5fe-7be8a93d69f7" "chain-echo.istio-vt-t65.svc.homelab.local" "10.255.159.132:5678" outbound|80|v1|chain-echo.istio-vt-t65.svc.cluster.local 10.255.126.38:59712 10.250.174.73:80 10.255.126.38:45788 - - [2026-07-05T00:11:19.931Z] "GET / HTTP/1.1" 503 NC cluster_not_found - "-" 0 0 0 - "-" "curl/8.14.1" "4ba0e0e3-3491-4be9-b3e7-d658c0d611a8" "chain-echo.istio-vt-t65.svc.homelab.local" "-" - - 10.250.174.73:80 10.255.126.38:45794 - - === CMD: istioctl proxy-config cluster - check v2-undefined-in-dr count (expect 0) === 0 === CMD: uh_case curl (x-target: uh) === uh_case=503 === CMD: istioctl proxy-config endpoint - v3-mismatched-selector cluster (expect empty, cluster exists 0 endpoints) === ENDPOINT STATUS OUTLIER CHECK CLUSTER === also confirm the cluster ITSELF exists in CDS (distinct from endpoint emptiness) === [ { "transportSocketMatches": [ { "name": "tlsMode-istio", "match": { "tlsMode": "istio" }, "transportSocket": { "name": "envoy.transport_sockets.tls", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext", "commonTlsContext": { "tlsParams": { "tlsMinimumProtocolVersion": "TLSv1_2", "tlsMaximumProtocolVersion": "TLSv1_3" }, "tlsCertificateSdsSecretConfigs": [ { "name": "default", "sdsConfig": { "apiConfigSource": { "apiType": "GRPC", "transportApiVersion": "V3", "grpcServices": [ { "envoyGrpc": { "clusterName": "sds-grpc" } } ], "setNodeOnFirstMessageOnly": true }, "initialFetchTimeout": "0s", "resourceApiVersion": "V3" } } ], "combinedValidationContext": { "defaultValidationContext": { "matchSubjectAltNames": [ { "exact": "spiffe://cluster.local/ns/istio-vt-t65/sa/default" } ] }, "validationContextSdsSecretConfig": { "name": "ROOTCA", "sdsConfig": { "apiConfigSource": { "apiType": "GRPC", "transportApiVersion": "V3", "grpcServices": [ { "envoyGrpc": { "clusterName": "sds-grpc" } } ], "setNodeOnFirstMessageOnly": true }, "initialFetchTimeout": "0s", "resourceApiVersion": "V3" } } }, "alpnProtocols": [ "istio-peer-exchange", "istio" ] }, "sni": "outbound_.80_.v3-mismatched-selector_.chain-echo.istio-vt-t65.svc.cluster.local" } } }, { "name": "tlsMode-disabled", "match": {}, "transportSocket": { "name": "envoy.transport_sockets.raw_buffer", "typedConfig": { "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer" } } } ], "name": "outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local", "altStatName": "outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local;", "type": "EDS", "edsClusterConfig": { "edsConfig": { "ads": {}, "initialFetchTimeout": "0s", "resourceApiVersion": "V3" }, "serviceName": "outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local" }, "connectTimeout": "10s", "lbPolicy": "LEAST_REQUEST", "circuitBreakers": { "thresholds": [ { "maxConnections": 4294967295, "maxPendingRequests": 4294967295, "maxRequests": 4294967295, "maxRetries": 4294967295 } ] }, "commonLbConfig": {}, "metadata": { "filterMetadata": { "istio": { "config": "/apis/networking.istio.io/v1/namespaces/istio-vt-t65/destination-rule/chain-echo-dr", "services": [ { "host": "chain-echo.istio-vt-t65.svc.cluster.local", "name": "chain-echo", "namespace": "istio-vt-t65" } ], "subset": "v3-mismatched-selector" } } }, "filters": [ { "name": "istio.metadata_exchange", "typedConfig": { "@type": "type.googleapis.com/udpa.type.v1.TypedStruct", "typeUrl": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange", "value": { "enable_discovery": true, "protocol": "istio-peer-exchange" } } } ] } ] === CMD: tail istio-proxy log for uh_case entry === [2026-07-05T00:11:34.875Z] "GET / HTTP/1.1" 503 UH no_healthy_upstream - "-" 0 19 0 - "-" "curl/8.14.1" "84e93308-529a-4dd5-b5ff-8195c99f219a" "chain-echo.istio-vt-t65.svc.homelab.local" "-" outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local - 10.250.174.73:80 10.255.126.38:60076 - - === SUMMARY OF OBSERVED RESULTS === baseline_v1 = 200 (access log: 200 - via_upstream, cluster=outbound|80|v1|chain-echo.istio-vt-t65.svc.cluster.local) nc_case = 503 (access log: 503 NC cluster_not_found, upstream_cluster field = "-") -> istioctl proxy-config cluster count of any cluster containing "v2-undefined-in-dr" = 0 (the cluster was NEVER compiled into CDS at all, because no DestinationRule subset named v2-undefined-in-dr exists) uh_case = 503 (access log: 503 UH no_healthy_upstream, upstream_cluster field = outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local) -> istioctl proxy-config cluster confirms the cluster outbound|80|v3-mismatched-selector|chain-echo.istio-vt-t65.svc.cluster.local EXISTS in CDS (type=EDS, subset metadata present, DestinationRule reference resolved) but istioctl proxy-config endpoint --cluster returned an EMPTY table (header row only, 0 endpoint rows) because the subset's label selector (version=v3-does-not-exist) matches no real Pod. CONCLUSION: NC (cluster_not_found) and UH (no_healthy_upstream) are clearly two distinct, separately observable failure points in the Listener->Route->Cluster->Endpoint chain: - NC occurs when the route references a cluster name that was never compiled into CDS at all (route points to a subset that doesn't exist in any DestinationRule) -- confirmed by cluster count=0 for the referenced subset name. - UH occurs when the cluster DOES exist in CDS (subset is defined in DestinationRule) but EDS resolves 0 healthy endpoints for it (label selector matches no live Pod) -- confirmed by cluster existing in CDS dump while proxy-config endpoint returns empty. This matches pass_criteria exactly.