=== T90 double-mTLS egress verification — 2026-07-05T04:48:42Z ===
istio 1.30.0 / k8s 1.30.6; NOTE: istiod --domain homelab.local as of 2026-07-05 04:40Z (registry names now *.svc.homelab.local; injection webhook deactivated -> pods created via istioctl kube-inject)

--- [pre] partner sanity (out-of-band, sidecar passthrough to ClusterIP:443) ---
with client cert : 200 'dmtls-partner ok: verify=SUCCESS dn=CN=egress-client'
without cert     : 400 (nginx: No required SSL certificate was sent)

--- [probe] guide-style bare-number tcpKeepalive rejected by CRD validation ---
The DestinationRule "dmtls-egress-hop1" is invalid: 
* spec.subsets[0].trafficPolicy.connectionPool.tcp.tcpKeepalive.time: Invalid value: "integer": spec.subsets[0].trafficPolicy.connectionPool.tcp.tcpKeepalive.time in body must be of type string: "integer"
* spec.subsets[0].trafficPolicy.connectionPool.tcp.tcpKeepalive.interval: Invalid value: "integer": spec.subsets[0].trafficPolicy.connectionPool.tcp.tcpKeepalive.interval in body must be of type string: "integer"
* <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation

=== (a) POSITIVE: client curl http://api.partner.example/ (dummy IP 240.240.34.90) ===
* Added api.partner.example:80:240.240.34.90 to DNS cache
* Hostname api.partner.example was found in DNS cache
*   Trying 240.240.34.90:80...
* Connected to api.partner.example (240.240.34.90) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: api.partner.example
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure< HTTP/1.1 503 Service Unavailable
< content-length: 121
< content-type: text/plain
< date: Sun, 05 Jul 2026 04:48:49 GMT
< server: envoy
< x-envoy-upstream-service-time: 43
< 
{ [121 bytes data]
* Connection #0 to host api.partner.example left intact

=== (a-debug) first attempt 503: gateway log 'TLS_error: Secret_is_not_supplied_by_SDS' ===
istiod: 'proxy dmtls-egress-...-istio-vt-t90 attempted to access unauthorized certificates dmtls-partner-client-cred: default/istio-vt-t90 is not authorized to read secrets'
cause: dedicated gateway SA needs secrets get/watch/list Role (SubjectAccessReview-based SDS authz); applied gw-sds-rbac.yaml
RESOURCE NAME                                     TYPE           STATUS     VALID CERT     SERIAL NUMBER                                NOT AFTER                NOT BEFORE
default                                           Cert Chain     ACTIVE     true           8dab6f34f3dea5bcab20be9eb246f73d             2026-07-06T04:47:01Z     2026-07-05T04:45:01Z
kubernetes://dmtls-partner-client-cred            Cert Chain     ACTIVE     true           377f992390f9bab5c83a264ef92f5f3ec1be1482     2026-08-04T04:36:55Z     2026-07-05T04:36:55Z
ROOTCA                                            CA             ACTIVE     true           930b9447bc7281e05035800e55c0b465             2036-05-29T02:08:42Z     2026-06-01T02:08:42Z
kubernetes://dmtls-partner-client-cred-cacert     CA             ACTIVE     true           5f6eb4d007806fb51bafb60dcbf736ff28c1fc26     2026-08-04T04:36:54Z     2026-07-05T04:36:54Z

=== (a) POSITIVE retry after SDS RBAC fix — 2026-07-05T04:54:59Z ===
HTTP/1.1 200 OK
server: envoy
date: Sun, 05 Jul 2026 04:54:59 GMT
content-type: text/plain
content-length: 53
x-client-verify: SUCCESS
x-client-dn: CN=egress-client
x-envoy-upstream-service-time: 6

dmtls-partner ok: verify=SUCCESS dn=CN=egress-client

HTTP:200

=== (b) PATH PROOF ===
--- b1. gateway pod access log (hop actually traversed) ---
[2026-07-05T04:48:50.394Z] "GET / HTTP/1.1" 503 URX,UF upstream_reset_before_response_started{remote_connection_failure|TLS_error:_Secret_is_not_supplied_by_SDS} - "TLS_error:_Secret_is_not_supplied_by_SDS" 0 121 34 - "10.255.126.11" "curl/8.14.1" "82f875ab-f01a-4ef1-a14e-e5b5cf3d3332" "api.partner.example" "10.250.244.174:443" outbound|443||api.partner.example - 10.255.126.12:8443 10.255.126.11:35714 api.partner.example -
[2026-07-05T04:54:59.270Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 53 3 3 "10.255.126.11" "curl/8.14.1" "4813557f-b36b-4a7f-9488-1fe5c6637f5f" "api.partner.example" "10.250.244.174:443" outbound|443||api.partner.example 10.255.126.12:36712 10.255.126.12:8443 10.255.126.11:60184 api.partner.example -

--- b2. client sidecar access log (hop1 cluster chosen) ---
[2026-07-05T04:54:59.269Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 53 7 6 "-" "curl/8.14.1" "d3ef6cdf-60ac-4cb0-a5dc-61c220542d27" "api.partner.example" "10.255.126.12:8443" outbound|8443|partner|dmtls-egress.istio-vt-t90.svc.homelab.local 10.255.126.11:60184 240.240.34.90:80 10.255.126.11:60332 - -

--- b3. nginx (partner) access log: TLS peer = gateway pod IP, verify result ---
2026/07/05 04:48:50 [info] 7#7: *3 client closed connection while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
2026/07/05 04:48:50 [info] 7#7: *4 client closed connection while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
2026/07/05 04:48:50 [info] 7#7: *5 client closed connection while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
10.255.126.12 "GET / HTTP/1.1" 200 verify=SUCCESS dn="CN=egress-client"
gateway pod IP: 10.255.126.12

--- b4. CLIENT sidecar hop1 cluster: ISTIO_MUTUAL + sni + connectionPool compiled ---
{
  "name": "outbound|8443|partner|dmtls-egress.istio-vt-t90.svc.homelab.local",
  "dr": "/apis/networking.istio.io/v1/namespaces/istio-vt-t90/destination-rule/dmtls-egress-hop1",
  "maxConnections": 77,
  "connectTimeout": "3s",
  "tcpKeepalive": {
    "keepaliveProbes": 3,
    "keepaliveTime": 300,
    "keepaliveInterval": 30
  },
  "transportSocket_name": "envoy.transport_sockets.tls",
  "tls_sni": "api.partner.example",
  "tls_sds": [
    "default"
  ]
}

--- b5. GATEWAY hop2 cluster: MUTUAL(credentialName SDS) + connectionPool compiled ---
{
  "name": "outbound|443||api.partner.example",
  "dr": "/apis/networking.istio.io/v1/namespaces/istio-vt-t90/destination-rule/dmtls-partner-originate",
  "maxConnections": 83,
  "connectTimeout": "3s",
  "tcpKeepalive": {
    "keepaliveProbes": 3,
    "keepaliveTime": 300,
    "keepaliveInterval": 30
  },
  "tls_sni": "api.partner.example",
  "client_cert_sds": [
    "kubernetes://dmtls-partner-client-cred"
  ],
  "validation_sds": "kubernetes://dmtls-partner-client-cred-cacert",
  "san_match": []
}

--- b5-correction: SAN pin IS compiled (deprecated field name matchSubjectAltNames) ---
  defaultValidationContext.matchSubjectAltNames: [{"exact":"api.partner.example"}]

=== (c) NEGATIVE 1: no client cert => partner must reject — 2026-07-05T04:56:03Z ===
--- c1. DR-hop2 deleted entirely (gateway sends PLAINTEXT to tls port) ---
destinationrule.networking.istio.io "dmtls-partner-originate" deleted from istio-vt-t90 namespace
HTTP/1.1 400 Bad Request
server: envoy
date: Sun, 05 Jul 2026 04:56:10 GMT
content-type: text/html
content-length: 255
x-envoy-upstream-service-time: 3

<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.27.5</center>
</body>
</html>

HTTP:400
--- nginx log after c1 ---
2026/07/05 04:56:04 [info] 7#7: *6 client 10.255.126.12 closed keepalive connection
2026/07/05 04:56:10 [info] 7#7: *7 client sent plain HTTP request to HTTPS port while reading client request headers, client: 10.255.126.12, server: api.partner.example, request: "GET / HTTP/1.1", host: "api.partner.example"
10.255.126.12 "GET / HTTP/1.1" 400 verify=- dn="-"

--- c2. DR-hop2 replaced with SIMPLE (TLS, no client cert) ---
destinationrule.networking.istio.io/dmtls-partner-originate created
HTTP/1.1 503 Service Unavailable
content-length: 121
content-type: text/plain
date: Sun, 05 Jul 2026 04:56:16 GMT
server: envoy
x-envoy-upstream-service-time: 65

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure
HTTP:503
--- nginx error+access log after c2 ---
2026/07/05 04:56:16 [info] 7#7: *8 SSL_do_handshake() failed (SSL: error:0A000418:SSL routines::tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
2026/07/05 04:56:16 [info] 7#7: *9 SSL_do_handshake() failed (SSL: error:0A000418:SSL routines::tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
2026/07/05 04:56:16 [info] 7#7: *10 SSL_do_handshake() failed (SSL: error:0A000418:SSL routines::tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443

--- c3. DR-hop2 = SIMPLE + credentialName (server CA trusted, NO client cert presented) ---
destinationrule.networking.istio.io/dmtls-partner-originate configured
HTTP/1.1 400 Bad Request
server: envoy
date: Sun, 05 Jul 2026 04:56:47 GMT
content-type: text/html
content-length: 237
x-client-verify: NONE
x-envoy-upstream-service-time: 5

<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.27.5</center>
</body>
</html>

HTTP:400
--- nginx error+access log after c3 ---
2026/07/05 04:56:16 [info] 7#7: *10 SSL_do_handshake() failed (SSL: error:0A000418:SSL routines::tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking, client: 10.255.126.12, server: 0.0.0.0:8443
10.255.126.12 "GET / HTTP/1.1" 400 verify=NONE dn="-"
2026/07/05 04:56:47 [info] 7#7: *11 client sent no required SSL certificate while reading client request headers, client: 10.255.126.12, server: api.partner.example, request: "GET / HTTP/1.1", host: "api.partner.example"

--- restore check: full MUTUAL DR-hop2 re-applied ---
dmtls-partner ok: verify=SUCCESS dn=CN=egress-client
HTTP:200

=== (d) NEGATIVE 2 (guide pitfall 3): ALLOW_ANY + VS mesh route removed — 2026-07-05T04:57:29Z ===
meshConfig.outboundTrafficPolicy: unset => ALLOW_ANY
virtualservice.networking.istio.io/dmtls-partner-via-egress configured
HTTP/1.1 400 Bad Request
server: envoy
date: Sun, 05 Jul 2026 04:57:36 GMT
content-type: text/html
content-length: 255
x-envoy-upstream-service-time: 0

<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.27.5</center>
</body>
</html>

HTTP:400
--- client sidecar access log (which upstream cluster was chosen) ---
[2026-07-05T04:57:29.717Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 53 5 5 "-" "curl/8.14.1" "4d57a62b-338a-4ea1-b739-6cc04e8a7721" "api.partner.example" "10.255.126.12:8443" outbound|8443|partner|dmtls-egress.istio-vt-t90.svc.homelab.local 10.255.126.11:35714 240.240.34.90:80 10.255.126.11:60546 - -
--- gateway access log request count before=5 after=6 (unchanged = gateway NOT traversed) ---
--- nginx log (direct plaintext hit from CLIENT pod, not gateway) ---
2026/07/05 04:57:36 [info] 7#7: *13 client sent plain HTTP request to HTTPS port while reading client request headers, client: 10.255.126.11, server: api.partner.example, request: "GET / HTTP/1.1", host: "api.partner.example"
10.255.126.11 "GET / HTTP/1.1" 400 verify=- dn="-"
bypass-curl-2 HTTP:400
--- (d) clean re-capture ---
gateway 'GET /' access-log count: before=6 after=6  (UNCHANGED => gateway silently bypassed)
client sidecar log for bypass request (upstream cluster = SE cluster, NOT the gateway cluster):
[2026-07-05T04:57:36.056Z] "GET / HTTP/1.1" 400 - via_upstream - "-" 0 255 1 0 "-" "curl/8.14.1" "083b4c8e-a726-45b8-acf1-1cfc6a8888ae" "api.partner.example" "10.250.244.174:443" outbound|80||api.partner.example 10.255.126.11:43168 240.240.34.90:80 10.255.126.11:45166 - default
nginx peer for bypass request:
10.255.126.11 "GET / HTTP/1.1" 400 verify=- dn="-"
client pod IP: 10.255.126.11 / gateway pod IP: 10.255.126.12

=== final: full wiring restored — positive path re-confirmed ===
dmtls-partner ok: verify=SUCCESS dn=CN=egress-client
HTTP:200

=== environment notes ===
- istiod --domain homelab.local (changed 2026-07-05 04:40Z by concurrent operator): registry names are *.svc.homelab.local; DR-hop1/VS hosts adapted accordingly (empirically verified via proxy-config cluster)
- injection webhook was deactivated (istio.io/deactivated=never-match) during setup; pods created via istioctl kube-inject; webhook self-healed at ~04:52Z
- dedicated gateway (istio: dmtls-egress) via inject.istio.io/templates=gateway; shared istio-egressgateway untouched