# T96 result — 2026-07-09, Istio 1.30.0 / k8s 1.30.6 (homelab) # 실행 순서: setup.sh -> t96-backends-slow.yaml apply -> run.sh (run1) -> run.sh (run2, 재현성) -> 사후 라우팅 체크 -> cleanup.sh ==================== RUN 1 (t96-run.sh) ==================== == T96 == backend-a=10.250.197.197 backend-b=10.250.12.61 (기대 응답 크기 307200 bytes ≈ 30s) ================ VARIANT=strict ================ --- snap[S0 pre-request (gslb->A)] @ 08:48:16 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 1 >> slow GET 시작 @ 08:48:16 (완주 시 ~30s) --- snap[S1 mid-flight, pre-flip (+8s)] @ 08:48:24 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 1 .upstream_cx_active: 1 .upstream_cx_destroy: 0 .upstream_cx_destroy_local: 0 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 1 .upstream_rq_total: 1 >> FLIP A->B @ 08:48:24 --- snap[S2 post-flip (+12s; 절단이면 이미 죽음 / graceful이면 아직 전송 중)] @ 08:48:36 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 2 .upstream_cx_active: 1 .upstream_cx_destroy: 0 .upstream_cx_destroy_local: 0 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 1 .upstream_rq_total: 1 [curl 결과 — strict] @ 08:48:46 code=200 bytes=307200 time=30.043534s rc=0 x-backend: backend-a --- snap[S3 after] @ 08:48:48 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 2 .upstream_cx_active: 0 .upstream_cx_destroy: 1 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 1 .upstream_rq_total: 1 ================ VARIANT=logical ================ --- snap[S0 pre-request (gslb->A)] @ 08:48:57 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 3 .upstream_cx_active: 0 .upstream_cx_destroy: 1 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 1 .upstream_rq_total: 1 >> slow GET 시작 @ 08:48:57 (완주 시 ~30s) --- snap[S1 mid-flight, pre-flip (+8s)] @ 08:49:05 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 3 .upstream_cx_active: 1 .upstream_cx_destroy: 1 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 2 .upstream_rq_total: 2 >> FLIP A->B @ 08:49:05 --- snap[S2 post-flip (+12s; 절단이면 이미 죽음 / graceful이면 아직 전송 중)] @ 08:49:18 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 3 .upstream_cx_active: 1 .upstream_cx_destroy: 1 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 2 .upstream_rq_total: 2 [curl 결과 — logical] @ 08:49:27 code=200 bytes=307200 time=30.040531s rc=0 x-backend: backend-a --- snap[S3 after] @ 08:49:29 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 3 .upstream_cx_active: 1 .upstream_cx_destroy: 1 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 0 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 2 .upstream_rq_total: 2 == 판정 가이드 == 절단(가설 A): curl rc!=0(18=partial/56=reset), bytes<307200, destroy_with_active_rq 델타 +1 graceful(가설 B): curl rc=0, bytes=307200, time≈30s, destroy는 완주 후 with_active_rq 0 ==================== RUN 2 (재현성 확인) ==================== == T96 == backend-a=10.250.197.197 backend-b=10.250.12.61 (기대 응답 크기 307200 bytes ≈ 30s) ================ VARIANT=strict ================ --- snap[S0 pre-request (gslb->A)] @ 08:51:39 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 5 .upstream_cx_active: 0 .upstream_cx_destroy: 2 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 2 .upstream_rq_total: 2 >> slow GET 시작 @ 08:51:39 (완주 시 ~30s) --- snap[S1 mid-flight, pre-flip (+8s)] @ 08:51:47 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 5 .upstream_cx_active: 1 .upstream_cx_destroy: 2 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 3 .upstream_rq_total: 3 >> FLIP A->B @ 08:51:47 --- snap[S2 post-flip (+12s; 절단이면 이미 죽음 / graceful이면 아직 전송 중)] @ 08:51:59 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 6 .upstream_cx_active: 1 .upstream_cx_destroy: 2 .upstream_cx_destroy_local: 1 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 3 .upstream_rq_total: 3 [curl 결과 — strict] @ 08:52:09 code=200 bytes=307200 time=30.041137s rc=0 x-backend: backend-a --- snap[S3 after] @ 08:52:11 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 6 .upstream_cx_active: 0 .upstream_cx_destroy: 3 .upstream_cx_destroy_local: 2 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 3 .upstream_rq_total: 3 ================ VARIANT=logical ================ --- snap[S0 pre-request (gslb->A)] @ 08:52:20 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 7 .upstream_cx_active: 0 .upstream_cx_destroy: 3 .upstream_cx_destroy_local: 2 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 3 .upstream_rq_total: 3 >> slow GET 시작 @ 08:52:20 (완주 시 ~30s) --- snap[S1 mid-flight, pre-flip (+8s)] @ 08:52:28 dig=10.250.197.197 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.197.197:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 7 .upstream_cx_active: 1 .upstream_cx_destroy: 3 .upstream_cx_destroy_local: 2 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 4 .upstream_rq_total: 4 >> FLIP A->B @ 08:52:28 --- snap[S2 post-flip (+12s; 절단이면 이미 죽음 / graceful이면 아직 전송 중)] @ 08:52:40 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 7 .upstream_cx_active: 1 .upstream_cx_destroy: 3 .upstream_cx_destroy_local: 2 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 4 .upstream_rq_total: 4 [curl 결과 — logical] @ 08:52:50 code=200 bytes=307200 time=30.037515s rc=0 x-backend: backend-a --- snap[S3 after] @ 08:52:52 dig=10.250.12.61 [endpoints] ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.250.12.61:443 HEALTHY OK outbound|443||gslb.lab.internal [cluster|443 cx] .membership_change: 7 .upstream_cx_active: 1 .upstream_cx_destroy: 3 .upstream_cx_destroy_local: 2 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 1 .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 4 .upstream_rq_total: 4 == 판정 가이드 == 절단(가설 A): curl rc!=0(18=partial/56=reset), bytes<307200, destroy_with_active_rq 델타 +1 graceful(가설 B): curl rc=0, bytes=307200, time≈30s, destroy는 완주 후 with_active_rq 0 ==================== 사후 체크: flip 후 신규 요청의 행선지 ==================== == [logical, flip 후 대기 상태] 신규 짧은 요청 3발 — pinned 연결 재사용 기대(backend-a) backend-b backend-b backend-b == strict 로 교체 (dig는 이미 backend-b) 후 신규 요청 3발 — backend-b 기대 backend-b backend-b backend-b # 해석: logical의 backend-b는 예상(pin=backend-a)과 어긋남 -> 원인 확인: # run2 종료 ~수분 뒤 관측. nginx keepalive_timeout 65s 가 유휴 pinned 연결을 원격 종료. == stats (사후) == .upstream_cx_active: 0 .upstream_cx_destroy: 8 .upstream_cx_destroy_local: 4 .upstream_cx_destroy_local_with_active_rq: 0 .upstream_cx_destroy_remote: 4 <- 원격 종료 4건 (backend nginx keepalive_timeout) .upstream_cx_destroy_remote_with_active_rq: 0 .upstream_cx_destroy_with_active_rq: 0 .upstream_cx_total: 8 == backend nginx 기본값 == keepalive_timeout 65; # 결론: LOGICAL의 stale pin 수명 = 연결 수명. 어느 쪽이든(원격 유휴 종료 포함) 연결이 닫히면 # 다음 요청은 최신 DNS를 따른다.