=== $ kubectl apply -f manifest.yaml === pod/client created deployment.apps/killobs-single created service/killobs-single created deployment.apps/killobs-double created service/killobs-double created === $ kubectl -n istio-vt-t33 wait --for=condition=available deploy/killobs-single deploy/killobs-double --timeout=90s === deployment.apps/killobs-single condition met deployment.apps/killobs-double condition met === $ kubectl -n istio-vt-t33 wait --for=condition=Ready pod -l app=client --timeout=120s === pod/client condition met === $ kubectl -n istio-vt-t33 get pods -o wide === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 10s 10.255.194.102 k8s-worker1 killobs-double-7f75f6b8cc-2cszq 2/2 Running 0 10s 10.255.194.103 k8s-worker1 killobs-double-7f75f6b8cc-cg82p 2/2 Running 0 10s 10.255.126.47 k8s-worker2 killobs-single-bdf9c88cd-df5rk 2/2 Running 0 10s 10.255.126.50 k8s-worker2 === CASE 1: HTTP/1.1 long request (10s delay) vs forceful pod kill === $ ( kubectl -n istio-vt-t33 exec client -c curl -- curl -s -o /dev/null -w 'h1_result=%{http_code} exit_marker=OK ' --max-time 15 http://killobs-single.istio-vt-t33.svc.homelab.local/delay/10 > /tmp/killobs_h1.log 2>&1; echo "h1_curl_exit=$?" >> /tmp/killobs_h1.log & ) $ sleep 2 $ POD1=$(kubectl -n istio-vt-t33 get pod -l app=killobs-single -o jsonpath='{.items[0].metadata.name}'); kubectl -n istio-vt-t33 delete pod $POD1 --grace-period=0 --force POD1=killobs-single-bdf9c88cd-df5rk pod "killobs-single-bdf9c88cd-df5rk" force deleted from istio-vt-t33 namespace $ sleep 12 $ cat /tmp/killobs_h1.log h1_result=200 exit_marker=OK h1_curl_exit=0 === DIAGNOSTIC: unexpected clean 200 in Case 1 -- investigating whether old pod process survived force-delete === $ kubectl -n istio-vt-t33 get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES client 2/2 Running 0 2m39s 10.255.194.102 k8s-worker1 killobs-double-7f75f6b8cc-2cszq 2/2 Running 0 2m39s 10.255.194.103 k8s-worker1 killobs-double-7f75f6b8cc-cg82p 2/2 Running 0 2m39s 10.255.126.47 k8s-worker2 killobs-single-bdf9c88cd-cbl7n 2/2 Running 0 116s 10.255.126.51 k8s-worker2 $ kubectl -n istio-vt-t33 get events --sort-by=.lastTimestamp | tail -40 2m38s Normal Pulled pod/killobs-double-7f75f6b8cc-cg82p Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m38s Normal Created pod/killobs-double-7f75f6b8cc-cg82p Created container istio-init 2m38s Normal Started pod/client Started container curl 2m38s Normal Pulling pod/killobs-double-7f75f6b8cc-cg82p Pulling image "kennethreitz/httpbin" 2m38s Normal Pulled pod/killobs-double-7f75f6b8cc-2cszq Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m38s Normal Pulled pod/killobs-single-bdf9c88cd-df5rk Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m38s Normal Created pod/client Created container curl 2m38s Normal Pulled pod/client Container image "curlimages/curl:8.14.1" already present on machine 2m37s Normal Started pod/client Started container istio-proxy 2m37s Normal Created pod/killobs-double-7f75f6b8cc-2cszq Created container httpbin 2m37s Normal Pulled pod/killobs-double-7f75f6b8cc-2cszq Successfully pulled image "kennethreitz/httpbin" in 1.073s (1.073s including waiting). Image size: 214174220 bytes. 2m36s Normal Created pod/killobs-double-7f75f6b8cc-cg82p Created container istio-proxy 2m36s Normal Started pod/killobs-double-7f75f6b8cc-cg82p Started container httpbin 2m36s Normal Pulled pod/killobs-double-7f75f6b8cc-cg82p Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m36s Normal Started pod/killobs-double-7f75f6b8cc-2cszq Started container httpbin 2m36s Normal Pulled pod/killobs-double-7f75f6b8cc-2cszq Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m36s Normal Started pod/killobs-double-7f75f6b8cc-cg82p Started container istio-proxy 2m36s Normal Created pod/killobs-double-7f75f6b8cc-2cszq Created container istio-proxy 2m36s Normal Started pod/killobs-double-7f75f6b8cc-2cszq Started container istio-proxy 2m36s Normal Pulled pod/killobs-double-7f75f6b8cc-cg82p Successfully pulled image "kennethreitz/httpbin" in 1.136s (1.136s including waiting). Image size: 214174220 bytes. 2m36s Normal Created pod/killobs-double-7f75f6b8cc-cg82p Created container httpbin 2m35s Normal Created pod/killobs-single-bdf9c88cd-df5rk Created container istio-proxy 2m35s Normal Started pod/killobs-single-bdf9c88cd-df5rk Started container istio-proxy 2m35s Normal Pulled pod/killobs-single-bdf9c88cd-df5rk Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2m35s Normal Started pod/killobs-single-bdf9c88cd-df5rk Started container httpbin 2m35s Normal Created pod/killobs-single-bdf9c88cd-df5rk Created container httpbin 2m35s Normal Pulled pod/killobs-single-bdf9c88cd-df5rk Successfully pulled image "kennethreitz/httpbin" in 1.098s (2.23s including waiting). Image size: 214174220 bytes. 116s Normal Killing pod/killobs-single-bdf9c88cd-df5rk Stopping container httpbin 116s Normal SuccessfulCreate replicaset/killobs-single-bdf9c88cd Created pod: killobs-single-bdf9c88cd-cbl7n 116s Normal Killing pod/killobs-single-bdf9c88cd-df5rk Stopping container istio-proxy 115s Normal Started pod/killobs-single-bdf9c88cd-cbl7n Started container istio-init 115s Normal Created pod/killobs-single-bdf9c88cd-cbl7n Created container istio-init 115s Normal Pulled pod/killobs-single-bdf9c88cd-cbl7n Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 114s Normal Pulling pod/killobs-single-bdf9c88cd-cbl7n Pulling image "kennethreitz/httpbin" 113s Normal Pulled pod/killobs-single-bdf9c88cd-cbl7n Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 113s Normal Started pod/killobs-single-bdf9c88cd-cbl7n Started container istio-proxy 113s Normal Pulled pod/killobs-single-bdf9c88cd-cbl7n Successfully pulled image "kennethreitz/httpbin" in 1.122s (1.122s including waiting). Image size: 214174220 bytes. 113s Normal Created pod/killobs-single-bdf9c88cd-cbl7n Created container istio-proxy 113s Normal Started pod/killobs-single-bdf9c88cd-cbl7n Started container httpbin 113s Normal Created pod/killobs-single-bdf9c88cd-cbl7n Created container httpbin $ kubectl -n istio-vt-t33 get events -o custom-columns=TIME:.lastTimestamp,REASON:.reason,OBJ:.involvedObject.name,MSG:.message --sort-by=.lastTimestamp | grep -i killobs-single Scheduled killobs-single-bdf9c88cd-cbl7n Successfully assigned istio-vt-t33/killobs-single-bdf9c88cd-cbl7n to k8s-worker2 Scheduled killobs-single-bdf9c88cd-df5rk Successfully assigned istio-vt-t33/killobs-single-bdf9c88cd-df5rk to k8s-worker2 2026-07-04T23:24:02Z SuccessfulCreate killobs-single-bdf9c88cd Created pod: killobs-single-bdf9c88cd-df5rk 2026-07-04T23:24:02Z ScalingReplicaSet killobs-single Scaled up replica set killobs-single-bdf9c88cd to 1 2026-07-04T23:24:03Z Created killobs-single-bdf9c88cd-df5rk Created container istio-init 2026-07-04T23:24:03Z Pulling killobs-single-bdf9c88cd-df5rk Pulling image "kennethreitz/httpbin" 2026-07-04T23:24:03Z Started killobs-single-bdf9c88cd-df5rk Started container istio-init 2026-07-04T23:24:03Z Pulled killobs-single-bdf9c88cd-df5rk Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2026-07-04T23:24:06Z Created killobs-single-bdf9c88cd-df5rk Created container istio-proxy 2026-07-04T23:24:06Z Started killobs-single-bdf9c88cd-df5rk Started container istio-proxy 2026-07-04T23:24:06Z Pulled killobs-single-bdf9c88cd-df5rk Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2026-07-04T23:24:06Z Started killobs-single-bdf9c88cd-df5rk Started container httpbin 2026-07-04T23:24:06Z Created killobs-single-bdf9c88cd-df5rk Created container httpbin 2026-07-04T23:24:06Z Pulled killobs-single-bdf9c88cd-df5rk Successfully pulled image "kennethreitz/httpbin" in 1.098s (2.23s including waiting). Image size: 214174220 bytes. 2026-07-04T23:24:45Z Killing killobs-single-bdf9c88cd-df5rk Stopping container httpbin 2026-07-04T23:24:45Z SuccessfulCreate killobs-single-bdf9c88cd Created pod: killobs-single-bdf9c88cd-cbl7n 2026-07-04T23:24:45Z Killing killobs-single-bdf9c88cd-df5rk Stopping container istio-proxy 2026-07-04T23:24:46Z Started killobs-single-bdf9c88cd-cbl7n Started container istio-init 2026-07-04T23:24:46Z Created killobs-single-bdf9c88cd-cbl7n Created container istio-init 2026-07-04T23:24:46Z Pulled killobs-single-bdf9c88cd-cbl7n Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2026-07-04T23:24:47Z Pulling killobs-single-bdf9c88cd-cbl7n Pulling image "kennethreitz/httpbin" 2026-07-04T23:24:48Z Pulled killobs-single-bdf9c88cd-cbl7n Container image "registry.istio.io/release/proxyv2:1.30.0" already present on machine 2026-07-04T23:24:48Z Started killobs-single-bdf9c88cd-cbl7n Started container istio-proxy 2026-07-04T23:24:48Z Pulled killobs-single-bdf9c88cd-cbl7n Successfully pulled image "kennethreitz/httpbin" in 1.122s (1.122s including waiting). Image size: 214174220 bytes. 2026-07-04T23:24:48Z Created killobs-single-bdf9c88cd-cbl7n Created container istio-proxy 2026-07-04T23:24:48Z Started killobs-single-bdf9c88cd-cbl7n Started container httpbin 2026-07-04T23:24:48Z Created killobs-single-bdf9c88cd-cbl7n Created container httpbin === NOTE: first Case 1 attempt above was split across separate Bash tool calls -- inter-turn latency means the 'sleep 2' gap was not tightly controlled (real elapsed time between curl launch and pod kill is unknown and could exceed 10s). Re-running Case 1 as a single atomic script below with explicit timestamps for a trustworthy result. === $ kubectl -n istio-vt-t33 get pods -l app=killobs-single -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES killobs-single-bdf9c88cd-cbl7n 2/2 Running 0 3m43s 10.255.126.51 k8s-worker2 === CASE 1 (retry, atomic script, timestamped): HTTP/1.1 long request (10s delay) vs forceful pod kill === T0=23:28:37.899 kickoff T2=23:28:39.901 about to delete pod POD1=killobs-single-bdf9c88cd-cbl7n pod "killobs-single-bdf9c88cd-cbl7n" force deleted from istio-vt-t33 namespace Tkill=23:28:40.007 delete command returned Tend=23:28:52.998 script end === /tmp/killobs_h1.log === h1_result=000 exit_marker=OK command terminated with exit code 28 h1_curl_exit=28 Tresp=23:28:52.996 curl finished $ kubectl -n istio-vt-t33 wait --for=condition=available deploy/killobs-single --timeout=60s deployment.apps/killobs-single condition met NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES killobs-single-bdf9c88cd-2wc82 2/2 Running 0 37s 10.255.126.59 k8s-worker2 === CASE 2 (atomic script, timestamped): HTTP/2 (h2c) long-lived stream vs forceful pod kill === T0=23:29:25.753 kickoff T2=23:29:27.755 about to delete pod POD1B=killobs-single-bdf9c88cd-2wc82 pod "killobs-single-bdf9c88cd-2wc82" force deleted from istio-vt-t33 namespace Tkill=23:29:27.871 delete command returned Tend=23:29:27.872 script end === /tmp/killobs_h2.log === h2_result=200 exit_marker=OK h2_curl_exit=0 Tresp=23:29:25.855 curl finished === NOTE: Case 2 as originally specified (curl vs /stream/60) completed in ~0.1s (Tresp=23:29:25.855 vs T0=23:29:25.753) -- i.e. kennethreitz/httpbin's /stream/N endpoint emits all N chunks essentially instantly with no per-chunk delay, so it finished BEFORE the pod was even killed at T+2s. It is not actually a 'long-lived stream' and does not exercise the intended in-flight-kill condition. Substituting httpbin's /drip endpoint (purpose-built for a throttled, duration-based streaming response: /drip?duration=N&numbytes=N) to genuinely test a slow HTTP/2 stream interrupted mid-transfer. Verifying /drip exists and its timing first (no kill), then re-running the actual kill test. === deployment.apps/killobs-single condition met $ kubectl -n istio-vt-t33 exec client -c curl -- curl -s -o /dev/null --http2-prior-knowledge -w 'probe_result=%{http_code} time_total=%{time_total} ' --max-time 20 'http://killobs-single.istio-vt-t33.svc.homelab.local/drip?duration=10&numbytes=100' probe_result=000 time_total=20.002417 === /drip over HTTP/2 hung to full 20s timeout with no response at all (probe_result=000, exit=28) even without any kill -- testing whether /drip works over plain HTTP/1.1 to isolate whether this is an h2c-specific issue === $ kubectl -n istio-vt-t33 exec client -c curl -- curl -s -o /dev/null -w 'probe_result=%{http_code} time_total=%{time_total} ' --max-time 20 'http://killobs-single.istio-vt-t33.svc.homelab.local/drip?duration=5&numbytes=50' probe_result=200 time_total=4.931412 === isolating whether /drip+h2c always hangs or only for longer durations === $ curl --http2-prior-knowledge /drip?duration=1&numbytes=10 --max-time 8 probe_result=200 time_total=1.125160 $ curl --http2-prior-knowledge /get (baseline sanity, no streaming) --max-time 8 probe_result=000 time_total=8.002233 === h2c baseline appears intermittent -- repeating plain h2c /get 5x to check flakiness rate (unrelated to kill, just protocol viability) === --- attempt 1 --- probe_result=000 time_total=6.002786 --- attempt 2 --- probe_result=000 time_total=6.001750 --- attempt 3 --- probe_result=000 time_total=6.001495 --- attempt 4 --- probe_result=200 time_total=0.008618 --- attempt 5 --- probe_result=200 time_total=0.005917 === FINDING: plain h2c (--http2-prior-knowledge) requests through the Istio sidecar to this httpbin backend are INTERMITTENTLY hanging with NO response at all (probe_result=000, exit=28 timeout) even with the pod healthy and no kill involved -- 3/5 baseline /get attempts hung for the full timeout, 2/5 succeeded instantly. This is pre-existing h2c connection-establishment flakiness, independent of any pod kill, and will confound Case 2's H1-vs-H2 comparison. Recording this as a genuine finding, then attempting the actual kill test against /drip (a real throttled-streaming endpoint, unlike /stream/N which emits instantly) to see if the kill signal is distinguishable above this baseline noise. === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES killobs-single-bdf9c88cd-9fmkw 2/2 Running 0 3m 10.255.194.118 k8s-worker1 === CASE 2 (adapted, atomic script, timestamped): HTTP/2 (h2c prior-knowledge) throttled /drip stream vs forceful pod kill mid-transfer === T0=23:32:36.308 kickoff T2=23:32:38.311 about to delete pod POD1B=killobs-single-bdf9c88cd-9fmkw pod "killobs-single-bdf9c88cd-9fmkw" force deleted from istio-vt-t33 namespace Tkill=23:32:38.419 delete command returned Tend=23:32:51.405 script end === /tmp/killobs_h2.log === h2_result=200 exit_marker=OK command terminated with exit code 28 h2_curl_exit=28 Tresp=23:32:51.403 curl finished === CASE 3 (atomic script): replicas=2 masking -- forceful kill of ONE replica while a concurrent request loop runs against the Service === T0=23:34:59.021 kickoff T1=23:35:00.023 about to delete pod POD2=killobs-double-7f75f6b8cc-2cszq pod "killobs-double-7f75f6b8cc-2cszq" force deleted from istio-vt-t33 namespace Tkill=23:35:00.132 delete command returned Tend=23:35:12.102 loop finished === sort /tmp/killobs_double.log | uniq -c === 58 200 2 503