=== kubectl apply -f manifest.yaml === pod/client created deployment.apps/backend-a created service/backend-a created deployment.apps/backend-b created service/backend-b created configmap/haproxy-no-redispatch created deployment.apps/haproxy-test created service/haproxy-test created === kubectl -n istio-vt-t28 wait --for=condition=available deploy/backend-a deploy/backend-b deploy/haproxy-test --timeout=60s === deployment.apps/backend-a condition met deployment.apps/backend-b condition met deployment.apps/haproxy-test condition met === kubectl -n istio-vt-t28 wait --for=condition=Ready pod/client --timeout=120s === pod/client condition met === kubectl -n istio-vt-t28 get pods -o wide === NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES backend-a-79b85bb465-cxrqj 1/1 Running 0 15s 10.255.126.17 k8s-worker2 backend-b-6fc75fdbf-5fddz 1/1 Running 0 15s 10.255.126.18 k8s-worker2 client 2/2 Running 0 15s 10.255.194.78 k8s-worker1 haproxy-test-5cd777b849-j2rz7 1/1 Running 0 15s 10.255.126.19 k8s-worker2 === CMD1: sanity - roundrobin through haproxy-test (6 requests) === for i in $(seq 1 6); do kubectl -n istio-vt-t28 exec client -c curl -- curl -s http://haproxy-test.istio-vt-t28.svc.homelab.local:8080/; done no healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstreamno healthy upstream=== NOTE: manifest fixed - removed 'global / stats socket /run/haproxy/admin.sock' from ConfigMap (bind failure: dir doesn't exist in haproxy:2.8 image, caused CrashLoopBackOff, unrelated to redispatch test) === === kubectl apply -f manifest.yaml (retry with fixed config) === pod/client configured deployment.apps/backend-a unchanged service/backend-a unchanged deployment.apps/backend-b unchanged service/backend-b unchanged configmap/haproxy-no-redispatch configured deployment.apps/haproxy-test unchanged service/haproxy-test unchanged === rollout restart haproxy-test to pick up fixed configmap === deployment.apps/haproxy-test restarted Waiting for deployment "haproxy-test" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... deployment "haproxy-test" successfully rolled out NAME READY STATUS RESTARTS AGE backend-a-79b85bb465-cxrqj 1/1 Running 0 82s backend-b-6fc75fdbf-5fddz 1/1 Running 0 82s client 2/2 Running 0 82s haproxy-test-5cd777b849-j2rz7 0/1 Terminating 3 82s haproxy-test-67ccf549b4-92tff 1/1 Running 0 9s === CMD1 (retry): sanity - roundrobin through haproxy-test (6 requests) === for i in $(seq 1 6); do kubectl -n istio-vt-t28 exec client -c curl -- curl -s http://haproxy-test.istio-vt-t28.svc.homelab.local:8080/; done backend-a backend-b backend-a backend-b backend-a backend-b === CMD2: kubectl -n istio-vt-t28 scale deployment backend-a --replicas=0 === deployment.apps/backend-a scaled === CMD3: for i in $(seq 1 10); do kubectl -n istio-vt-t28 exec client -c curl -- curl -s -o /dev/null -w '%{http_code} ' http://haproxy-test.istio-vt-t28.svc.homelab.local:8080/; done === 200 200 200 200 200 200 200 200 200 200 === diag: haproxy-test logs (health check transitions) === [NOTICE] (1) : New worker (9) forked [NOTICE] (1) : Loading success. [WARNING] (9) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. === diag: backend-a deployment replicas === NAME READY UP-TO-DATE AVAILABLE AGE backend-a 0/0 0 0 2m3s === NOTE: first attempt at CMD3 missed the DOWN-detection window because each kubectl exec invocation incurs ~0.5-1s API-server round-trip overhead; by request #1 the fall=3/inter=1s health check had already marked backend-a DOWN (see log: 'Server be/a is DOWN' before our 10-request loop completed). Retrying with backend-a restored, then scaled to 0 again, using a SINGLE kubectl exec running an inline fast loop to catch the narrow window. === === restore backend-a to 1 replica and wait for it to be marked UP by haproxy === deployment.apps/backend-a scaled deployment.apps/backend-a condition met [NOTICE] (1) : New worker (9) forked [NOTICE] (1) : Loading success. [WARNING] (9) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] (9) : Server be/a is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. === CMD2 (retry): kubectl -n istio-vt-t28 scale deployment backend-a --replicas=0 === deployment.apps/backend-a scaled === CMD3 (retry, fast inline loop via single exec): for i in $(seq 1 30); do curl -s -o /dev/null -w '%{http_code} '; done === 200 503 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 === diag: haproxy-test logs after CMD3 retry (confirm DOWN timing vs when 503 was returned) === [NOTICE] (1) : New worker (9) forked [NOTICE] (1) : Loading success. [WARNING] (9) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] (9) : Server be/a is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. [WARNING] (9) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. === CMD4: create configmap haproxy-with-redispatch (no-redispatch config + 'option redispatch' after 'retries 3') === configmap/haproxy-with-redispatch created === verify new configmap content === defaults mode http timeout connect 2s timeout client 5s timeout server 5s retries 3 option redispatch frontend fe bind *:8080 default_backend be backend be balance roundrobin option httpchk GET / server a backend-a.istio-vt-t28.svc.homelab.local:5678 check inter 1s rise 1 fall 3 server b backend-b.istio-vt-t28.svc.homelab.local:5678 check inter 1s rise 1 fall 3=== CMD5: patch deployment haproxy-test to use haproxy-with-redispatch configmap === deployment.apps/haproxy-test patched === CMD6: kubectl -n istio-vt-t28 rollout status deploy/haproxy-test --timeout=60s === Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... === NOTE: CMD4's --from-literal capture via jsonpath strips the trailing newline, causing haproxy to fail with 'Missing LF on last line' -> CrashLoopBackOff. Retrying CMD4 with a trailing newline appended (environmental/tooling fix, config content otherwise identical incl 'option redispatch'). === configmap "haproxy-with-redispatch" deleted from istio-vt-t28 namespace configmap/haproxy-with-redispatch created === verify new configmap content (with trailing newline) === backend be$ balance roundrobin$ option httpchk GET /$ server a backend-a.istio-vt-t28.svc.homelab.local:5678 check inter 1s rise 1 fall 3$ server b backend-b.istio-vt-t28.svc.homelab.local:5678 check inter 1s rise 1 fall 3$ === CMD6 (retry): kubectl -n istio-vt-t28 rollout restart deploy/haproxy-test && rollout status --timeout=60s === deployment.apps/haproxy-test restarted Waiting for deployment "haproxy-test" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "haproxy-test" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "haproxy-test" rollout to finish: 0 out of 1 new replicas have been updated... Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "haproxy-test" rollout to finish: 1 old replicas are pending termination... deployment "haproxy-test" successfully rolled out NAME READY STATUS RESTARTS AGE backend-b-6fc75fdbf-5fddz 1/1 Running 0 5m7s client 2/2 Running 0 5m7s haproxy-test-67ccf549b4-92tff 1/1 Terminating 0 3m54s haproxy-test-7dbb79cbdd-dv7zf 1/1 Running 0 0s === NOTE: spec CMD7 (final loop) assumed backend-a stays scaled-to-0 from CMD2, but by now (after configmap edits + 2 rollouts) far more than the fall=3/inter=1s DOWN-detection window has elapsed, so backend-a would already be confirmed DOWN and skipped by plain roundrobin -- that would NOT exercise redispatch at all (trivial pass). To actually test redispatch behavior, restoring backend-a to healthy UP first, then re-scaling to 0 and immediately firing a fast inline loop (same technique as the CMD2/CMD3 retry) to land inside the narrow window again, now against the redispatch-enabled haproxy config. === === restore backend-a to 1 replica, wait for haproxy to mark it UP === deployment.apps/backend-a scaled deployment.apps/backend-a condition met [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success. [WARNING] (8) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] (8) : Server be/a is UP, reason: Layer7 check passed, code: 200, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. === CMD2 (redispatch scenario): kubectl -n istio-vt-t28 scale deployment backend-a --replicas=0 === deployment.apps/backend-a scaled === CMD7: for i in $(seq 1 30); do curl -s http://haproxy-test.../ ; done (with option redispatch active) === backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] backend-b [200] === diag: haproxy-test logs after CMD7 (confirm DOWN timing + redispatch evidence) === [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success. [WARNING] (8) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] (8) : Server be/a is UP, reason: Layer7 check passed, code: 200, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. [WARNING] (8) : Server be/a is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.