https://blog.homelab89.com/tags/failure-testing/ Recent content in Failure-Testing on homelab89 Hugo ko-KR Mon, 29 Jun 2026 00:00:00 +0000 https://blog.homelab89.com/docs/k8s/etcd-failure-experiments/ Mon, 29 Jun 2026 00:00:00 +0000 https://blog.homelab89.com/docs/k8s/etcd-failure-experiments/ <h2 id="etcd-raft-쿼럼-기본-원리">etcd Raft 쿼럼 기본 원리</h2> <pre tabindex="0"><code>3-node etcd 클러스터: - 쿼럼(과반수) = 2 - 1개 장애: 쿼럼 유지 (2/3 살아있음) → 클러스터 정상 운영 - 2개 장애: 쿼럼 붕괴 (1/3만 살아있음) → 클러스터 읽기 전용 / 쓰기 불가 - 3개 장애: 완전 다운 장애 허용 공식: floor((n-1)/2) 3노드: floor((3-1)/2) = 1개까지 허용 </code></pre><h2 id="실험-전-준비">실험 전 준비</h2> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">KUBECONFIG</span><span class="o">=</span>~/.kube/k8s-etcd-lab.conf </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># 실험 전 etcd 상태 베이스라인 기록</span> </span></span><span class="line"><span class="cl">kubectl get nodes </span></span><span class="line"><span class="cl">kubectl get pods -n kube-system </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># etcd 헬스체크 (ctrl1에서)</span> </span></span><span class="line"><span class="cl">ssh k8s-ctrl1 <span class="s2">&#34;sudo ETCDCTL_API=3 etcdctl \ </span></span></span><span class="line"><span class="cl"><span class="s2"> --endpoints=https://203.0.113.11:2379,https://203.0.113.12:2379,https://203.0.113.13:2379 \ </span></span></span><span class="line"><span class="cl"><span class="s2"> --cacert=/etc/ssl/etcd/ssl/ca.pem \ </span></span></span><span class="line"><span class="cl"><span class="s2"> --cert=/etc/ssl/etcd/ssl/member-k8s-ctrl1.pem \ </span></span></span><span class="line"><span class="cl"><span class="s2"> --key=/etc/ssl/etcd/ssl/member-k8s-ctrl1-key.pem \ </span></span></span><span class="line"><span class="cl"><span class="s2"> endpoint health -w table&#34;</span> </span></span></code></pre></div><hr> <h2 id="시나리오-1-단일-노드-장애-쿼럼-유지">시나리오 1: 단일 노드 장애 (쿼럼 유지)</h2> <p><strong>목표</strong>: etcd 노드 1개 정지 후 클러스터 운영 영향 확인</p>