https://blog.homelab89.com/tags/llm/ Recent content in Llm on homelab89 Hugo ko-KR Sun, 28 Jun 2026 00:00:00 +0000 https://blog.homelab89.com/docs/homelab/nvidia-driver-setup/ Sun, 28 Jun 2026 00:00:00 +0000 https://blog.homelab89.com/docs/homelab/nvidia-driver-setup/ <p><strong>Ubuntu 24.04 LTS | RTX 3090 | Qwen2.5-7B-Instruct</strong></p> <p>Verified: 2026-02-17</p> <hr> <h2 id="verified-environment">Verified Environment</h2> <table> <thead> <tr> <th>Component</th> <th>Detail</th> </tr> </thead> <tbody> <tr> <td>GPU</td> <td>NVIDIA GeForce RTX 3090 (GA102, Compute Capability 8.6, Ampere)</td> </tr> <tr> <td>OS</td> <td>Ubuntu 24.04.4 LTS, Linux 6.17.0-14-generic</td> </tr> <tr> <td>NVIDIA Driver</td> <td>nvidia-driver-590-open, 590.48.01 (CUDA 13.1)</td> </tr> <tr> <td>Docker</td> <td>29.2.1</td> </tr> <tr> <td>NVIDIA Container Toolkit</td> <td>1.18.2</td> </tr> <tr> <td>vLLM Image</td> <td>vllm/vllm-openai:latest (v0.15.1, ~29.5GB)</td> </tr> <tr> <td>Model</td> <td>Qwen/Qwen2.5-7B-Instruct (bf16, ~22GB VRAM)</td> </tr> </tbody> </table> <hr> <h2 id="prerequisites">Prerequisites</h2> <p>The following must be installed before starting:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># Docker</span> </span></span><span class="line"><span class="cl">sudo apt install docker.io </span></span><span class="line"><span class="cl">sudo usermod -aG docker <span class="nv">$USER</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># NVIDIA Container Toolkit (enables --gpus flag)</span> </span></span><span class="line"><span class="cl"><span class="c1"># https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html</span> </span></span><span class="line"><span class="cl">curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"> sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg </span></span><span class="line"><span class="cl">curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"> sed <span class="s1">&#39;s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g&#39;</span> <span class="p">|</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"> sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list </span></span><span class="line"><span class="cl">sudo apt update </span></span><span class="line"><span class="cl">sudo apt install nvidia-container-toolkit </span></span><span class="line"><span class="cl">sudo nvidia-ctk runtime configure --runtime<span class="o">=</span>docker </span></span><span class="line"><span class="cl">sudo systemctl restart docker </span></span></code></pre></div><hr> <h2 id="part-1-nvidia-driver-installation">Part 1: NVIDIA Driver Installation</h2> <h3 id="step-1-check-recommended-driver">Step 1. Check recommended driver</h3> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ubuntu-drivers devices </span></span></code></pre></div><p>Look for the <code>recommended</code> line:</p>