A local model can manage a brief question without any issues, but it may have difficulty when dealing with a complete document. The model's internal settings stay the same, but the effort needed and the memory required to process the request increase a lot.

In Part 1, I explained how to run a quantized model on a smaller GPU.

The next step is to choose a task, measure how long it takes, and find a setup that works within the GPU's limits.

It includes memory calculations, a fixed starting configuration, and a Python-based benchmark that you can run on your own computer. The numerical memory estimates are calculated values, and this article does not claim to have measured improvements in GPU performance.

To understand the actual delay you experience, there are two main waiting periods in a conversation. The first occurs when the model processes the initial prompt, often called the prefill phase. The second happens when the model generates the answer one word at a time, known as the decode phase.

If a document takes several seconds to process, fast decoding won't eliminate that initial delay. However, if the first words appear quickly but the rest of the response is slow, then the decoding phase is the area that needs optimization.

Keep these measurements separate:

A target such as "start a short answer within two seconds" can guide your own experiments. It is a product requirement you choose, not a promised result for every 8GB card.

Budget for the KV cache as well as the weights

The model file is only part of the memory requirement. During generation, the runtime also stores attention keys and values for tokens already processed. This KV cache avoids recomputing those states, but it needs memory of its own. Working buffers and other GPU applications need space too. [1]

For a conventional full-attention transformer, an approximate KV-cache payload is:

KV bytes = 2 x layers x stored tokens x KV heads x head dimension

x bytes per cache element

The initial 2 accounts for keys and values. Use the number of KV heads, which can differ from the number of query heads.

Qwen2.5-7B-Instruct has 28 layers, 4 KV heads, and a head dimension of 128, obtained from its hidden size of 3,584 divided by 28 query heads. With FP16 cache elements, the calculation becomes 2 x 28 x tokens x 4 x 128 x 2. [2]

These figures describe one sequence and exclude allocation padding, runtime buffers, and model weights. They are not measurements of total VRAM usage. Other architectures, cache layouts, and attention types need different calculations.

This is why a large advertised context window needs its own memory budget. On this model, moving from 4K to 32K tokens adds about 1.53 GiB of calculated FP16 cache payload. Whether that increase fits depends on everything else already occupying the GPU.

Weight quantization and cache quantization are separate choices. A Q4_K_M model can still use an FP16 KV cache. Reducing cache precision may free up space, but it needs a quality check and a latency measurement. Hugging Face's cache documentation also cautions that cache quantization can hurt latency in some short-context situations. [1]

Establish one repeatable baseline

The shell commands below target Linux or WSL2 with an NVIDIA GPU. Install a working NVIDIA driver, a compatible CUDA Toolkit, Git, CMake, and a C++ compiler first. The benchmark itself requires Python 3.10 or newer and no third-party Python packages.

Use the current CMake build route in the llama.cpp build documentation: [3]

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4
git rev-parse HEAD > llama-commit.txt
./build/bin/llama-server --version > llama-version.txt
./build/bin/llama-server --help > llama-help.txt
nvidia-smi > gpu-environment.txt

Keep that checkout unchanged throughout the experiment. Save the model filenames and their SHA-256 hashes too, including every shard if the download is split. A model tag alone is not enough to reproduce an experiment after its files change.

I use the official Qwen GGUF repository because its model configuration is available to inspect. It is a concrete baseline, not a claim that this is the best available model. [4]

Start the server from the llama.cpp directory:

./build/bin/llama-server \
  -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \
  --n-gpu-layers all \
  --ctx-size 4096 --parallel 1 \
  --batch-size 512 --ubatch-size 128 \
  --flash-attn off \
  --cache-type-k f16 --cache-type-v f16 \
  --fit off --no-context-shift --perf \
  --host 127.0.0.1 --port 8080 \
  2> baseline-server.log

The first launch downloads the selected model. Exclude that download and startup from request timings. Once the server is ready, inspect its log for the CUDA device, layer placement, cache allocation, and effective context settings. If an installed version rejects an option, consult its saved help output before continuing. The current server options are documented upstream. [5]

This configuration requests full GPU offload; it does not guarantee that every 8GB system has enough free memory. If startup fails, first close another GPU workload or test a smaller context. If partial offload is necessary, record the actual GPU layer count and use it consistently across the comparison.

The baseline has one request slot. Keep other inference clients stopped while measuring. With automatic fitting disabled, a failed configuration is visible instead of becoming a different experiment through an adjusted setting.

Run a benchmark that preserves the evidence</

A local model can manage a brief question without any issues, but it may have difficulty when dealing with a complete document. The model's internal settings stay the same, but the effort needed and the memory required to process the request increase a lot.

In Part 1, I explained how to run a quantized model on a smaller GPU.

The next step is to choose a task, measure how long it takes, and find a setup that works within the GPU's limits.

It includes memory calculations, a fixed starting configuration, and a Python-based benchmark that you can run on your own computer. The numerical memory estimates are calculated values, and this article does not claim to have measured improvements in GPU performance.

To understand the actual delay you experience, there are two main waiting periods in a conversation. The first occurs when the model processes the initial prompt, often called the prefill phase. The second happens when the model generates the answer one word at a time, known as the decode phase.

If a document takes several seconds to process, fast decoding won't eliminate that initial delay. However, if the first words appear quickly but the rest of the response is slow, then the decoding phase is the area that needs optimization.

Keep these measurements separate:

A target such as "start a short answer within two seconds" can guide your own experiments. It is a product requirement you choose, not a promised result for every 8GB card.

Budget for the KV cache as well as the weights

The model file is only part of the memory requirement. During generation, the runtime also stores attention keys and values for tokens already processed. This KV cache avoids recomputing those states, but it needs memory of its own. Working buffers and other GPU applications need space too. [1]

For a conventional full-attention transformer, an approximate KV-cache payload is:

KV bytes = 2 x layers x stored tokens x KV heads x head dimension

x bytes per cache element

The initial 2 accounts for keys and values. Use the number of KV heads, which can differ from the number of query heads.

Qwen2.5-7B-Instruct has 28 layers, 4 KV heads, and a head dimension of 128, obtained from its hidden size of 3,584 divided by 28 query heads. With FP16 cache elements, the calculation becomes 2 x 28 x tokens x 4 x 128 x 2. [2]

These figures describe one sequence and exclude allocation padding, runtime buffers, and model weights. They are not measurements of total VRAM usage. Other architectures, cache layouts, and attention types need different calculations.

This is why a large advertised context window needs its own memory budget. On this model, moving from 4K to 32K tokens adds about 1.53 GiB of calculated FP16 cache payload. Whether that increase fits depends on everything else already occupying the GPU.

Weight quantization and cache quantization are separate choices. A Q4_K_M model can still use an FP16 KV cache. Reducing cache precision may free up space, but it needs a quality check and a latency measurement. Hugging Face's cache documentation also cautions that cache quantization can hurt latency in some short-context situations. [1]

Establish one repeatable baseline

The shell commands below target Linux or WSL2 with an NVIDIA GPU. Install a working NVIDIA driver, a compatible CUDA Toolkit, Git, CMake, and a C++ compiler first. The benchmark itself requires Python 3.10 or newer and no third-party Python packages.

Use the current CMake build route in the llama.cpp build documentation: [3]

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4
git rev-parse HEAD > llama-commit.txt
./build/bin/llama-server --version > llama-version.txt
./build/bin/llama-server --help > llama-help.txt
nvidia-smi > gpu-environment.txt

Keep that checkout unchanged throughout the experiment. Save the model filenames and their SHA-256 hashes too, including every shard if the download is split. A model tag alone is not enough to reproduce an experiment after its files change.

I use the official Qwen GGUF repository because its model configuration is available to inspect. It is a concrete baseline, not a claim that this is the best available model. [4]

Start the server from the llama.cpp directory:

./build/bin/llama-server \
  -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \
  --n-gpu-layers all \
  --ctx-size 4096 --parallel 1 \
  --batch-size 512 --ubatch-size 128 \
  --flash-attn off \
  --cache-type-k f16 --cache-type-v f16 \
  --fit off --no-context-shift --perf \
  --host 127.0.0.1 --port 8080 \
  2> baseline-server.log

The first launch downloads the selected model. Exclude that download and startup from request timings. Once the server is ready, inspect its log for the CUDA device, layer placement, cache allocation, and effective context settings. If an installed version rejects an option, consult its saved help output before continuing. The current server options are documented upstream. [5]

This configuration requests full GPU offload; it does not guarantee that every 8GB system has enough free memory. If startup fails, first close another GPU workload or test a smaller context. If partial offload is necessary, record the actual GPU layer count and use it consistently across the comparison.

The baseline has one request slot. Keep other inference clients stopped while measuring. With automatic fitting disabled, a failed configuration is visible instead of becoming a different experiment through an adjusted setting.

Run a benchmark that preserves the evidence</