What is vLLM tuning?

This is the second article in the Internal LLM Serving series. Why serve with vLLM on internal GPUs? covers the adoption decision and when vLLM fits.

1. The engineering starts after the model loads

I work with on-premises AI in a large Korean enterprise. Security requirements around nationally designated core technologies mean that sending every request to an external commercial API is not an option. We serve open-source models through vLLM in Kubernetes Pods, with internal applications and agents sharing the inference service. This article describes the engineering decisions involved, without disclosing an organization’s infrastructure or operating data.

A model that loads successfully and returns a response is a useful milestone. It is not yet a capacity plan. Long-document RAG competes with short conversations. An agent may call the model to plan, interpret tool results, and compose its answer. Adding another application can therefore add far more inference traffic than its user count suggests. When additional GPUs are difficult to obtain, those interactions become the engineering problem.

vLLM is also a scheduler: it decides how limited GPU resources are shared among inference requests. Treating it only as an HTTP server misses the decisions that shape performance under load.

The objective is maximum sustainable throughput per GPU within the service’s latency and error SLOs. Sustainable means a representative workload can keep running without an ever-growing queue or deteriorating reliability. A brief peak in tokens per second does not establish that capacity.

Documentation checked: September 20, 2026, against the official stable documentation and its V1 guidance. These URLs move over time. Record the installed version, image digest, CLI help, startup logs, and /metrics schema for each experiment. All manifests, SLOs, workload quantities, and performance numbers below are illustrative, not measurements or disclosures from an employer.

2. What runs inside the Kubernetes Pod

The basic path is Internal Application → Kubernetes Service → vLLM Pod → GPU. A gateway can provide authentication and admission control in front of that path.

A Deployment maintains the Pods; a Service gives clients a stable destination when Pods change. GPU scheduling requires the appropriate driver and device plugin. If GPU requests and limits are both specified, they must match; specifying only the limit also establishes the request. CPU and RAM matter too: tokenization or frontend work can limit a GPU service. Kubernetes GPU scheduling

This illustrative manifest is a starting point, not a deployment-ready configuration. Replace the image placeholder with an approved pinned version, preferably also a digest. Provision verified model files in the PVC first. Size CPU, RAM, shared memory, and probe thresholds from measurements.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-example
spec:
  replicas: 1
  # Illustrative: avoids needing a spare GPU during replacement.
  # Recreate causes downtime; choose an availability plan separately.
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: inference-example
  template:
    metadata:
      labels:
        app: inference-example
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:REPLACE_WITH_PINNED_VERSION
          command: ["vllm", "serve"]
          args:
            - /models/approved-model
            - --host
            - "0.0.0.0"
            - --port
            - "8000"
            - --tensor-parallel-size
            - "1"
            - --max-model-len
            - "8192"
            - --gpu-memory-utilization
            - "0.90"
            - --max-num-seqs
            - "64"
            - --max-num-batched-tokens
            - "8192"
          ports:
            - name: http
              containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: 16Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: 32Gi
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model
              mountPath: /models
              readOnly: true
            - name: shm
              mountPath: /dev/shm
          startupProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 10
          livenessProbe:
            httpGet: {path: /health, port: http}
            periodSeconds: 30
            failureThreshold: 5
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: approved-model-pvc
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: inference-example
spec:
  type: ClusterIP
  selector:
    app: inference-example
  ports:
    - port: 8000
      targetPort: http

A startup probe gives loading and warmup time before liveness checks can trigger restarts. Readiness controls traffic eligibility; liveness determines when a restart is necessary. Passing /health does not prove that latency meets an SLO. Tune timeouts and failure thresholds so overload does not become a restart loop. Memory-backed /dev/shm also consumes the container’s memory budget. Kubernetes probes, vLLM Kubernetes deployment

Tensor parallelism spreads a model’s computation across GPUs. Changing its degree requires checking the GPU allocation, model compatibility, and communication cost. It is different from increasing replicas: replicas are separate servers with their own weights and caches, while tensor-parallel workers cooperate on one model execution.

The manifest explicitly chooses 0.90 as an example. At the review date, the serve CLI documents 0.92 as the default. This is an executor memory budget, not a target for compute utilization. An explicit --kv-cache-memory-bytes overrides that automatic KV sizing. Pin a validated release and remeasure after upgrades.

3. Why expose a separate inference API?

With Agents / RAG / Chat → API → vLLM → GPU, applications can evolve independently of model deployment. Each application no longer needs to load its own weights or manage its own GPU process. HTTP also gives different client languages a common interface and provides a place to manage authentication, rate limits, usage, and observability.

The OpenAI-compatible API helps reuse existing clients, but compatibility is not a promise that every model supports every feature identically. Validate chat templates, parameters, tool calling, and streaming against the pinned runtime. Official server documentation

A shared service is also a shared bottleneck. Set input and output limits, bounded waiting, timeouts, cancellation propagation, and retry budgets. Otherwise, an agent retry loop can degrade unrelated applications. Check streaming through the entire path: a buffering proxy can delay visible output even when generation is fast.

An API key is not a complete security boundary. vLLM documents endpoints outside its API-key protection. Restrict exposed paths through a gateway or reverse proxy, enforce network isolation and NetworkPolicy, and provide TLS, user authorization, and quotas. Keep metrics on an internal monitoring path. A ClusterIP alone does not implement these controls. vLLM Security

4. Prefill and decode have different costs

Prefill processes the prompt and prepares the state needed for generation. A long RAG context increases the amount of input work before the first answer token. Decode repeatedly generates the next token using the preceding tokens and stored state.

Prefill is often relatively compute-intensive; decode is often more sensitive to memory bandwidth. Hardware, model architecture, and batching can change the balance, so treat this as a diagnostic starting point. vLLM tuning guide

“Responses are slow” is not specific enough to choose a setting. Is the user waiting for output to begin, or does the stream slow down after it starts? A long input and a long output occupy the shared system in different ways.

5. KV cache is the memory cost of keeping requests active

Attention reuses keys and values for earlier tokens. The KV cache stores that state instead of repeatedly computing it. Active generation retains state across iterations, so fitting the weights in memory does not imply ample capacity for concurrent requests. vLLM’s APC introduction

A useful intuition is:

KV cache pressure ≈ active sequences
                  × sequence length
                  × model-dependent KV size

Sequence length includes the prompt and output generated so far. This is not an exact allocation formula: layers, KV heads, dtype, attention architecture, block allocation, prefix sharing, and parallelism all matter. Use runtime-reported KV capacity and measured peaks for sizing.

Long contexts, high concurrency, and long outputs compound each other. A request that fits when admitted can increase pressure as it generates. Also distinguish reserved GPU memory from occupancy within the KV pool. Framebuffer usage alone does not reveal how much additional concurrency the cache can sustain.

6. Continuous batching reuses capacity as requests finish

Requests rarely have identical input lengths or finish together. A static batch can leave capacity unused while its longest requests complete. Continuous batching rebuilds the runnable set at scheduling steps, removing completed requests and admitting eligible work.

Eligibility still depends on several constraints: available sequence slots, this iteration’s token budget, KV capacity, and other scheduling restrictions. This is why “increase the batch size” is an incomplete recommendation. Scheduler configuration

7. Sequence limits and token budgets are different controls

max_num_seqs caps sequences processed per iteration. max_num_batched_tokens caps tokens processed per iteration. Neither is the total queue length or requests per second. Their CLI forms are --max-num-seqs and --max-num-batched-tokens. serve CLI

For an illustrative ordinary one-token decode step, 32 decoding sequences need roughly 32 tokens of iteration budget. Adding a 1,000-token prefill chunk would require roughly 1,032 tokens and 33 sequence slots. Yet those decoding requests may retain much more historical KV state. Tokens computed in this step are not the same as tokens stored in memory. Features such as speculative decoding change this simple accounting.

Increasing the sequence cap creates an opportunity for more concurrent work, but can increase KV demand and step duration. Increasing the token budget cannot bypass the sequence cap, and increasing the sequence cap cannot bypass the token or memory constraints.

Do not memorize a universal default for either control. The current EngineArgs implementation resolves defaults using hardware, usage context, and configuration. Record the effective configuration, not just the command-line overrides.

8. Chunked prefill keeps long inputs from monopolizing a step

A large prefill can delay ongoing decode work. Chunked prefill spreads input computation over scheduling steps while preserving a single request’s context. It does not mean splitting a document into separate application-level prompts.

V1 enables it by default where supported. Check the effective state of --enable-chunked-prefill and model support in startup logs; do not transfer V0 scheduling assumptions into V1. Chunked prefill, default resolution

9. Measure four groups together

Read latency, scheduling, GPU/KV state, and throughput on the same time axis. The table lists current documented V1 metric families. Counters normally expose _total samples; histograms expose _bucket, _sum, and _count. Confirm exact names, labels, units, and HELP/TYPE in the running Pod’s /metrics. A missing metric is not a zero. Production Metrics

Track vllm:inter_token_latency_seconds separately. KV occupancy is a ratio: 1 means 100%. Older dashboards may reference vllm:gpu_cache_usage_perc; verify before reusing them. Check whether input counters include cached or newly computed tokens.

Useful NVIDIA exporter fields include DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, and DCGM_FI_DEV_MEM_COPY_UTIL. Memory activity is different from memory capacity usage and is not directly a GB/s measurement. More detailed DRAM profiling depends on hardware support and collector configuration; do not assume it is enabled. NVIDIA counter definitions

These illustrative PromQL queries assume the series and labels have been verified. Add job, model, and deployment selectors appropriate to the monitored service.

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:time_to_first_token_seconds_bucket[5m])
  )
)

sum by (model_name) (rate(vllm:generation_tokens_total[5m]))

sum by (model_name) (increase(vllm:num_preemptions_total[5m]))

Aggregate histogram buckets before calculating a fleet percentile; do not average Pod p95s. Sparse observations or coarse buckets make p99 unreliable. Preserve per-replica views to detect imbalance. Client latency and gateway rejections, timeouts, and cancellations require additional instrumentation: successful server requests alone cannot describe the user experience.

10. TTFT and TPOT locate different symptoms

TTFT measures arrival to first token. A client measurement includes network and gateway delays that a server measurement may exclude. For streaming, define first content carefully so an empty role event does not count as an answer token.

TTFT ≈ queue + prefill + scheduling/frontend overhead
TPOT = (last token time − first token time) / (output tokens − 1)

The TPOT formula applies to requests producing at least two tokens. Record how single-token outputs are handled. TPOT produces one average per request; ITL records individual intervals. Their percentiles therefore differ. Use TPOT to characterize overall generation speed and ITL to reveal pauses within streams. Align server and benchmark-client definitions before comparing values. vLLM metrics design

These are hypotheses, not diagnoses. If only client-visible TTFT worsens, inspect proxy buffering and the network before changing the scheduler.

11. Why look at p95 and p99, not just the average

An average can hide the minority of requests that wait much longer. p95 marks a boundary containing 95% of observations; p99 describes a rarer tail. Comparisons require consistent sample counts, time windows, and workload composition.

Agents chain model calls or wait for the slowest parallel branch. As an illustrative calculation, if ten independent calls each have a 95% chance of staying within a boundary, the chance that all ten do so is 0.95^10 ≈ 60%. Real calls can be correlated through shared load, so this is an intuition, not a workflow forecast. Nor can you obtain workflow p95 by adding individual p95s. Measure workflow E2E directly.

12. Choosing an operating point with numbers

Every value below is illustrative, not measured. Imagine the same workload, GPU allocation, and observation window. GPU/KV utilization and waiting requests represent window averages; a real scorecard also needs peaks, distributions, and trends.

For illustrative limits of TTFT p95 below 1.5s and TPOT p95 below 50ms, A misses the first-token target and C misses the generation target. C delivers about 7.7% more output throughput than B, but its TPOT p95 is about 95% longer. Its shorter queue does not compensate for the missed SLO, limited KV headroom, and preemptions that need investigation.

B is a candidate based on those two latency targets, not a proven winner. The table lacks E2E, error, OOM, and queue-trend evidence. If those also pass and repeated trials remain stable, B may be the better operating point. Distinguish an isolated preemption from sustained recomputation pressure.

Both parameters change across columns, so the table cannot establish which caused an improvement. Controlled experiments must change one at a time. Also, sustainable completions at a fixed offered load are ultimately arrival-limited. Different completion rates in a finite window are not automatically different capacity ceilings: inspect unfinished work, failures, output lengths, and arrival-rate sweeps.

13. Every parameter change spends something

With chunked prefill, a smaller token budget can favor decode ITL, while a larger budget can favor prefill progress and TTFT. These are tuning directions, not guarantees. Official tuning guide

Ask whether a larger step addresses the current bottleneck. If first-token latency is already healthy but generation slows, define what would make a larger budget a successful experiment before trying it.

Apply the same reasoning to the sequence cap: is reduced queueing worth increased KV pressure, preemption, or tail latency? Recomputation can keep the GPU busy without increasing useful completions. Raising gpu_memory_utilization spends memory headroom; it is not a remedy that can ignore OOM risk or other GPU users.

There is no magic number. Long-context RAG and short chat replies need not share the same optimum. If one configuration cannot meet both SLOs, compare workload-specific admission controls or separate serving pools, including their resource cost.

14. Repeated prompts and prefix caching

Enterprise agents often repeat system instructions, shared prefixes, and RAG boilerplate. APC reuses matching prefix KV state to reduce input computation. It is neither semantic similarity search nor an answer cache, and it does not eliminate generation of new output. Automatic Prefix Caching

Default activation depends on model support. Verify --enable-prefix-caching and the effective configuration rather than assuming APC is active. EngineArgs

Once the endpoint is verified, compare increases in vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total for a token-based hit ratio, alongside TTFT. Zero queries means the ratio is unavailable. Separate cold and warm cache tests and preserve realistic prefix repetition; repeatedly sending one identical prompt can exaggerate the benefit. Metric definitions

Do not merge user-specific instructions or permissions to manufacture cache hits. Shared prefixes do not grant access to data, and tenant cache-isolation policies still apply.

15. A practical tuning loop

Step 1 — Describe the workload. Separate chat, RAG, and agent traffic. Record prompt/output token p50/p95/p99, arrival RPS and bursts, concurrency, and mean and tail model calls per workflow. Equal daily request counts can hide very different peak demand.

Step 2 — Set SLOs. Illustrative starting targets are TTFT p95 < 1.5s, TPOT p95 < 50ms, and E2E p95 < 8s. Define the error budget and treatment of cancellations, timeouts, and rejections. Check whether the output-length distribution is compatible with the E2E target, and apply targets by workload.

Step 3 — Measure the unchanged baseline. Exclude warmup and hold dataset, output policy, arrival trace, and cache state constant across repeated runs. Open-loop RPS sweeps expose backlog because arrivals continue when responses slow. Closed-loop concurrency tests reduce arrivals as response times rise, potentially concealing overload. Use each deliberately and verify the load generator has capacity. vLLM benchmark CLI

Step 4 — Form a bottleneck hypothesis. Low GPU utilization with a large queue warrants inspecting CPU throttling, tokenization, communication, gateway behavior, and replica imbalance as well as scheduler limits. High GPU/KV activity with preemption suggests a context/output/concurrency interaction. High GPU utilization with a small queue and healthy latency may already be a good result.

Step 5 — Change one variable. First vary the token budget at a fixed sequence cap; then compare sequence caps at the chosen budget. Explore interactions in a second pass. At each candidate and load level, report repetitions, sample counts, and variability, then establish the sustainable SLO-compliant limit.

The final scorecard should include offered/admitted/completed RPS, input/output tok/s, throughput per allocated GPU, SLO-compliant goodput, latency percentiles, errors, queue trend, KV peaks, and preemption rate. More raw tokens can mean less service value if more responses miss their targets. Define rollback triggers and preserve the previous image and configuration before testing.

16. A copyable prompt for a coding assistant

“Optimize vLLM” invites plausible configuration guesses. The prompt below asks for inspection, reproducible experiments, and an evidence-based recommendation. It starts with read-only access, requires an approved test scope, and requests a patch proposal without applying it. Copy the whole prompt with the button, or select the text directly.

Act as an inference performance engineer for an on-premises Kubernetes vLLM service.

Objective: maximize sustainable throughput subject to TTFT/TPOT/E2E/error SLO and no persistent OOM/preemption.
Do not optimize for 100% GPU utilization. Optimize useful completed work per allocated GPU, with stable queues and latency headroom.

Safety and scope
- Begin with read-only inspection in the explicitly authorized environment.
- Do not apply configuration patches, restart Pods, scale workloads, change routing, install exporters, or run production load tests.
- Prepare the benchmark plan first. Execute tests only in an approved isolated environment and within an agreed load budget.
- Keep prompts, outputs, credentials, hostnames, inventory, and internal metrics inside the authorized environment. Redact report identifiers. Do not export raw production payloads.
- Label every result as measured, derived, inferred, or illustrative. Never fabricate measurements or silently substitute unavailable metrics.

1. Inspect the runtime
Record the exact vLLM version, image digest, engine, startup command, effective config, model/tokenizer revision (redacted identifiers), dtype, quantization, KV dtype, context limit, parallel layout, replica count, and allocated GPUs.
Inspect driver/runtime compatibility, resource requests/limits, CPU throttling, memory, volumes, startup/readiness/liveness probes, restarts, gateway admission control, and streaming/proxy buffering.
Read installed-version CLI help and matching official docs. Do not import defaults or metric names from another version or engine.
Record effective max_num_seqs, max_num_batched_tokens, gpu_memory_utilization, explicit KV-memory overrides, chunked prefill, and prefix caching. Capture support constraints from startup logs.

2. Establish metric definitions
Inspect the actual /metrics endpoint and exporter configuration. List exact series names, HELP/TYPE, units, labels, histogram buckets, counter suffixes, and unavailable metrics. Save a sanitized schema snapshot.
Distinguish server TTFT from client-visible TTFT; exclude empty role-only SSE events from first-content timing. Record client E2E, timeouts, cancellations, errors, and rejected requests.
Define per-request TPOT as (last-token time - first-token time)/(output tokens - 1) for outputs longer than one token; report exclusions. Do not equate TPOT percentiles with inter-token-latency percentiles.
Report TTFT, TPOT, E2E p50/p95/p99; running/waiting requests; queue-time distributions; preemption rates; KV occupancy; GPU utilization; framebuffer memory; supported DRAM activity/bandwidth counters; CPU/network constraints.
Report input/output tokens per second, successful requests per second, offered/admitted/completed RPS, SLO-compliant goodput, and per-GPU throughput. Explain whether cached input tokens count as processed input.
Use rate/increase for counters. Aggregate histogram buckets before quantiles; never average per-Pod percentiles. Keep workload/model/replica attribution and sample counts.

3. Profile the workload
For chat, long-context RAG, and agent traffic separately, measure prompt/output token p50/p95/p99, arrival RPS and burstiness, in-flight concurrency, mean and tail LLM calls per agent workflow, and repeated-prefix frequency.
Use a sanitized representative dataset with fixed sampling parameters, seeds where supported, and token-length distributions. Include long prompts and long outputs, not only short synthetic requests.
Obtain explicit TTFT/TPOT/E2E/error SLOs, duration, minimum sample count, and stop conditions. If missing, propose illustrative targets labeled as unapproved.

4. Baseline and controlled experiments
Start with unchanged config. Warm up compilation, kernels, and connections; separate cold-cache and warm-cache tests and exclude warmup from scored intervals.
Use the same dataset, arrival trace, GPU allocation, measurement window, and cache-state policy. Repeat each candidate at least three times where the approved budget permits; report variability and insufficient evidence.
Use open-loop arrival-rate sweeps to expose backlog; separately report closed-loop concurrency tests and their tendency to reduce offered load as latency grows. Verify the load generator is not the bottleneck.
First hold max_num_seqs fixed and vary only max_num_batched_tokens. Then hold the chosen token budget fixed and vary only max_num_seqs. Revisit interactions in a documented second pass.
Inspect chunked-prefill effectiveness using long-prompt/streaming traffic; test alternatives only if the installed model/engine supports them. For prefix caching, compare realistic shared prefixes against unique prefixes and cold/warm states; measure token hit ratio and TTFT together.
Stop on agreed error/OOM/latency thresholds or persistent queue growth. Do not interpret burst throughput as sustainable capacity.

5. Scorecard and diagnosis
Produce one row per candidate, load level, cache condition, and repeat. Include config diff, versions, sample count, offered/admitted/completed RPS, input/output tok/s, per-GPU goodput, latency percentiles, errors, queue trend, preemptions, KV peak/headroom, GPU/DRAM metrics, and repeat variability.
For each suspected bottleneck, state evidence, competing explanations, and the next discriminating measurement. Low GPU plus high queue is not proof that concurrency is too low; check CPU tokenization, gateway, replica imbalance, scheduling constraints, and communication.
Reject candidates violating any SLO or showing persistent OOM/preemption/backlog. If all fail, say so. Recommend the stable candidate with the highest sustainable SLO-compliant throughput and sufficient headroom, not the highest utilization or raw tokens/sec.

6. Deliverables, without applying changes
Provide a recommendation, runtime inventory summary, metric dictionary, reproducible benchmark commands for the installed version, complete scorecard, and measured-versus-inferred findings.
Propose a minimal configuration patch only. Include expected trade-offs, uncertainties, validation gates, and rollback to the previous image digest/config and routing, with explicit rollback triggers and post-rollback checks.
End with what remains unmeasured and what authorization is required before execution or deployment.

17. GPU utilization is not the objective function

Tuning is a feedback loop: characterize workload, benchmark, inspect metrics, form a bottleneck hypothesis, change a parameter, and test again. The justification for a setting matters more than the setting itself.

High utilization and raw throughput are useful observations, but neither substitutes for response-time and reliability commitments. The operating point to choose is the one that sustainably completes the most useful work on the available GPUs while honoring those commitments.

Loading the model onto a GPU is only the beginning. The real inference engineering starts after that.