Inference Engineering: A Practical Tutorial from GPU Fundamentals to Production LLM Serving

What Is Inference Engineering?

Every AI model goes through two distinct phases:

• Training — The model learns its weights from data. This is computationally expensive and typically done once (or periodically retrained).
• Inference — The trained model serves real requests in production. This runs continuously and needs to be fast, cost-effective, and reliable.

With traditional ML models like XGBoost or logistic regression, inference was straightforward — a CPU could handle it without much thought. Generative AI changed that equation entirely. Large language models, image generators, and voice models demand specialized hardware, careful memory management, and purpose-built serving infrastructure. Simply loading weights onto a GPU and calling it a day doesn’t cut it.

💡 Key Concept

Inference engineering is the practice of making generative AI models run faster, cost less, and fail less often in production — all while preserving output quality.

This tutorial walks through the full inference stack with hands-on examples you can run on consumer hardware.

Key Concepts You Should Know

Before diving in, here are foundational terms you’ll see throughout this tutorial. If you’re already comfortable with these, skip ahead.

What are parameters? When you hear “a 7B model,” that means the model has 7 billion learnable numbers — called parameters or weights. Think of them like knobs on a mixing board: each one is a tiny adjustment that shapes the model’s behavior. More parameters generally means more knowledge capacity, but also more memory and compute to run.

What are tokens? LLMs don’t read raw text — they break it into chunks called tokens. A token is roughly a word or part of a word. “Hello world” is 2 tokens, but “unbelievable” might be split into “un” + “believ” + “able” (3 tokens). Token count matters because it directly affects how long inference takes and how much it costs.

What are weights? Weights are the numbers the model learned during training. They’re stored as numbers in a specific format — for example, FP16 (16-bit floating point) uses 2 bytes per weight. This is why a 7B-parameter model needs roughly 14 GB of memory: 7 billion × 2 bytes = 14 GB.

What is VRAM? VRAM is the GPU’s own memory, separate from your computer’s system RAM. Models must fit in VRAM to run on a GPU. A consumer GPU like the RTX 3060 has 12 GB of VRAM; a datacenter H100 has 80 GB. If your model is too large, you need to either quantize it (shrink it) or split it across multiple GPUs.

What are FLOPS? FLOPS stands for Floating-point Operations Per Second — it measures how fast a GPU can do math. TFLOPS means trillion FLOPS. More FLOPS means faster matrix multiplications, which means faster inference. When you see “H100: ~2 PFLOPS FP8,” that’s 2 quadrillion math operations per second.

Why Inference Engineering Matters Now

Why Not Just Use Claude or GPT?

If frontier APIs like Claude and GPT keep getting better and cheaper, why bother running your own models at all? It’s a fair question. Here’s the honest answer: for many use cases, you shouldn’t — API-based models are the right choice when you’re prototyping, when your volume is low, or when the task demands frontier reasoning capabilities.

But APIs have structural limitations that no amount of model improvement can fix:

• You share infrastructure with every other customer. When demand spikes (product launches, viral moments), API latency degrades for everyone. Your SLA is their SLA. You cannot guarantee P99 latency to your own users.
• You can’t customize the model. Fine-tuning through APIs is limited and opaque. If your product needs a model that deeply understands medical imaging, legal contracts, or your company’s proprietary codebase, you need to train or fine-tune your own — and that means running your own inference.
• You can’t control where data goes. Regulated industries (healthcare, finance, defense) often require that data never leaves your infrastructure. API calls send user data to a third party.
• Cost scales linearly forever. API pricing is pay-per-token with no volume efficiency. Once you’re processing millions of tokens per day, a dedicated GPU running a smaller fine-tuned model can be 10–50× cheaper while delivering comparable quality for your specific task.
• You’re locked to someone else’s roadmap. API providers deprecate models, change pricing, alter behavior between versions, and impose rate limits. Your production system depends on decisions made by someone else’s product team.

The sweet spot for inference engineering is when you’ve found product-market fit, have meaningful scale, and your use case doesn’t require the absolute frontier of general reasoning. A 7B model fine-tuned on your domain, quantized to FP8, served on a single GPU, can outperform a general-purpose frontier model on your specific task — at a fraction of the cost and latency.

What “Open” Actually Means for LLMs

The term “open” in AI doesn’t mean the same thing as “open source” in software. In traditional open source (Linux, PostgreSQL, React), you get the complete source code, build tools, and the right to modify and redistribute everything. AI models have a more nuanced spectrum:

Open Architecture — The model’s design (number of layers, attention mechanism, MoE routing strategy) is published in a research paper or technical report. Anyone can read how it works and reimplement it from scratch. Most major labs publish their architectures. This is useful for researchers but doesn’t directly help you run the model.

Open Weights — The trained model weights are downloadable. This is what most people mean by “open models.” You can load the weights, run inference, fine-tune on your data, quantize to smaller formats, and deploy on your own hardware. This is what matters most for inference engineering — you need the weights to do anything described in this tutorial. Examples: Llama 3, DeepSeek-V3, Qwen3, Mistral.

However, “open weights” doesn’t necessarily mean “open source.” Many open-weight models come with restrictive licenses (Meta’s Llama license limits commercial use above certain scales, some models prohibit certain applications). The training code, data, and full training recipe are rarely published.

Open on Hugging Face — Hugging Face hosts over 2 million models that anyone can download. These range from frontier 235B-parameter models from major labs to small fine-tuned variants created by individual developers. “Available on Hugging Face” means you can download the weights in standard formats (safetensors, GGUF) and run them locally — but always check the license before commercial use.

Why open weights matter to you as a developer: Without downloadable weights, you can’t quantize, fine-tune, benchmark on your data, control your deployment, or run offline. Open weights are what make inference engineering possible. Closed models (GPT-4, Claude) are powerful, but they’re black boxes — you interact through an API and accept whatever latency, cost, and behavior the provider decides.

The Open Model Ecosystem

The quality gap between closed-source APIs (GPT, Claude) and open-weight models (DeepSeek, Llama, Qwen) has shrunk dramatically. DeepSeek-V3 (December 2024) and R1 (January 2025) matched or surpassed several closed models on major benchmarks. Closed models retain advantages in complex agentic tasks and safety tuning, but the trend is clear: for many production use cases, open models deliver sufficient quality at a fraction of the cost.

Running your own models gives you control over three things that API providers optimize differently:

• Latency — You tune for your users’ experience, not the provider’s aggregate throughput.
• Availability — Dedicated infrastructure can achieve 99.99% uptime, compared to ~99% typical of shared APIs.
• Cost — At scale, self-hosted models often cost 80%+ less than per-token API pricing.

The Three Layers of Inference

Inference engineering spans three layers, each with its own concerns:

┌─────────────────────────────────────────────┐
│              TOOLING                        │
│   Developer experience, APIs, abstractions  │
├─────────────────────────────────────────────┤
│           INFRASTRUCTURE                    │
│   Autoscaling, multi-cloud, load balancing  │
├─────────────────────────────────────────────┤
│              RUNTIME                        │
│   CUDA → PyTorch → Inference Engine → GPU   │
│   Quantization, Speculation, Caching, etc.  │
└─────────────────────────────────────────────┘

1. Runtime — Squeezing maximum performance from a single model on a single GPU (or multi-GPU node). This is where quantization, KV caching, and kernel optimization live.
2. Infrastructure — Orchestrating across clusters, regions, and cloud providers — autoscaling, load balancing, and deployment without downtime.
3. Tooling — Providing the right abstractions for engineers. Too much abstraction and you lose control; too little and you’re writing CUDA kernels for every model swap.

Inference Metrics: TTFT, TPS, and What to Measure

Optimization without measurement is guesswork. Before tuning anything, define what “good” means for your specific use case — the tradeoffs between latency, throughput, and output quality differ for every product.

Shared vs. Dedicated Inference

The transition point comes when you have scale (enough request volume that per-GPU pricing beats per-token), specialization (fine-tuned models or strict latency requirements), or orchestration needs (multi-model pipelines where you control routing).

Key Metrics: TTFT and TPS

💡 Key Concept

Always track percentiles (P50, P90, P99), not averages. Averages mask the worst-case experiences that drive users away. A P99 TTFT of 2s means 1 in 100 requests sits there for two full seconds before anything appears.

Model Selection for Inference

The highest-leverage decision you’ll make: choose the smallest model that meets your quality bar. A 3B fine-tuned model generating at 200 TPS will feel dramatically better to users than a 70B model at 30 TPS — if both produce acceptable outputs.

• Run domain-specific evals, not just public benchmarks. Benchmark rankings are gamed (Goodhart’s Law in action).
• Fine-tuning adjusts a pre-trained model’s weights on your data — same architecture, specialized behavior.
• Distillation trains a compact “student” model to reproduce the output distributions of a larger “teacher” model.

🔧 Try It Yourself — Run your first local model and measure TTFT/TPS

Install Ollama and run a small model:

# Pull and run a small model:
ollama run qwen3:4b

# To measure metrics, use the API:
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:4b",
  "prompt": "Explain inference engineering in 3 sentences.",
  "stream": true
}'
# Watch the stream — the first token's arrival is your TTFT.
# Count tokens over time for TPS.

How LLMs Work Under the Hood

Neural Network Refresher

At their core, neural networks are stacks of layers. Each layer takes an input vector, multiplies it by a weight matrix, adds a bias term, and passes the result through a nonlinear activation function. The high-level structure:

Input Layer → Hidden Layer 1 → Hidden Layer 2 → ... → Output Layer
                (ReLU)            (ReLU)                (Softmax)

• Linear layers do the heavy lifting via matrix multiplication: y = Wx + b
• Activation functions (ReLU, SiLU, SwiGLU) introduce nonlinearity — without them, stacking layers would be mathematically equivalent to a single layer.
• Hidden states are the intermediate representations flowing between layers.

LLM Inference Mechanics: Prefill and Decode

LLMs are autoregressive — they generate one token at a time, where each token choice is influenced by everything that came before it. Think of it like completing a sentence word by word: once you write “The cat sat on the,” the next word is shaped by every word you already wrote.

Tokenization: Before processing, text gets broken into subword tokens from a vocabulary of ~100K+ entries. The word “inference” might map to a single token, while “engineering” could split into two. Fewer tokens means less work for the model.

LLM inference happens in two distinct phases:

Phase 1: PREFILL
┌──────────────────────────────────┐
│ Process all input tokens at once │
│ Build the KV Cache              │
│ Generate first output token     │
│ ⏱ This determines TTFT          │
└──────────────────────────────────┘
           ↓
Phase 2: DECODE (repeat until stop token)
┌──────────────────────────────────┐
│ Forward pass → logit vector      │
│ Sample token from probabilities  │
│ Update KV Cache                  │
│ ⏱ This determines TPS            │
└──────────────────────────────────┘

Each decode step produces a logit vector with one value per vocabulary entry. After softmax normalization, these become probabilities. Sampling parameters — temperature, top-k, and top-p — control how the model selects from this distribution: deterministically (low temperature) or creatively (high temperature).

Attention: How Transformers Connect Tokens

The attention mechanism is what lets each token “look at” every previous token in the sequence. It works through three learned projections:

• Q (Queries) — “What am I looking for?”
• K (Keys) — “What information do I offer?”
• V (Values) — “What content do I carry?”

Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V

Modern transformers use multi-head attention — running several attention operations in parallel. Each head can specialize in different relationship types (syntactic structure, semantic similarity, positional patterns).

💡 Key Concept

The KV Cache stores previously computed key-value pairs so they don’t need to be recomputed for each new token.

Analogy: Imagine a student taking notes during a lecture. Without notes, every time the professor says something new, the student would have to mentally replay the entire lecture from the beginning to understand how the new point connects. With notes, they just glance at what they wrote and keep going. The KV cache is those notes — it saves the “memory” of all previous tokens so the model doesn’t redo that work.

The tradeoff: The KV cache grows with sequence length. Every new token adds another entry, and each entry consumes VRAM. For long conversations or documents, the KV cache can consume more VRAM than the model weights themselves.

Without the KV cache, attention would be O(n²) on every step — recomputing all relationships from scratch. With it, each decode step is O(n) — still expensive for long sequences, but far more practical.

Mixture of Experts (MoE) Architecture

MoE models like DeepSeek-V3 and Qwen3-235B take a different approach to scaling. Instead of making every layer wider, they pack multiple specialized “expert” sub-networks into each layer and use a lightweight router to select which experts process each token.

The practical impact: Qwen3-235B-A22B contains 235B total parameters but only activates ~22B for any single request. For a single user running inference locally, this is remarkably efficient. The catch shows up in production: when serving many concurrent requests, different requests activate different experts, and in aggregate nearly all experts get used. The full model still needs to reside in VRAM.

🔧 Try It Yourself — Explore a model’s architecture

# Install huggingface_hub:
# pip install huggingface_hub

from huggingface_hub import hf_hub_download
import json

path = hf_hub_download("Qwen/Qwen3-4B", "config.json")
config = json.load(open(path))
print(f"Architecture: {config['architectures']}")
print(f"Hidden size:  {config['hidden_size']}")
print(f"Num layers:   {config['num_hidden_layers']}")
print(f"Num heads:    {config['num_attention_heads']}")
print(f"Vocab size:   {config['vocab_size']}")

GPU Hardware for AI Inference

How GPUs Differ from CPUs

GPUs are built for parallelism. Where a CPU has a handful of powerful cores optimized for complex sequential logic, a GPU packs thousands of simpler cores designed to execute the same operation across massive data arrays. This architecture is ideal for the matrix multiplications that dominate neural network inference.

💡 Key Concept

GPU performance has two independent bottlenecks:

• Compute (FLOPS) — limits prefill speed and image generation.
• Memory bandwidth (TB/s) — limits LLM decode speed.

The ops:byte ratio reveals which one you’ll hit first. For the H100 in FP16: 989 TFLOPS ÷ 3.35 TB/s ≈ 295 operations per byte loaded. If your workload does fewer than 295 ops per byte of data moved, you’re memory-bound. More than that, you’re compute-bound.

GPU Generations at a Glance

The H100 SXM remains the most widely deployed GPU for production inference as of mid-2026, delivering 1,979 TFLOPS dense FP8 and 3.35 TB/s HBM3 bandwidth. Blackwell approximately doubles both figures. Rubin is expected to bring HBM4 and architectural changes specifically targeting the prefill phase.

Practical Hardware for Local Inference

RTX 3060 12 GB — An accessible starting point:

• VRAM: 12 GB GDDR6
• Bandwidth: ~360 GB/s
• Compute: 12.7 TFLOPS FP32 (no FP8 Tensor Cores on consumer Ampere)
• What fits: 4B–8B models at FP16, up to ~14B with Q4 quantization

MacBook Air M4 — Apple Silicon with unified memory:

• Memory: 16 or 24 GB shared between CPU and GPU
• Bandwidth: ~120 GB/s
• Key advantage: The GPU can address all system memory — no separate VRAM limit
• What fits: 8B–14B at FP16, 30B+ with aggressive quantization

Apple’s unified memory lets you run larger models than discrete GPUs with similar memory totals, because there’s no VRAM/RAM split. The tradeoff is lower bandwidth.

🔧 Try It Yourself — Check your GPU capabilities

# On NVIDIA GPUs (Windows/Linux):
nvidia-smi
# Look for: Memory total, GPU utilization, driver version

# Python check for NVIDIA:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")

# On Apple Silicon (Mac M-series):
import torch
print(f"MPS available: {torch.backends.mps.is_available()}")
# MPS = Metal Performance Shaders (Apple GPU backend)

The Inference Software Stack: CUDA, PyTorch, and Inference Engines

Between your model’s weights and the GPU silicon, several software layers handle translation and optimization:

         Higher abstraction, easier to use
              ┌──────────────┐
              │ NVIDIA Dynamo│  Distributed orchestration
              ├──────────────┤
              │  Inference   │  vLLM, SGLang,
              │  Engines     │  TensorRT-LLM
              ├──────────────┤
              │  Frameworks  │  PyTorch, Transformers,
              │  & Libraries │  Diffusers, ONNX
              ├──────────────┤
              │  CUDA /      │  Kernels, cuBLAS,
              │  Drivers     │  FlashAttention
              └──────────────┘
         Lower abstraction, more control

CUDA: The Foundation Layer

CUDA is NVIDIA’s programming platform for GPU computation. A CUDA kernel is a function written to run in parallel across thousands of GPU threads. The ecosystem includes several critical libraries:

• cuBLAS — Optimized kernels for linear algebra, especially general matrix multiplication (GEMM).
• cuDNN — Primitives for common neural network operations.
• CUTLASS / CuTe — Template libraries for building custom GPU kernels without writing raw CUDA.
• FlashAttention — A highly specialized attention kernel (tens of thousands of lines of C++) that restructures computation to minimize expensive VRAM round-trips.
• FlashInfer — A collection of GPU kernels specifically optimized for LLM inference workloads.

💡 Key Concept

Kernel fusion merges multiple sequential GPU operations into a single kernel launch, eliminating intermediate reads and writes to VRAM. For example: multiplying by 2 then multiplying by 3 requires four memory operations (read-compute-write-read-compute-write). Fusing them into a single multiply-by-6 requires only two (read-compute-write).

PyTorch’s Role in Inference

PyTorch dominates as the framework for both training and inference. Its key inference optimization is torch.compile, which analyzes your computation graph and selects optimized kernels for your specific GPU architecture.

⚠️ Watch Out

torch.compile cannot fuse third-party kernels like FlashAttention or DeepGEMM. Its value is greatest for non-standard architectures or chains of lightweight operations. For mainstream LLM architectures, inference engines provide more impactful optimizations.

How model weights get stored and loaded:

• safetensors — The standard format. Contains raw tensor data with no executable code (eliminating a class of security risks). Supports memory-mapped loading for fast startup.
• ONNX — Bundles weights with a computation graph. Designed for portability across different hardware and runtime targets.
• GGUF — A binary format widely used for local and quantized inference, native to llama.cpp and Ollama.

Inference Engines: vLLM vs SGLang vs TensorRT-LLM

All three implement the same core optimizations: continuous batching, quantization, speculative decoding, prefix caching, parallelism, and prefill-decode disaggregation. Community adoption as of early 2026: vLLM leads with ~50K+ GitHub stars, roughly double SGLang (~15K) and TensorRT-LLM (~12K).

🔧 Try It Yourself — Run vLLM

# Install vLLM (requires CUDA 11.8+ and Linux/WSL2):
pip install vllm

# Serve a small model (4B fits in 12 GB VRAM):
vllm serve Qwen/Qwen3-4B --dtype float16

# In another terminal, query it:
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B",
    "messages": [{"role": "user", "content": "What is the KV cache?"}],
    "max_tokens": 200
  }'

# Note: 8B models may need --dtype bfloat16 --max-model-len 2048
# to fit within 12 GB. Experiment with --gpu-memory-utilization 0.9

Note: vLLM requires Linux or WSL2. On native Windows, use Ollama or llama.cpp instead.

🔧 Try It Yourself — Run llama.cpp on Mac

# Install llama-cpp-python with Metal support:
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

# Or use Ollama (easier):
ollama run llama3.2:3b

# For more control, download a GGUF model directly:
# Visit huggingface.co and search for "GGUF" versions

LLM Optimization Techniques: Quantization, Speculation, Caching, and Parallelism

These are the core techniques that separate naive inference from production-grade serving. Each targets a different bottleneck.

Quantization: Trading Precision for Speed and Memory

Analogy: Quantization is like saving a photo as JPEG instead of RAW. The file gets much smaller, it looks almost the same to the human eye, but you lose some fine detail. For models, “fine detail” means subtle differences between weight values — and for most tasks, those tiny differences don’t affect output quality.

In concrete terms: a model stored in FP16 uses 2 bytes per weight. Quantizing to INT4 uses just 0.5 bytes per weight — the model takes 4x less memory. A 7B model that needed 14 GB in FP16 fits in ~3.5 GB at INT4. That’s the difference between “won’t fit on my GPU” and “runs comfortably.”

The core idea is straightforward: represent model weights (and optionally activations and KV cache) using fewer bits. Native training typically uses FP16 or BF16 (16 bits). Quantization drops that to FP8, INT8, FP4, or INT4.

Why it works so well for inference:

• Prefill (compute-bound): Lower-precision Tensor Cores deliver up to 2x the FLOPS of their higher-precision counterparts.
• Decode (memory-bound): Half the bytes per weight means effectively double the memory bandwidth utilization.
• In practice: Expect ~30–50% throughput improvement per precision step. The theoretical 2x doesn’t fully materialize due to quantization/dequantization overhead.

Number formats used in practice:

Floating-point formats preserve dynamic range through their exponent bits — they can represent both very large and very small values, which matters for outlier weights that disproportionately affect quality. Integer formats sacrifice this range for simpler hardware implementation.

What to quantize, ordered by risk:

1. Weights (linear layers) — Safest target. Start here.
2. Activations — Moderate quality risk, but significant throughput gains.
3. KV Cache — Moderate risk, with large benefits for memory-constrained long-context workloads.
4. Attention (softmax outputs) — Highest risk. Quantization errors here compound across every subsequent token. Usually left at full precision.

🔧 Try It Yourself — Compare quantization levels with Ollama

# Ollama uses GGUF quantization behind the scenes.
# Compare a full model vs. quantized version:

# Run a larger model that only fits quantized on 12 GB VRAM:
ollama run qwen3:14b
# This auto-selects Q4_K_M quantization (~8 GB) to fit in 12 GB.
# Notice the speed vs. running qwen3:4b at full precision.

# Test quality — ask both models the same question and compare:
# "What is the capital of Australia and why was it chosen?"

🔧 Try It Yourself — Quantize a model with AutoGPTQ

# Requires CUDA (Linux/WSL2):
# pip install auto-gptq optimum transformers

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name = "Qwen/Qwen2.5-1.5B"
quant_config = BaseQuantizeConfig(bits=4, group_size=128)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_pretrained(model_name, quant_config)

# Prepare calibration data (a few sample texts):
examples = [tokenizer("Inference engineering optimizes AI model serving.",
            return_tensors="pt")]
model.quantize(examples)
model.save_quantized("./qwen-1.5b-gptq-4bit")
print("Quantized model saved!")

Speculative Decoding: Using Spare Compute to Skip Ahead

Since LLM decode is memory-bound, the GPU’s compute units sit partially idle during token generation. Speculative decoding puts that spare compute to work: a small, cheap model drafts several tokens ahead, and the full model verifies them all in a single forward pass.

  1. Small "drafter" generates: ["The", "quick", "brown", "fox"]
  2. Target model validates all 4 in ONE forward pass
  3. Accepts "The", "quick", "brown" ✓  Rejects "fox" ✗
  4. Target generates its own token "dog" instead
  5. Result: 4 tokens for 1 forward pass (instead of 4 passes)

The key constraint: verification must produce mathematically identical outputs to what the target model would have generated on its own. This isn’t an approximation — it’s provably lossless.

Common speculation strategies:

⚠️ Watch Out

Speculation delivers the biggest wins at low batch sizes where compute capacity goes unused. At high concurrency, the GPU is already saturated and the overhead of running a drafter hurts more than it helps — engines must disable speculation dynamically. Additionally, high temperature reduces acceptance rates because the drafter struggles to predict creative outputs.

KV Caching and Prefix Caching

Within a single request, every inference engine caches KV pairs — that’s table stakes. The real optimization opportunity is reusing cached KV data across requests.

Prefix caching exploits the fact that many requests share identical beginnings — system prompts, document context, or shared instructions. If the engine recognizes a matching prefix, it skips prefill for those tokens entirely:

Request 1: "You are a helpful assistant. What is Python?"
                └── Cached prefix ──┘ └── Unique ──┘

Request 2: "You are a helpful assistant. What is Rust?"
                └─ Hit! Skip prefill ─┘ └── Unique ──┘

Practical tip: Structure your prompts so the variable parts come last. “Weather in Paris?” and “Weather in Tokyo?” share the prefix “Weather in “. But “Paris weather?” and “Tokyo weather?” share nothing — the variable token comes first.

Where cached KV data can live, ordered by access speed:

Model Parallelism: When One GPU Isn’t Enough

Large models that exceed a single GPU’s VRAM require splitting across multiple devices. The right strategy depends on your hardware topology:

💡 Key Concept

Estimating GPU requirements: Min GPUs = (model_params × bytes_per_param × 1.8) / VRAM_per_GPU

The 1.8× multiplier accounts for KV cache and runtime overhead beyond the raw weights. Example: DeepSeek-V3 (671B params) at FP8 (1 byte/param): 671 × 1 × 1.8 / 192 = ~6.3 → Round up to 8 GPUs (1 node of B200s)

Beyond LLMs: Inference for Vision, Speech, and Image Generation

Generative model inference falls into two broad patterns:

1. Autoregressive token generation — LLMs, VLMs, embedding models, speech recognition, text-to-speech
2. Iterative denoising — Image and video generation via diffusion models

Vision Language Models (VLMs)

VLMs like Qwen-VL and Mistral Large 3 pair a vision encoder (typically ~2B parameters) with a standard LLM backbone. Images get converted into ~1000 tokens that are prepended to the text input. The primary inference challenge: these extra tokens inflate the KV cache proportionally.

Speech Recognition (ASR) and Text-to-Speech (TTS)

ASR — OpenAI’s Whisper (1.55B parameters) dominates open-source speech recognition. It uses an encoder-decoder architecture where the decoder is an autoregressive transformer — meaning all the LLM optimization techniques (quantization, KV caching, batching) apply directly.

• Real-time use: Target 200ms round-trip latency per audio chunk. WebSocket streaming is the standard approach.
• Batch processing: Parallelizing chunk processing across multiple GPU partitions can transcribe 1 hour of audio in under 4 seconds (~1000× real-time).

TTS — Modern text-to-speech models like Orpheus TTS are built on top of LLM architectures (e.g., fine-tuned Llama 3.2 3B). They generate discrete audio tokens that get decoded into waveforms. For real-time playback without gaps, you need 80–100 TPS sustained.

🔧 Try It Yourself — Run Whisper locally for speech transcription

# pip install faster-whisper

from faster_whisper import WhisperModel

# On NVIDIA GPU — use float16 for GPU acceleration:
model = WhisperModel("large-v3", device="cuda", compute_type="float16")

# On Mac M-series — use CPU with int8 (MPS not yet supported):
# model = WhisperModel("large-v3", device="cpu", compute_type="int8")

# Transcribe an audio file:
segments, info = model.transcribe("your_audio.mp3")
print(f"Language: {info.language} (prob: {info.language_probability:.2f})")
for seg in segments:
    print(f"[{seg.start:.1f}s → {seg.end:.1f}s] {seg.text}")

# Measure RTF (Real-Time Factor):
import time
start = time.time()
segments, info = model.transcribe("your_audio.mp3")
for _ in segments: pass  # consume the generator
elapsed = time.time() - start
rtf = info.duration / elapsed
print(f"RTF: {rtf:.0f}x real-time")

Image and Video Generation

Image generation models aren’t single models — they’re pipelines chaining a text encoder, a denoiser, and a VAE decoder. Generation works through iterative denoising: start from pure noise and progressively refine it over 30–50 steps.

• Each step involves two forward passes (one with the text prompt, one without — this is classifier-free guidance). A 50-step generation means 100 forward passes total.
• Unlike LLM decode, this workload is compute-bound — the GPU’s math units are the bottleneck, not memory bandwidth.
• Useful trick: Classifier-free guidance contributes most in the early steps. Disabling it after the first ~80% of steps saves roughly 20% of compute with negligible quality difference.

🔧 Try It Yourself — Generate images with Stable Diffusion XL

import torch
from diffusers import StableDiffusionXLPipeline

# SDXL base fits in 12 GB VRAM with float16:
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

# Enable memory-efficient attention:
pipe.enable_xformers_memory_efficient_attention()

# Generate an image:
image = pipe(
    prompt="A futuristic GPU datacenter, cinematic lighting, 4K",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]
image.save("datacenter.png")

On Mac M-series, replace .to("cuda") with .to("mps") and expect slower generation. For faster results, try stabilityai/sdxl-turbo with only 4 steps.

Production Deployment: Containers, Autoscaling, and Observability

Containerizing Model Servers

Production inference services run inside Docker containers — self-contained environments with pinned dependencies. Key practices:

• Pin every version explicitly — CUDA toolkit, PyTorch, inference engine. The AI stack has frequent breaking changes across versions.
• Minimize image size — only include what the server actually needs.
• Start from official base images — vLLM, SGLang, and NVIDIA all publish maintained base images.
• NIMs (NVIDIA Inference Microservices) provide pre-packaged containers for common models.

Autoscaling

Autoscaling adjusts the number of serving replicas based on demand. Kubernetes handles the orchestration. The parameters you tune:

How requests get batched:

Static:     Wait for full batch → run all at once → SLOW start
Dynamic:    Wait for batch OR timeout → run → better
Continuous: Process tokens as they come, swap requests in/out → BEST

Every modern inference engine uses continuous batching — processing at the token level rather than waiting for complete requests. This maximizes GPU utilization across requests with different lengths.

Cold starts are the latency tax on scaling up: allocating a GPU, pulling the container image, loading model weights into VRAM, and compiling the engine. Mitigation strategies: use quantized weights (smaller to transfer), cache compiled engine artifacts, keep container images lean, and maintain warm replica pools.

Deployment Strategies and Monitoring

Rolling out model updates without downtime:

• Blue-green: Maintain two identical environments, switch traffic atomically. Reliable but expensive — requires 2× the GPUs during transitions.
• Canary (usually preferred): Route a small percentage of traffic to the new version, increase gradually while monitoring metrics. Roll back instantly if something degrades. Pairs well with autoscaling to avoid doubling infrastructure costs.

What to watch in a production inference deployment:

• Request volume and HTTP status codes (2XX, 4XX, 5XX)
• Latency distributions: TTFT, TPS, end-to-end at P50/P90/P99
• GPU utilization percentage and VRAM consumption
• Replica count and request queue depth
• Input and output sequence length distributions

💡 Key Concept

The economics shift when you self-host. API pricing ($/million tokens) is simple to model. Dedicated GPU pricing ($/GPU-hour) depends on batch sizes, traffic patterns, sequence lengths, and utilization rates. Before committing, estimate your total weekly cost under both models — the crossover point is often later than you’d expect.

🔧 Try It Yourself — Monitor your local inference

# While running Ollama or vLLM, monitor GPU in real-time:
nvidia-smi dmon -s pucvmet -d 1
# Shows: power, utilization, clocks, VRAM, memory BW, temperature
# every 1 second. Send queries and watch utilization spike!

# Or use gpustat:
pip install gpustat
gpustat --watch

# Send multiple concurrent requests to see batching in action:
import asyncio, aiohttp, time

async def query(session, prompt):
    start = time.time()
    async with session.post("http://localhost:11434/api/generate",
        json={"model": "qwen3:4b", "prompt": prompt}) as r:
        await r.read()
    return time.time() - start

async def main():
    async with aiohttp.ClientSession() as s:
        tasks = [query(s, f"Count to {i*10}") for i in range(5)]
        times = await asyncio.gather(*tasks)
        print(f"Latencies: {[f'{t:.1f}s' for t in times]}")

asyncio.run(main())

Hardware Lab: Complete Setup Guide

NVIDIA GPU Setup (Windows / Linux)

# 1. Install NVIDIA drivers (latest Game Ready or Studio driver)
#    https://www.nvidia.com/Download/index.aspx

# 2. Install CUDA Toolkit 12.x
#    https://developer.nvidia.com/cuda-downloads

# 3. Install Python 3.10+ and create a virtual environment:
python -m venv inference-lab
inference-lab\Scripts\activate    # Windows
# source inference-lab/bin/activate  # Linux

# 4. Install PyTorch with CUDA:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

# 5. Install key packages:
pip install transformers diffusers accelerate
pip install faster-whisper
pip install huggingface_hub

# 6. Install Ollama:
#    https://ollama.com/download/windows

# 7. (Optional) Install WSL2 for Linux-only tools like vLLM:
#    wsl --install

# Verify:
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Apple Silicon Mac Setup

# 1. Install Homebrew (if not already):
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# 2. Install Python 3.10+:
brew install python@3.12

# 3. Create a virtual environment:
python3 -m venv inference-lab
source inference-lab/bin/activate

# 4. Install PyTorch (MPS backend for Apple Silicon):
pip install torch torchvision torchaudio

# 5. Install key packages:
pip install transformers diffusers accelerate
pip install faster-whisper
pip install huggingface_hub

# 6. Install Ollama:
brew install ollama

# 7. (Optional) Install llama.cpp with Metal:
brew install llama.cpp

# Verify:
python3 -c "import torch; print('MPS:', torch.backends.mps.is_available())"

Model Size Reference: VRAM Requirements

Rule of thumb: FP16 VRAM ≈ 2 × parameter count in GB. Q4 ≈ 0.5–0.6 × parameter count in GB.

Further Resources and References

Key Papers

• “Attention Is All You Need” (Vaswani et al., 2017) — The foundational transformer architecture.
• FlashAttention (Dao et al., 2022–2025) — Memory-efficient attention through tiling and recomputation.
• PagedAttention / vLLM (Kwon et al., 2023) — Virtual memory management for KV cache.
• EAGLE (Li et al., 2024) — Speculative decoding via hidden-state prediction.
• DeepSeek-V3 (DeepSeek AI, 2024) — MoE architecture with Multi-Latent Attention.

Tools

• Ollama — Easiest way to run models locally (macOS, Windows, Linux).
• llama.cpp — C++ inference engine optimized for CPU/Metal/CUDA.
• vLLM — Production inference engine (Linux/WSL2).
• faster-whisper — CTranslate2-optimized Whisper for speech recognition.
• ComfyUI — Visual pipeline builder for image generation.

Communities

• Hugging Face — Models, datasets, and spaces
• r/LocalLLaMA — Reddit community for local inference
• NVIDIA GTC talks — Free recordings on YouTube

Recommended Reading

• “Inference Engineering” by Philip Kiely (Baseten Books, 2026) — Comprehensive reference for engineers building production inference systems.

Frequently Asked Questions

What is inference engineering?

Inference engineering is the practice of optimizing how generative AI models run in production — reducing latency, cutting costs, and improving reliability while maintaining output quality. It spans GPU hardware, low-level kernel optimization, inference engine configuration, and production infrastructure including autoscaling and monitoring.

What is the difference between training and inference?

Training is the process of learning model weights from data — computationally expensive and done infrequently. Inference is using those learned weights to generate outputs for real user requests. Training is a batch job that runs on large GPU clusters; inference is a live service that must respond quickly and stay available around the clock.

What are TTFT and TPS in LLM inference?

TTFT (Time to First Token) is the delay before a user sees the first generated token. It’s determined by the prefill phase, which is compute-bound. TPS (Tokens Per Second) measures the streaming rate after the first token appears. It’s determined by the decode phase, which is memory-bandwidth-bound. Both should be measured at P50, P90, and P99 percentiles rather than as averages.

How much VRAM do I need to run an LLM locally?

The quick formula: FP16 VRAM ≈ 2× the parameter count in GB. A 7B model needs ~14 GB at full precision. With 4-bit quantization, that drops to roughly 0.5–0.6× the parameter count — so a 14B model fits in about 8 GB. An RTX 3060 with 12 GB handles 4B–8B models at full precision or up to 14B quantized.

What is the KV cache and why does it matter?

During autoregressive generation, the model computes key and value vectors for each token’s attention. The KV cache saves these vectors so they aren’t recomputed from scratch for every new token — reducing per-step complexity from O(n²) to O(n). The downside: the cache grows linearly with sequence length and is a major consumer of VRAM, sometimes exceeding the model weights themselves for long contexts.

What is quantization and when should I use it?

Quantization stores model weights (and optionally activations and KV cache) in lower-precision number formats — for example, FP8 or INT4 instead of FP16. This reduces memory usage proportionally and typically yields 30–50% throughput improvement per precision step. Quality impact is minimal for most tasks when starting with weight-only quantization.

What is speculative decoding?

A technique that uses a small draft model to predict several tokens ahead, then verifies the entire batch in one forward pass of the main model. Tokens that match are accepted for free; mismatches are corrected. This exploits idle GPU compute during the memory-bound decode phase and can produce 3–8 tokens per forward pass at low batch sizes.

How do vLLM, SGLang, and TensorRT-LLM compare?

vLLM has the widest model support and largest community. SGLang performs well with MoE architectures. TensorRT-LLM squeezes out the highest raw throughput but only runs on NVIDIA hardware and requires more configuration effort. All three share the same core feature set: continuous batching, quantization, speculative decoding, and prefix caching.

What does “7B parameters” mean in an LLM?

A 7B model contains 7 billion learnable parameters — numerical values adjusted during training to encode patterns in language. More parameters generally means greater capacity for knowledge and reasoning, but also higher memory and compute requirements. At FP16 precision (2 bytes per parameter), loading a 7B model requires approximately 14 GB of VRAM.

What are FLOPS and why do they matter for inference?

FLOPS (Floating-point Operations Per Second) quantifies a GPU’s raw computational throughput. LLM inference is fundamentally matrix multiplication, so a GPU’s FLOPS rating directly affects how quickly it can process tokens. TFLOPS = trillion operations/second; PFLOPS = quadrillion. Higher FLOPS particularly benefits the prefill phase (processing input prompts), which is compute-bound. The decode phase is more constrained by memory bandwidth.

Do I need a powerful GPU to get started?

No. Any NVIDIA GPU with 6–8 GB of VRAM or an Apple Silicon Mac can run small models locally. Ollama makes setup trivial for models in the 1.5B–4B range. Quantization extends your hardware further — a 14B model quantized to 4-bit needs only about 8 GB. You can explore all the core concepts in this tutorial on consumer hardware.

When should I switch from API-based inference to self-hosted?

Consider self-hosting when you have enough request volume that per-GPU-hour pricing beats per-token API pricing, when you need fine-tuned or custom models, when you require tighter latency guarantees than a shared API can offer, or when you’re building multi-model pipelines. Before switching, model your total weekly cost under both approaches — include GPU utilization rates, batch efficiency, and traffic variability.