LLM Inference Engineering: Theory, Practicals and Research

Trace every step from tokenization through the forward pass to the autoregressive decoding loop

Master KV cache mechanics — chunked prefill, prefix caching, compression, H2O, and StreamingLLM

Understand why inference is fundamentally different from training and requires its own engineering discipline

Learn SMs, tensor cores, memory hierarchy, and how to read a roofline mode

Identify whether your workload is compute-bound or memory-bound: and know exactly how to fix it

Apply this analysis to make informed decisions about model size, batch size, and hardware selection

Implement FP16, BF16, INT8, INT4 quantization using GPTQ, AWQ, and GGUF

Understand the precision-performance-quality tradeoffs that matter at serving time

Learn when quantization breaks model quality and how to detect it before production

Deploy with vLLM (PagedAttention, continuous batching), SGLang (RadixAttention, structured generation), and TensorRT-LLM

Implement FlashAttention and kernel fusion for IO-aware attention computation

Use Ray Serve and Megatron-LM for distributed serving and model parallelism at scale

Implement draft-target decoding, n-gram prediction, EAGLE, and Medusa for real speedups

Design production serving systems with cold start optimization, canary deployments, and cache-aware routing

Build structured output pipelines using JSON schema enforcement, logit biasing, and guided decoding

Run inference on your laptop using llama.cpp and MLX on Apple Silicon

Deploy quantized models on Raspberry Pi 4 and compare INT4 vs INT8 on ARM

Benchmark GPU vs CPU throughput on Jetson Orin Nano with TensorRT-LLM