vLLM

vLLM is a fast and easy-to-use library designed for efficient LLM inference and serving. Originally developed at UC Berkeley's Sky Computing Lab, it's now a community-driven project. It achieves high throughput via PagedAttention, which efficiently manages attention key and value memory. Continuous batching optimizes incoming requests. The engine supports fast model execution using CUDA/HIP graphs and offers quantizations like GPTQ, AWQ, INT4, INT8, and FP8. Optimized CUDA kernels, FlashAttention, and FlashInfer integrations contribute to its speed. vLLM offers speculative decoding and chunked prefill. It integrates seamlessly with Hugging Face models and supports various decoding algorithms, including parallel sampling and beam search. Tensor, pipeline, data, and expert parallelism facilitate distributed inference. An OpenAI-compatible API server enables streaming outputs. vLLM supports diverse hardware, including NVIDIA GPUs, AMD CPUs/GPUs, Intel CPUs/GPUs, PowerPC CPUs, Arm CPUs, and TPUs, plus hardware plugins like Intel Gaudi, IBM Spyre, and Huawei Ascend. Prefix caching and Multi-LoRA support are also included.