vLLM

The open-source inference engine for serving LLMs at high throughput on your GPUs.

vLLM is an inference and serving engine for large language models: PagedAttention manages the KV cache in pages, like virtual memory, and continuous batching keeps the GPU saturated, multiplying throughput compared with a naive generation loop. It exposes an OpenAI-compatible API, loads Hugging Face models, and supports quantization (FP8, AWQ, GPTQ), tensor parallelism, prefix caching and multi-LoRA. It's our choice when an open-weights model has to be served in production, on your own infrastructure, at a controlled cost.