Ollama runs models locally on macOS, Windows and Linux, and has an official Docker image. It exposes a REST API and OpenAI-compatible endpoints for chat, completions and embeddings. By default it listens only on the local machine, and its default context window is small, so we raise it when the use case needs more. It handles one request per model at a time by default, which you can raise.
vLLM is an open-source library for LLM inference and serving. It manages attention memory with PagedAttention and uses continuous batching, so it can serve many requests at once. Its server provides OpenAI-compatible endpoints, can require an API key and can split a model across several GPUs with tensor parallelism. It runs on NVIDIA and AMD GPUs and on CPUs, with other hardware through plugins.
Hugging Face hosts the model weights. vLLM loads models from the Hugging Face Hub by default, and lists the Llama and Mistral families among its supported models. Each model card shows a license. Some models are gated: you must log in, accept the author's terms and send an access request before you can download them.