Private LLM development: language models on your own servers

Open-weight language models can run on hardware you control, so prompts and data stay in your environment. We pick a model whose license fits your use, size the hardware, serve it with Ollama or vLLM behind an OpenAI-compatible endpoint, and connect it to your application.
Self-hosted language models
  • Open-weight models from Hugging Face, license checked
  • Ollama for development and small deployments
  • vLLM for serving many users
  • OpenAI-compatible endpoints your code already speaks

When a private LLM is the right choice

Self-hosting suits products where control matters more than having the strongest model:

  • Data must not leave your environment. Ollama's FAQ says that when you run locally, Ollama does not see your prompts or data.
  • The system must work offline or on an isolated network.
  • Your volume is steady and high, and you would rather pay for hardware you run than per request.
  • You want to fix the model version, so behavior does not change unless you change it.

When a private LLM is not the right choice

Running models yourself is real operations work. Plan around these:

  • You need the quality of the strongest hosted models. Open-weight models you can run on your own hardware may not match them for your task, so test first.
  • Your volume is small or irregular. Hardware sits idle, and a hosted API is usually simpler.
  • No one can operate GPU servers. Someone has to patch drivers, watch memory, update models and handle failures.
  • The model's license does not allow your use. Some licenses restrict commercial use or set their own terms, so read them before you build.

What we build with private LLMs

Typical projects, described in general terms.

Local prototypes

A model running through Ollama on a developer machine or a single server, to test a use case before any hardware is bought.

Production serving

vLLM on your GPU servers or cloud account, sized for the number of users and the context length you need.

OpenAI-compatible endpoints

A self-hosted endpoint that existing OpenAI client code can call by changing the base URL and model name.

Private retrieval

Assistants that answer from your documents with both the search index and the model inside your environment.

Locked-down deployments

Models on an internal network, with API keys on the endpoint and no outside calls.

Model and license review

A short list of candidate models, their license terms and a test on your own questions before you commit.

Ollama, vLLM and Hugging Face

Ollama runs models locally on macOS, Windows and Linux, and has an official Docker image. It exposes a REST API and OpenAI-compatible endpoints for chat, completions and embeddings. By default it listens only on the local machine, and its default context window is small, so we raise it when the use case needs more. It handles one request per model at a time by default, which you can raise.

vLLM is an open-source library for LLM inference and serving. It manages attention memory with PagedAttention and uses continuous batching, so it can serve many requests at once. Its server provides OpenAI-compatible endpoints, can require an API key and can split a model across several GPUs with tensor parallelism. It runs on NVIDIA and AMD GPUs and on CPUs, with other hardware through plugins.

Hugging Face hosts the model weights. vLLM loads models from the Hugging Face Hub by default, and lists the Llama and Mistral families among its supported models. Each model card shows a license. Some models are gated: you must log in, accept the author's terms and send an access request before you can download them.

What a self-hosted model needs beyond the model

  • License review: licenses on the Hub range from Apache 2.0 and MIT to model-specific community licenses and custom terms. We read the actual terms for your use before the model is chosen.
  • Memory sizing: the model weights must fit in GPU memory with room left for the context of every request in flight. Ollama's FAQ notes that memory grows with the number of parallel requests times the context length. Quantized models need less memory but may lose some precision, and a shorter maximum context length or a smaller batch also saves memory.
  • Chat templates: vLLM needs a chat template in the model's tokenizer configuration for chat requests. Without one, chat calls fail until a template is supplied.
  • Security: the endpoint is not public by default in Ollama. When we open it to other servers, we put it on a private network with authentication in front. In vLLM the API key covers the main API paths, not every endpoint, so the network matters too.
  • Testing on your task: we run your real questions against two or three candidate models before choosing one, and keep that set for each model update.
  • Operations: model files, drivers and server versions need updates, and someone has to watch memory and queue length. That work is part of the cost of running your own model.

How we work

  1. Decide whether self-hosting fits: data rules, volume and the quality you need, compared honestly with a hosted API
  2. Shortlist models and check licenses: candidate open-weight models tested on your own questions, with their license terms read
  3. Prototype, then size the hardware: Ollama for a quick prototype, then vLLM sized for your users and context length
  4. Hand over properly: the serving setup, model versions and endpoint security documented, with notes on reading memory and queue problems

Frequently asked questions

Can we keep our data completely inside our own environment?

Yes, if the model, the search index and the application all run on your servers or cloud account. Ollama's FAQ says it does not see your prompts or data when you run locally, and it has a setting to turn off its cloud features.

Will our existing OpenAI code work with a self-hosted model?

Usually with small changes. Ollama and vLLM both offer OpenAI-compatible endpoints, so you change the base URL and the model name. Some options are not supported. Ollama, for example, lists chat parameters it does not support, so we test the calls you use.

How much GPU memory do we need?

It depends on the model size, whether it is quantized, the context length and how many requests run at once. We test the chosen model with your expected load before you buy or reserve hardware. vLLM can also spread a model across several GPUs.

Is Ollama enough for production?

For a small internal tool, it can be. It handles one request per model at a time by default, and you can raise that. For many users at once, vLLM is built for high-throughput serving with continuous batching.

How can we work together?

There are three ways, on the same terms:

  • Dedicated team, best for ongoing work: interview the developers, then they work on your project full or part time.
  • Fixed scope, best when requirements are clear: we agree the scope and the price up front, and deliver against it.
  • Time and material, best when the scope will change: you pay for the time used and can change priorities as the work goes on.

Whichever you choose, we sign an NDA before work starts, you own the code we write, and every change is reviewed in your repository.

Can I interview the developers before we start?

Yes. For a dedicated team you meet the developers in an online call and interview them before they join your project.

Thinking about running a model on your own servers?

Tell us what data is involved, how many users and what the model must do. We will say whether self-hosting fits and what it takes to run.

Tell us about your project

Tell us what you need built, fixed or improved. We come back with questions, a suggested approach and an estimate.