How to self-host an LLM on your own GPU server
Hosting your own LLM comes down to five decisions: which model, how much VRAM, which runtime, how your application reaches it, and who else can. This guide walks through each one with the commands.
Key takeaways
- Self-hosting an LLM means running an open-weight model on a server you control and calling it through an API, instead of sending prompts to a model provider.
- Size the server by VRAM. Weights take about the parameter count times the bytes per parameter: a 70-billion-parameter model is about 35 GB at 4-bit and about 140 GB at 16-bit. Add headroom for context.
- Pick the runtime by load: Ollama for the simplest setup and a few users, vLLM when an application or many users call the model at once.
- Both runtimes speak the OpenAI API, so existing client code usually needs only a new base URL and model name.
- Never expose the model port to the internet without authentication. Keep it on localhost and use an SSH tunnel, a VPN or a reverse proxy with TLS.
On this page
What does it mean to self-host an LLM?
Self-hosting an LLM means running an open-weight language model on a server you control and calling it through an API, instead of sending your prompts to a model provider. The model file sits on your disk, the GPU in your server does the work, and your application talks to an endpoint that only you decide who can reach.
People do it for three reasons: the data may not leave their control, the load is steady enough that a fixed monthly cost beats a per-token bill, or they want a specific model that does not change under them. If none of those applies, a hosted API is simpler.
The whole job is five decisions, in this order.
1. Choose the model
Pick an open-weight model whose license allows your use. Models come in sizes counted in parameters, from a few billion to several hundred billion, and usually in several precisions: the original 16-bit weights and smaller quantized versions at 8-bit or 4-bit. Larger models answer better and cost more memory; quantized versions give up a little quality for a lot less memory.
Start with the smallest model that does the job. You can always move up.
2. Size the VRAM
The model has to fit in the GPU's memory. A workable estimate for the weights:
parameters × bytes per parameter, with half a byte at 4-bit, one byte at 8-bit and two bytes at 16-bit.
| Model size | 4-bit | 8-bit | 16-bit |
|---|---|---|---|
| 8 billion parameters | about 4 GB | about 8 GB | about 16 GB |
| 32 billion parameters | about 16 GB | about 32 GB | about 64 GB |
| 70 billion parameters | about 35 GB | about 70 GB | about 140 GB |
On top of the weights comes the context cache, which grows with the length of the context and with every simultaneous request, and the runtime's own overhead. Quantized files also run a little larger than the bare arithmetic. So treat these numbers as the floor, leave real headroom, and measure with your own context length before you rely on a size.
This is why GPU memory matters more than anything else on the spec sheet. A RyzenRDP GPU server has 1, 2, 4 or 8 NVIDIA RTX PRO 6000 Blackwell SE GPUs with 96 GB of VRAM each, so a 70-billion-parameter model at 8-bit fits on a single GPU and its 16-bit weights fit on the two-GPU server.
3. Pick a runtime: Ollama or vLLM
The runtime loads the model onto the GPU and answers requests.
- Ollama is the quickest to a working model: one install command, one command to download and run a model. By default it processes one request per model at a time and queues the others. It suits one person, a small team or a prototype.
- vLLM is a serving engine for load. It batches many simultaneous requests and can split one model across several GPUs. It suits an API that an application or many users call at once.
Both expose an OpenAI-compatible API, so you can start with one and move to the other without rewriting your application. The full comparison is in vLLM vs Ollama.
4. Install it and serve the model
Sign in to the server and confirm the GPU is visible before anything else:
ssh root@YOUR_SERVER_IP
nvidia-smi
With Ollama:
curl -fsSL https://ollama.com/install.sh | sh
ollama run MODEL_NAME
The installer sets up a systemd service that listens on 127.0.0.1:11434. ollama run downloads the
model and opens a chat in the terminal; ollama ps shows whether the model is on the GPU.
With vLLM, following its quickstart:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm serve MODEL_ID --api-key YOUR_SECRET_KEY
vLLM downloads the model and starts an OpenAI-compatible server on http://localhost:8000. If the
model needs more than one GPU, add --tensor-parallel-size with the number of GPUs.
Test it from the server itself:
curl http://localhost:8000/v1/chat/completions \
-H "Authorization: Bearer YOUR_SECRET_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "MODEL_ID", "messages": [{"role": "user", "content": "Say hello."}]}'
5. Connect your application, and nobody else
In an OpenAI client library, set the base URL to your server's endpoint and the model name to the one you loaded. Nothing else in the request changes.
How the application reaches the endpoint is the part to get right:
- Same server: the application calls
http://127.0.0.1:8000/v1directly. Nothing is exposed. - Your own computer: open an SSH tunnel with
ssh -L 8000:127.0.0.1:8000 root@YOUR_SERVER_IPand callhttp://127.0.0.1:8000/v1locally. - Another server or a team: put a reverse proxy with TLS and authentication in front, or connect the machines through a VPN.
Do not bind a model runtime to a public address without authentication. Ollama has no sign-in of its own, and an open endpoint is one that strangers will find and use. If your team wants a chat window instead of an API, add Open WebUI in front of the runtime; it brings its own accounts.
What you are signing up for
A self-hosted model is a service you operate. You apply the operating system updates, watch the disk fill with model files, back up anything you cannot download again, and replace the model when a better one appears. That is the price of control, and for the right workload it is worth paying.
For the server side, the LLM hosting page has the full sizing table and the plans, the apps page lists the tools with their ports, and private AI server covers the case where the data must stay under your control.
FAQ
Can I host my own LLM?
Yes. Any open-weight model can run on your own hardware or a rented GPU server. You need a GPU with enough VRAM for the model, a runtime such as Ollama or vLLM, and a way for your application to reach it. The model's license decides what you may use it for.
How much VRAM do I need to self-host an LLM?
Enough for the weights plus headroom. Weights take roughly the parameter count times the bytes per parameter: an 8-billion-parameter model is about 4 GB at 4-bit, a 32-billion-parameter model about 16 GB at 4-bit or 64 GB at 16-bit, and a 70-billion-parameter model about 35 GB at 4-bit or 140 GB at 16-bit. The context cache comes on top and grows with context length and the number of simultaneous requests.
Is Ollama or vLLM better for self-hosting?
Ollama is simpler: one install command, one command to run a model, and by default one request per model at a time. vLLM is built to batch many simultaneous requests and to split a model across several GPUs, so it suits an API that an application depends on. Both expose an OpenAI-compatible API.
Is self-hosting an LLM less expensive than using an API?
Not automatically. An API charges per token, while a GPU server costs the same every month whether it is busy or idle. Self-hosting pays off when the model works steadily around the clock, or when the data may not be sent to a third party at all.
Can I self-host an LLM without a GPU?
Small quantized models run on a CPU, but slowly: fine for tests and background jobs, not for a responsive chat. For models of tens of billions of parameters at useful speed you need a GPU.
How do I keep a self-hosted LLM private?
Leave the runtime listening on 127.0.0.1, reach it through an SSH tunnel or a VPN, and publish it only through a reverse proxy that adds TLS and authentication. Start vLLM with an API key. Open only the ports you need in the firewall.
Sources
- Ollama: Linux installation and FAQ (install script, default bind address 127.0.0.1:11434, parallel request settings, loading a model across GPUs)
- vLLM: Quickstart (installation,
vllm serve, default address, API key) - Open WebUI: Quick start (the Docker command)
- NVIDIA: RTX PRO 6000 Blackwell Server Edition (GPU memory of the card)