How to Run an LLM Locally: A Beginner's Guide
Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.
Approximately 4 min read
Running an LLM locally means loading model weights into a runtime on your own computer and using that runtime to generate responses. Your first goal is simple: run a small instruction model, get a coherent answer and confirm which hardware is doing the work.
Choose one interface for that first test. Once it works, larger models, longer context and application integrations become easier to evaluate because you have a baseline.
Choose a first runtime
| Tool | Good starting point when you want | What you manage |
|---|---|---|
| LM Studio | Graphical model discovery and chat | Model selection and application settings |
| Ollama | Simple model management and a local service | Supported model tags and service settings |
| llama.cpp | Direct GGUF and backend control | Builds or binaries, files and launch options |
LM Studio suits readers who want to download and chat through a desktop application. Ollama offers a compact command-line workflow. llama.cpp exposes more of the loading, offload and inference settings, which makes it useful for learning how the runtime behaves.
The tools overlap. Pick the one that matches the work you want to do rather than installing all three before trying a model.
Check the model and memory requirements
Start with a small instruction-tuned model supported by the runtime. A base model is trained for continuation and may need additional prompting or adaptation to behave like a chat assistant. The model card should identify its intended use, format, language coverage and recommended settings.
Check both system RAM and GPU memory. Weights occupy only part of the runtime budget; context and working buffers add more. A model that spills into host RAM may run much more slowly than a smaller one that stays on the GPU.
If downloading a GGUF, select one compatible quantization and keep its complete filename. For split artifacts, all shards must be present. The VRAM guide explains the budget in more detail.
Run a small first test
Ollama’s basic pattern is ollama run followed by a supported model name. For example:
ollama run llama3.2
For llama.cpp, a local GGUF can be opened with:
llama-cli -m model.gguf -cnv -c 4096 -n 256
Use the actual path to your downloaded file. If you are building llama.cpp yourself, the executable is typically under the build’s bin directory; the build-and-run tutorial walks through that setup.
In a graphical application, select the model, load it and open a new conversation. Start with a short prompt such as “Explain why the sky looks blue in three paragraphs.” Check that the response is coherent, generation finishes and memory remains stable.
Confirm placement and performance
On an NVIDIA machine, nvidia-smi shows GPU memory and active processes. The runtime’s loader logs explain which layers or tensors were placed on the GPU. Check both rather than inferring placement from whether a GPU is installed.
Record these first-run details:
| Item | Example of what to keep |
|---|---|
| Artifact | Repository, revision and exact filename |
| Runtime | Version, backend and launch command |
| Context | Configured limit and actual prompt length |
| Speed | Prompt processing and generation separately |
| Memory | GPU and host use during the request |
Time to first token and generation speed answer different questions. A slow start can come from model loading or prompt processing even when later tokens arrive quickly.
Improve the baseline one change at a time
First choose a useful model and a context that fits your normal tasks. Then compare one change: a neighbouring quantization, a longer context or a different offload setting. Keep the test prompt stable while measuring speed and memory.
Use the model’s recommended sampling and chat template. A template mismatch can damage answers even when the hardware is working correctly. Changing ten launch flags at once makes it difficult to identify which change helped.
Connect applications deliberately
Many runtimes can expose a local HTTP API for editors, chat interfaces or scripts. Start with loopback access and confirm a basic request before connecting a larger application. Internet access, file reading and tool execution are features of the surrounding application; they are not automatically supplied by loading model weights.
Local execution gives you more control over data flow, but applications can still download files, collect telemetry or contact external tools. Check the application’s settings and network behaviour for the privacy requirements of your task. Once the first path is dependable, add integrations one at a time.