Local AI

How to Run an LLM Locally: A Beginner's Guide

Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.

Approximately 4 min read

Running an LLM locally means loading model weights into a runtime on your own computer and using that runtime to generate responses. Your first goal is simple: run a small instruction model, get a coherent answer and confirm which hardware is doing the work.

Choose one interface for that first test. Once it works, larger models, longer context and application integrations become easier to evaluate because you have a baseline.

Choose a first runtime

Tool Good starting point when you want What you manage
LM Studio Graphical model discovery and chat Model selection and application settings
Ollama Simple model management and a local service Supported model tags and service settings
llama.cpp Direct GGUF and backend control Builds or binaries, files and launch options

LM Studio suits readers who want to download and chat through a desktop application. Ollama offers a compact command-line workflow. llama.cpp exposes more of the loading, offload and inference settings, which makes it useful for learning how the runtime behaves.

The tools overlap. Pick the one that matches the work you want to do rather than installing all three before trying a model.

Check the model and memory requirements

Start with a small instruction-tuned model supported by the runtime. A base model is trained for continuation and may need additional prompting or adaptation to behave like a chat assistant. The model card should identify its intended use, format, language coverage and recommended settings.

Check both system RAM and GPU memory. Weights occupy only part of the runtime budget; context and working buffers add more. A model that spills into host RAM may run much more slowly than a smaller one that stays on the GPU.

If downloading a GGUF, select one compatible quantization and keep its complete filename. For split artifacts, all shards must be present. The VRAM guide explains the budget in more detail.

Run a small first test

Ollama’s basic pattern is ollama run followed by a supported model name. For example:

ollama run llama3.2

For llama.cpp, a local GGUF can be opened with:

llama-cli -m model.gguf -cnv -c 4096 -n 256

Use the actual path to your downloaded file. If you are building llama.cpp yourself, the executable is typically under the build’s bin directory; the build-and-run tutorial walks through that setup.

In a graphical application, select the model, load it and open a new conversation. Start with a short prompt such as “Explain why the sky looks blue in three paragraphs.” Check that the response is coherent, generation finishes and memory remains stable.

Confirm placement and performance

On an NVIDIA machine, nvidia-smi shows GPU memory and active processes. The runtime’s loader logs explain which layers or tensors were placed on the GPU. Check both rather than inferring placement from whether a GPU is installed.

Record these first-run details:

Item Example of what to keep
Artifact Repository, revision and exact filename
Runtime Version, backend and launch command
Context Configured limit and actual prompt length
Speed Prompt processing and generation separately
Memory GPU and host use during the request

Time to first token and generation speed answer different questions. A slow start can come from model loading or prompt processing even when later tokens arrive quickly.

Improve the baseline one change at a time

First choose a useful model and a context that fits your normal tasks. Then compare one change: a neighbouring quantization, a longer context or a different offload setting. Keep the test prompt stable while measuring speed and memory.

Use the model’s recommended sampling and chat template. A template mismatch can damage answers even when the hardware is working correctly. Changing ten launch flags at once makes it difficult to identify which change helped.

Connect applications deliberately

Many runtimes can expose a local HTTP API for editors, chat interfaces or scripts. Start with loopback access and confirm a basic request before connecting a larger application. Internet access, file reading and tool execution are features of the surrounding application; they are not automatically supplied by loading model weights.

Local execution gives you more control over data flow, but applications can still download files, collect telemetry or contact external tools. Check the application’s settings and network behaviour for the privacy requirements of your task. Once the first path is dependable, add integrations one at a time.

Sources and further reading

Local AI

Ollama and llama.cpp: It's Complicated

A source-driven history of the increasingly uncomfortable relationship between Ollama and llama.cpp: upstream engineering, attribution disputes, forks, compatibility failures, community backlash, and the cost of hiding the engine.

By MapleKernel ·