If you want an AI assistant to work with private files, run offline, or avoid per-token API fees, you can run a large language model (LLM) on your own computer. The model files and the program that generates responses stay on your computer as long as you do not connect the setup to a cloud service.
Local inference means your computer processes each prompt and generates the answer. Once you download a model, you can use it offline. Response quality and speed depend on your hardware, and smaller local models lag behind leading cloud models on difficult reasoning tasks.
Choose a model your computer can handle
Start with available memory, especially graphics card memory, called VRAM. The model’s weights, which store what it learned during training, must fit in memory along with the conversation context. Longer prompts and chats need additional memory.
The model’s precision affects its file size and memory use. Quantization stores weights in fewer bits to reduce file size and memory needs. For the common Q4_K_M format, plan on about 0.6 to 0.7 GB of memory per billion parameters, plus room for context and other overhead. Use that estimate as a starting point; actual memory use varies with model architecture and context length, and also depends on the software. The local hardware guide discusses these estimates and their trade-offs.
As a rough choice, a 7B or 8B model is a reasonable place to start on a computer with 16 GB of memory. A GPU with 16 GB of VRAM can handle some 13B models entirely in graphics memory, while larger models need more capacity or run partly on system RAM. Apple Silicon Macs use unified memory shared by the processor and graphics hardware, so total memory matters. A model that spills from VRAM into system RAM still runs, but it generates responses more slowly.
Quantization also affects output quality. Q4_K_M is a common balance between size and quality; stronger compression, especially below Q3, makes coherence problems more noticeable. For everyday tasks such as summarizing a document or drafting a short code snippet, a small model may be enough. Check important answers, since smaller models are more prone to hallucinations, or confident statements that are wrong.
Install a local runner and try a model
A local runner downloads model files and provides a way to chat with them. With Ollama, you can download and run models from the command line. Its setup guide lists macOS, Linux, and Windows through WSL2 as supported environments, and recommends 16 GB of RAM; a compatible GPU helps but is optional. See the Ollama setup instructions before installing.
After installing a runner, choose a model that fits your memory. Follow the runner’s instructions to download it and start chatting. Keep the first test simple: ask for a summary of a short text or a draft of a small function. If responses are slow or the computer struggles, try a smaller model or a shorter context. Close memory-heavy programs before testing if you are using a laptop with limited RAM.
Many local runners support GGUF files, a model format used by llama.cpp. llama.cpp supports inference on CPUs, GPUs, or both. Mixing processors helps run models that exceed VRAM, but layers handled by the CPU reduce speed. Check the file format and hardware requirements listed by the runner before downloading a model.
Weigh privacy, cost, and quality
Local inference keeps prompt processing on your computer when the runner operates locally and does not send requests to an outside service. That can help when working with confidential documents or in places with unreliable internet. It does not make an organization automatically compliant with privacy laws or internal rules: access controls, physical security, and audit practices still matter.
After the initial hardware purchase, local use avoids per-token API charges, though electricity and storage still cost money. If you use AI only occasionally, buying hardware may cost more than using a cloud service. If you need frequent access, offline use, or tighter control over where prompts go, running locally may be a better fit.
Check a model’s license before using it for business. “Open source” is sometimes used loosely: some models publish their weights but impose conditions on commercial use, attribution, or redistribution. Read the specific model license before using its output or incorporating the model into a product. The overview of model licensing explains the distinction between open-source software and models with publicly available weights.