If you want to run an AI model on your own computer or server, choose one that fits the task, available memory, and privacy needs. A local large language model (LLM) generates responses on hardware you control. Many local options are open-weight, meaning you can download their trained weights. Their licenses may still restrict some uses.
Start with a small set of models and test them on real examples before buying hardware. For example, a 7B-parameter model can handle everyday writing, summarization, and retrieval on some computers. A larger model may respond too slowly or leave too little memory for other work. Local inference also does not guarantee that data stays on your device. Downloads and plugins can still send information elsewhere. Web search and external storage can do the same.
Choose a model for the task
The best model depends on what you need it to do. A smaller model may be enough for drafting, summarizing, classifying, or answering questions about a document collection. For difficult, open-ended reasoning, the largest cloud models often remain stronger than local options. Set expectations around the specific work you want to improve, then compare candidate models using representative prompts and judge accuracy, clarity, and speed.
For everyday conversation, look for an Instruct or Chat version. These models have been adapted to follow instructions. A Base model is primarily trained to continue text and is generally intended for further training or specialized use. A model card is the documentation published with a model. It describes the model’s capabilities, limitations, recommended prompt format, and license. Check it before downloading, especially if the model will support commercial work. License terms can vary between versions of the same model family and between distributors, as described in this overview of model files.
For coding, choose according to the kind of help you want. A model tuned for autocomplete may be a better fit for completing code as you type, while a coding agent needs to follow instructions and use tools. One comparison of local coding models suggests Qwen3-Coder 30B-A3B as a general-purpose starting point, Codestral for autocomplete and fill-in-the-middle tasks, and Devstral Small 2 for lighter coding agents. Treat these as candidates to test, not universal rankings: direct, same-task benchmark comparisons are limited.
Check memory before choosing model size
Model size affects memory use and speed. Quantization stores a model’s weights at lower precision. This often makes the files much smaller, with some loss in output quality. As a rough estimate, a 4-bit quantized model needs more memory as its parameter count grows, though actual needs vary. These estimates are for planning, not a full account of hardware needs. Longer conversations and other work also use memory. The local LLM hardware overview summarizes these approximate ranges.
For a 7B model in the Q4_K_M quantization format, one guide estimates about 6 GB of video memory (VRAM) or 8 GB of unified memory, which is shared by the processor and graphics hardware. The computer also needs memory to run its operating system and other programs, as well as to hold conversation context. With CPU-only inference, available system memory and memory bandwidth affect speed; if the model spills onto disk, responses slow sharply. CPU inference suits smaller models or tasks that do not need quick responses. For interactive use with models around 7B or larger, a GPU generally makes a substantial difference.
A practical choice is to test the smallest model that can complete your task well, then move up only if it fails. GPU memory affects which quantized models can fit, and larger models generally require more memory. Apple Silicon Macs use unified memory, so some models can run without a separate graphics card. Hardware recommendations depend on model size, conversation length, and the number of people using the model at once. Test those conditions before spending money.
Pick software that fits your computer
A model file needs software, called a runtime, to load it and generate responses. For a straightforward desktop setup, Ollama offers command-line installation and model management, while LM Studio provides a graphical interface for browsing and downloading models. Both make it easier to run models locally. The local inference setup overview describes these options.
GGUF is a widely used format for running quantized models locally. The llama.cpp runtime supports hardware acceleration through Apple Metal and NVIDIA CUDA. It also supports Vulkan, which covers many AMD and Intel graphics cards. Compatibility still depends on the specific computer and software versions, so confirm support before choosing a model format. For a production service that handles many requests, vLLM is commonly used to process requests at higher throughput. Ollama is simpler for personal or small-scale use, according to this self-hosting overview.
If you are replacing a cloud API in existing code, expect some integration work. A local service may use a different base address, model name, streaming behavior, or error format. Test the full request path instead of assuming that changing a model name is enough.
Treat privacy and upkeep as part of the choice
Local inference keeps prompts and responses on your hardware only if the rest of the setup does too. Model downloads and update checks can still send data outside your device. Web search and plugins can do the same, as can logs and external storage. Check where prompts, documents, and logs go, and disable connections you do not need. For sensitive documents, check that the document search index enforces the same access permissions as the source files.
A local model can help with data residency, but it does not automatically satisfy privacy laws or industry rules. Requirements such as lawful use, data minimization, retention limits, and access controls still apply. Local hosting also needs an owner who updates the runtime and model, checks performance, and reruns quality tests after changes.
Retrieval-augmented generation (RAG) lets a model search documents and use relevant passages in an answer. It does not update the model’s underlying weights; changing those weights requires a separate process called fine-tuning. When an answer from a document collection is wrong, check how the documents were split, whether the index is current, whether the right passages were retrieved, and whether permissions were applied. These problems can matter more than which model you choose.
Compare total cost with the cloud
Local inference avoids per-token API charges, but hardware is only part of the expense. Electricity, maintenance, monitoring, storage, and staff time also count. Compare those costs over several years with cloud costs. Include how many people will use the service and how much work they will send to it. Self-hosting tends to make sense when data must stay within a controlled environment. It also makes sense when usage is high enough to justify the hardware or when you need direct control over logging and model changes.
Cloud services still offer quick setup and access to the largest models without an upfront hardware purchase. A hybrid approach can keep sensitive document work local while sending less sensitive tasks to a cloud API. Choose based on the data and workload, then verify the arrangement with a small pilot.