Skip to content
explainroo
AI models explained

Choosing a GPU for AI

Updated 4 min read

With explainroo Text-to-Video Models vs. Code-Rendered Video for Explainers
On this page
  1. Estimate the VRAM your model needs
  2. Match VRAM to the work
  3. Compare speed, software, and power
  4. Decide whether to buy, rent, or use a workstation GPU

For local AI, choose a graphics processing unit (GPU) based on the work you plan to run and the card’s memory. GPU memory, also called video RAM or VRAM, determines whether a model fits on the card. Once the model fits, memory bandwidth, software support, power use, and price determine performance and ease of use.

For large language models (LLMs), start with VRAM. A model that does not fit may fail to load or run partly from slower system memory. For occasional experiments, renting a cloud GPU can cost less than buying hardware. For frequent local use, select a card with enough VRAM for the model and its working memory, then compare performance and total system cost.

Estimate the VRAM your model needs

Model weights take up most of the memory for many LLM workloads, but they are not the only requirement. The GPU also needs space for the KV cache, which stores information used to generate the next tokens, and for software overhead. The cache grows with the length of the conversation and the number of requests handled at once.

As a rough estimate, a model needs about 2 GB of VRAM per billion parameters at FP16, a format that uses 16 bits for each weight. A real 4-bit model file uses roughly 0.5 GB per billion parameters. Add about 1–2 GB for framework overhead, then leave additional room for the KV cache. Requirements vary by model and software. Check the exact model and runtime you plan to use; the LLM VRAM estimates explain how to estimate memory needs.

A large model can still need substantial VRAM at 4-bit quantization. FP16 weights need considerably more. Whether a card has enough memory depends on the model, quantization, cache, and software overhead. More aggressive quantization may reduce memory use, but the result depends on the model file and runtime.

Long context lengths also change the calculation. A Llama 3.1 estimate shows that a long-context request’s KV cache can need substantial memory beyond what the model weights use. If you need long conversations or several simultaneous requests, plan for substantially more VRAM than the weight file requires.

Match VRAM to the work

For local chat, coding assistance, and experimentation, 12–16 GB supports many 7B–13B models at conversational speeds, while 16 GB is a practical target for comfortable use with many 14B models. Around 24 GB gives more room for larger models at lower quantization, though performance and fit depend on the specific setup. These are useful starting points, not guarantees; check the model’s actual memory footprint before buying.

Image generation, video work, and model training also benefit from more VRAM, but their requirements depend on model architecture, image or video dimensions, batch size, and training method. For training, estimate memory for the weights, gradients, optimizer state, and input data rather than applying an LLM inference rule. When possible, test the exact framework and workload on rented hardware before purchasing.

Choose the smallest model that meets your needs, then buy enough VRAM to run it with your intended context length and workload. If a model barely fits, reducing context length or batch size can lower memory use. Quantization also reduces weight storage; reported quality effects vary by model and evaluation, so test output quality for your own use rather than relying on a single benchmark.

Compare speed, software, and power

Once the model fits, memory bandwidth affects how quickly the GPU moves data between memory and the GPU’s processors. This matters for many inference workloads, where the GPU repeatedly reads model weights. NVIDIA’s RTX 5090, for example, has 1.79 TB/s of memory bandwidth compared with 1.01 TB/s for the RTX 4090, according to Puget Systems’ testing. Bandwidth figures alone do not predict performance across different workloads, so look for benchmarks that match your model and software.

NVIDIA is a common choice for local AI because its CUDA software ecosystem has broad support across AI tools. AMD and Intel cards can work for some workloads, but support depends on the software and model runtime. Before buying a non-NVIDIA card, check that the exact framework and features you plan to use support it. The local LLM GPU comparison discusses the trade-offs.

Check that the rest of the computer can support the GPU. A high-end GPU needs a suitable power supply, enough case space, and adequate cooling. The RTX 5090 has a 575 W thermal design power rating, so check the card maker’s power-supply recommendations and your case’s cooling capacity before purchase. Do not assume a newer feature will help immediately: although Blackwell GPUs support FP4, Puget Systems reports that software support for practical end-user use remains limited.

Decide whether to buy, rent, or use a workstation GPU

If you use AI occasionally or need a model that exceeds your local VRAM, renting a cloud GPU avoids a large upfront purchase. Compare the provider’s hourly cost, availability, storage charges, and setup time with how often you expect to run the workload. Prices and access differ by provider, so a general claim about cloud savings cannot replace a calculation for your own usage.

For frequent local inference, a consumer GPU often offers a simpler starting point than a professional accelerator. For large-scale training or high-volume inference, workstation and data-center GPUs provide far more memory and are built for demanding workloads. NVIDIA’s H100 and AMD’s MI300X are options for this market. The MI300X offers 192 GB of memory. Before buying, also consider software maturity and performance for your workload. Machine-learning GPU comparisons cover these options, but benchmarks should match the framework and workload you intend to run.

Free and open source

Let your AI agent make the video

explainroo lets a coding agent like Claude Code or Codex make narrated explainer videos and product demos. The voice, the word timing and the rendering run on your own computer, with no API key and no cost per video. Optional outside services use your own key.

Copy this into your AI agent

Make me a short explainer video about [your topic]. Use explainroo for it: run npx --yes explainroo docs and follow what it prints.

Getting started Example videos GitHub

Made with explainroo: How explainroo makes a video

More on ai models explained

All guides on ai models explained