Local AI on Your Device: Hands‑On with Ollama & LM Studio

Parameters are tiny “knobs” a model uses to represent patterns in language. More knobs (7B → 13B → 70B) often means better reasoning but higher memory and slower responses.

• 7–8B: Great on laptops (e.g., Llama 3 8B Instruct, Qwen2.5 7B Instruct).
• 13B: Better reasoning; needs more memory (Llama 3 13B Instruct).
• 70B: Very capable; usually impractical locally.

Models read/write in tokens (small text chunks). The context window is the model’s short‑term memory—your instructions + any pasted docs + the model’s reply. If you exceed it, older parts fall off.

Example: A 4k token window fits roughly 2.5–3k words combined (prompt + answer).

Quantization compresses a model so it fits and runs faster—like saving a high‑res photo as a smaller JPEG. You lose a bit of fidelity but gain speed and lower memory use.

• Start with Q4 for balance on laptops.
• Tight memory? Try Q3.
• More headroom and quality? Try Q5/Q8.

Flavors like Q4_K_M are just different compression strategies.

Precision describes how detailed the model’s numbers are. FP16/BF16 are higher‑detail (common in GPU stacks). INT8/INT4 are smaller/faster (common in local GGUF files).

GGUF is for llama.cpp tools (Ollama, LM Studio). safetensors is for PyTorch/Transformers; not directly loaded here. For this guide, choose GGUF.

Base models are book‑smart but not very obedient. Instruct/Chat models are tuned to follow directions—use these for everyday tasks and assistants.

• Fine‑tuning: Teach a specialty (e.g., legal summaries).
• Instruction‑tuning: Improves following directions.
• Distillation: Big teacher → small student; fast yet capable.
• Adapters (LoRA/QLoRA): Snap‑on skill modules; QLoRA works on quantized bases.

Embeddings turn text into vectors; a vector DB (Chroma/FAISS) finds similar chunks. RAG retrieves those chunks and feeds them to the model, grounding answers in your sources for fewer hallucinations.

Temperature controls creativity (lower = safer, higher = more adventurous). top_p and top_k limit candidate next words. For strict formats like JSON, lower temperature helps.

Models must fit in memory, and while generating they build a KV cache that grows with prompt and output length. Longer contexts and replies use more memory.

• 8 GB RAM: 7–8B at Q4, keep context 2–4k tokens.
• 16 GB RAM: 7–13B at Q4 is comfortable.
• 32 GB RAM: 13B at Q4/Q5; larger if you accept slower speeds.

Local is private, offline, and cost‑predictable. Cloud offers the biggest, most capable models. Many prototype and handle private data locally, then use cloud for the toughest reasoning.

• Have 10–20 GB free disk space for multiple models.
• Close heavy apps to free RAM before loading models.
• If Python isn’t installed: Windows (Microsoft Store or python.org), macOS (Xcode tools or pyenv), Linux (distro package).