The LLM Field Manual: A Complete Guide to Language Model Terminology

I am not an expert in machine learning or natural language processing. This is a working reference I built for myself while experimenting with local LLMs on Apple Silicon. Treat it as field notes from a practitioner, not a textbook. If something is wrong or out of date, I would genuinely appreciate the correction.

What is an LLM?

A Large Language Model (LLM) is a neural network trained on massive amounts of text to predict the next token in a sequence. The "large" refers to the number of trainable parameters, typically ranging from a few billion to hundreds of billions. These parameters encode patterns in language: grammar, facts, reasoning strategies, coding ability.

LLMs are autoregressive text generators. Given a prompt, they produce one token at a time, feeding each generated token back into the model to produce the next. Everything else (chat interfaces, tool calling, code completion) is built on top of this mechanism.

The Transformer architecture

Nearly all modern LLMs use the Transformer architecture (2017). Know these components:

Attention variants

The attention variant determines KV cache size and inference speed. Know the differences:

Bottom line: GQA with 8 KV groups uses roughly 4x less memory for its KV cache than MHA. This is why newer models handle much longer context windows without running out of memory.

Positional encoding

Transformers have no built-in sense of token order. Positional encoding adds position information so the model distinguishes "the cat sat" from "sat the cat."

When a model advertises "128k context" but trained on 8k, it uses RoPE scaling. Quality at extended lengths depends on the scaling method and whether the model was fine-tuned at the longer length.

Flash Attention

Flash Attention computes attention in tiles that fit in fast on-chip SRAM instead of materializing the full attention matrix. Faster, less memory, same output.

In practice: enable Flash Attention in llama.cpp with the -fa flag. Required for KV cache quantization. Recommended for any context above 4k tokens.

# Enable Flash Attention on a server
llama-server \
  -m model.gguf \
  -fa

Parameters and model sizing

Parameter count is the primary size metric. Parameters are the weights and biases learned during training.

Quick math: each parameter in fp16 takes 2 bytes. A 70B model needs ~140 GB just for weights before quantization.

Tokens and tokenization

Models process tokens (subword units), not raw text. A tokenizer handles the conversion.

Rule of thumb: 1 token ≈ 3/4 of an English word. 128k tokens ≈ 96,000 words ≈ 300 pages.

Dense vs. Mixture of Experts (MoE)

Two architectures to know:

Dense models activate every parameter for every token. A 70B dense model reads all 70 billion parameters per token. High quality, slow inference.

MoE models have many parameters but only activate a subset ("experts") per token. A router picks which experts fire.

Reading the name: XB-AYB = X billion total, Y billion active per token. Qwen3 35B-A3B has 35B total but activates 3B per token, giving 3B-class speed with much higher quality.

Why this matters on Apple Silicon: LLM inference is memory bandwidth bound. MoE models only read their active parameters from memory each token, so a 35B-A3B model achieves the throughput of a 3B dense model. On bandwidth-rich Apple Silicon (400+ GB/s), this is the optimal architecture for local inference.

Quantization

Quantization reduces weight precision from the original 16 bits per parameter to fewer bits. Smaller model, less memory, faster inference, minimal quality loss.

Precision formats

Quantization methods and formats

How to read a quantization name

QK breaks down as:

• Q = quantized
• = approximate bits per weight (4, 5, 6, 8)
• K = K-quant method (importance-based mixed precision)
• = S (small), M (medium), L (large), XL (extra-large). Larger = more bits on important layers.

Q5_K_M = 5-bit, K-quant method, medium variant.

GGUF vs. MLX: model formats for Apple Silicon

Two ecosystems for running models on Mac: GGUF (llama.cpp) and MLX (Apple's framework). Same goal, different approaches.

Decision rule: for inference speed and model serving, use llama.cpp with GGUF. For experimentation, fine-tuning, or custom pipelines in Python, use MLX.

Start a llama.cpp server with an OpenAI-compatible API:

# Basic server (auto-detects Apple Silicon Metal backend)
llama-server \
  -m model.gguf \
  --port 8080
 
# With all the recommended flags for production use
llama-server -m model.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  -fa \
  --jinja \
  -c 32768 \
  -ctk q8_0 \
  -ctv q8_0

Test the API with curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }'

Many power users keep both: llama.cpp for daily model serving, MLX for trying architectures or quick fine-tunes.

Fine-tuning

Fine-tuning continues training a pretrained model on a specific dataset to change its behavior.

Methods

Alignment and preference tuning

After pretraining, models are aligned to be helpful, harmless, and honest:

Model stages

Distillation

Knowledge distillation trains a smaller "student" model to mimic a larger "teacher" model. The student learns from the teacher's output probabilities, which carry richer information than simple correct/incorrect labels.

When a model card says "distilled from" or "trained on synthetic data from" a larger model, this is what happened. Many Llama and Qwen variants are distilled from larger models in the same family or from proprietary models.

Chat templates and prompt formatting

LLMs require specific chat templates that structure conversations into roles (system, user, assistant). Wrong template = garbage output, even from a good model.

In practice: always use the --jinja flag in llama.cpp for automatic template detection. Manual template specification is error-prone. Let the model's embedded metadata handle it.

# Auto-detect the chat template from model metadata
llama-server \
  -m model.gguf \
  --jinja
 
# Override the system prompt
llama-cli -m model.gguf \
  --jinja \
  -sys "You are a helpful coding assistant." \
  -p "Write a Python function to merge two sorted lists."

Reasoning and thinking models

Reasoning models produce an internal chain of thought before answering. More tokens spent thinking, better answers on hard problems.

When to use thinking: complex coding, math, multi-step reasoning. When to skip it: simple questions, creative writing, straightforward lookups. The thinking overhead is wasted on easy tasks.

Tool calling and agentic use

Tool calling lets the model invoke external functions by generating structured output (usually JSON). The host executes the function and returns results.

Check before relying on it: not all models handle tool calling well. It requires specific training. Look for explicit "tool calling" or "function calling" in the model card.

RAG (Retrieval-Augmented Generation)

RAG augments the model's prompt with retrieved documents before generation. Reduces hallucination, enables access to private or current data, no fine-tuning required.

Multimodal and vision models

VLMs (Vision Language Models) process both text and images. Some handle audio, video, or other modalities.

For local inference, VLMs work with llama.cpp (--mmproj for vision adapter) and MLX. Expect higher memory usage than text-only models due to the vision encoder.

# Run a vision model (e.g., Gemma or Qwen-VL)
llama-cli -m vision-model.gguf \
  --mmproj vision-encoder.gguf \
  --jinja \
  --image photo.jpg \
  -p "What is in this image?"

Licenses

Model licenses determine what you can do with the weights. Check before deploying.

"Open weights" does not mean "open source." Always check the model card on Hugging Face before commercial deployment.

Inference terminology

Inference = running a trained model to generate output. Know these metrics and concepts.

Speed metrics

Speculative decoding and MTP

Standard inference generates one token per forward pass (sequential bottleneck). These techniques draft multiple tokens to break through it.

Key property: speculative decoding never changes output quality (rejected drafts are discarded). Costs extra memory for the draft model or heads. On memory-rich Apple Silicon, almost always worth enabling.

# Speculative decoding with a draft model (1.5 to 2.5x speedup)
llama-server -m large-model.gguf \
  -md small-draft-model.gguf \
  --draft-max 16 \
  --draft-min 4 \
  -ngl 99 \
  -fa \
  --jinja

Sampling parameters

The model outputs a probability distribution over its vocabulary. These parameters control token selection:

# Control sampling in llama-cli
llama-cli -m model.gguf \
  --jinja \
  --temp 0.7 \
  --top-p 0.9 \
  --top-k 40 \
  --min-p 0.05 \
  --repeat-penalty 1.1 \
  -p "Explain quantum computing in simple terms."
 
# Greedy decoding (deterministic, best for code/factual tasks)
llama-cli -m model.gguf \
  --jinja \
  --temp 0 \
  -p "Write a function that reverses a linked list."

Memory concepts

# Full GPU offload with KV cache quantization for long context
llama-server -m model.gguf \
  -ngl 99 \
  -fa \
  -c 131072 \
  -ctk q8_0 \
  -ctv q8_0
 
# Partial offload when the model doesn't fully fit in memory
llama-server -m huge-model.gguf \
  -ngl 20 \
  -fa \
  -c 8192

Local inference software

Model naming conventions

Official names

Format: Organization ModelFamily-Size-Variant

Reading examples:

• Llama-3.1-70B-Instruct = Meta, Llama v3.1, 70B params, instruction-tuned
• Qwen3-235B-A22B = Qwen v3, 235B total / 22B active (MoE)
• Gemma-4-27B-IT = Google Gemma v4, 27B params, Instruction Tuned

Community quantization names

Format: ModelName-Quant.gguf

Example: Qwen3-35B-A3B-UD-Q5_K_XL.gguf

Decoded: Qwen3, 35B total / 3B active, Unsloth Dense quantization, Q5 K-quant extra-large variant, GGUF format.

The Hugging Face ecosystem

Hugging Face is the central hub for open-source models. Know these conventions:

Download a GGUF model from Hugging Face:

# Install the CLI if you haven't
pip install huggingface-hub
 
# Download a specific quantization
huggingface-cli download \
  unsloth/Qwen3-30B-A3B-GGUF \
  Qwen3-30B-A3B-UD-Q4_K_XL.gguf \
  --local-dir ./models

Benchmarks and evaluation

Putting it all together

When someone says "I'm running Qwen3 35B-A3B UD-Q5_K_XL on llama.cpp with Flash Attention and q8_0 KV cache at 128k context," decode it:

• Qwen3: Qwen family, version 3
• 35B-A3B: 35 billion total, 3 billion active per token (MoE with GQA)
• UD-Q5_K_XL: Unsloth Dense quantization, ~5.5 bits per weight, extra-large variant, GGUF format
• llama.cpp: inference engine, C/C++ with Metal backend on Apple Silicon
• Flash Attention: tiled attention for memory efficiency, enabled with -fa
• q8_0 KV cache: attention cache compressed to 8-bit, halving its memory footprint
• 128k context: up to 128,000 tokens per session, using RoPE for position encoding

That setup: ~25 GB in memory, ~99 t/s on an M5 Max, entire codebase fits in context. Once you know the terminology, you can evaluate any model configuration at a glance.

The full command:

llama-server \
  -m Qwen3-35B-A3B-UD-Q5_K_XL.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -ngl 99 \
  -fa \
  --jinja \
  -c 131072 \
  -ctk q8_0 \
  -ctv q8_0

Terminology current as of mid-2026. The field moves fast. When in doubt, check the model card on Hugging Face and the docs for your inference engine.