Running Local LLMs on Apple Silicon: An Agentic Workflow for M5 Max

Why run LLMs locally?

Cloud LLM APIs are convenient, but they come with latency, cost per token, rate limits, and data leaving your network. For agentic coding workflows, where an AI assistant makes dozens of tool calls per minute, local inference eliminates all of these constraints. With Apple Silicon's unified memory architecture, a single MacBook Pro can serve models that rival cloud APIs at interactive speeds.

This post documents the infrastructure I built to run local LLMs on an M5 Max as a persistent service powering my development workflow.

The hardware

The key insight with Apple Silicon: unified memory means the GPU can access the full 128 GB without PCIe bottlenecks. A model that would require a multi-GPU server on NVIDIA hardware fits entirely in a single laptop's memory.

The inference stack

I use llama.cpp (build b9430) with the Metal backend. It provides:

• Near-theoretical memory bandwidth utilization
• OpenAI-compatible API via llama-server
• Support for quantized models (Q4 through Q8)
• Flash Attention and KV cache quantization
• Speculative decoding (MTP and draft models)

The server runs as a launchd service, always on and always available on the local network.

Model selection: why MoE dominates on Apple Silicon

Token generation on Apple Silicon is entirely memory-bandwidth bound. The theoretical ceiling is:

tg_max = bandwidth / active_parameters_in_bytes

This means Mixture-of-Experts (MoE) models crush dense models on this hardware. An MoE model with 35B total parameters but only 3B active per token achieves the same generation speed as a 3B dense model while having the quality of a much larger model.

The difference is dramatic:

Dense models are 3 to 6x slower for the same quality tier. On Apple Silicon, always prefer MoE.

Benchmark results

I benchmarked 8 models across prompt processing (512 to 65,536 tokens) and token generation (128 to 1,024 tokens). Here are the top 5 ranked by overall usability for coding, documentation, and troubleshooting:

All five models generate faster than most people can read. At 82 to 99 tokens per second, the output feels instantaneous for coding tasks.

Context scaling: the hidden differentiator

Raw generation speed doesn't tell the full story. Some models look fast at short context but collapse at longer sequences. I measure "context scaling retention," the percentage of pp512 speed the model retains at pp65536:

Qwen3-30B-A3B is the cautionary tale: fastest generation (103 t/s) but completely unusable for long agent sessions because prompt processing collapses. Always benchmark at your actual working context length.

The agentic workflow

The local llama-server exposes an OpenAI-compatible API, accessible from any machine on my private network. I configure my coding assistant to use it as a provider:

{
  "provider": {
    "local-llama": {
      "type": "openai",
      "url": "http://<server-ip>:8080/v1",
 
      "models": {
        "qwen-35b-fast": {
          "id":            "qwen-35b-fast",
          "name":          "Qwen 3.6 35B-A3B (local)",
          "contextLength": 131072,
          "supportsTools": true
        }
      }
    }
  }
}

This gives me:

• Zero latency to first token, with no network round trip to a cloud API
• No rate limits because the model handles back to back requests instantly
• 128k context so entire codebases fit in a single session
• Full privacy since code never leaves the network
• Zero cost because after hardware, inference is free

Deployment as a persistent service

The server runs via macOS launchd. It starts at boot, restarts on crash, and requires zero maintenance:

launchctl load ~/Library/LaunchAgents/com.local.llama-server.plist

Key configuration flags:

Memory budget

With 128 GB unified memory and ~115 GB available for GPU workloads:

There's enough room to run a second model simultaneously for complex reasoning tasks, such as the 122B model at reduced context alongside the 35B.

What I learned

1. MoE architecture is essential for Apple Silicon. Dense models can't compete on generation speed.
2. sudo purge between benchmarks is critical. Without it, residual memory from prior model loads can halve performance and produce misleading results (I initially measured Qwen 3.6 at 28 t/s instead of the real 99 t/s due to this).
3. Don't set iogpu.wired_limit_mb on M5 Max. The default (0) gives you the most GPU memory. Setting any value reduces available memory.
4. Context scaling varies dramatically between models with identical architectures. Always benchmark at your target context length before deploying.
5. Token generation stays above 75 t/s at tg1024 for all top 5 models. Long form output (documentation, reports) is fast enough to be practical.

What's next

I'm evaluating several candidate models:

• Qwen 3.6 35B-A3B with MTP (Multi-Token Prediction): same model but with prediction heads that draft 2 to 3 tokens per forward pass. Expected 1.5 to 1.8x generation speedup.
• Qwen3-Coder-30B-A3B: a coding specialized MoE. Need to verify context scaling doesn't collapse like its predecessor.
• Qwen3-Coder-Next-80B-A3B: SWE-bench 70.6%. Speed already known (~90 t/s), evaluating coding quality.

The goal is always the same: find the fastest model that doesn't compromise on quality for the specific workload. For agentic coding, that means fast generation, reliable context scaling, and good tool calling capabilities.

All benchmarks run on Apple M5 Max, 128 GB unified memory, llama.cpp b9430, Metal backend. Full reproducibility instructions and raw data available on request.