Tech 08 Jul 2026 8 min read

Local LLMs on Apple Silicon and consumer GPUs

For years, running serious open-weights language models meant renting cloud instances with 80GB Nvidia A100s at $3.50 an hour. Today, an off-the-shelf Mac Studio with unified memory or a desktop equipped with dual RTX 3090s can serve 70B parameter models at interactive reading speeds (20–35 tokens/sec).

The breakthrough was not faster compute silicon; it was the maturation of post-training quantisation formats (GGUF, AWQ, EXL2) and runtime kernels optimised for memory-bandwidth-constrained execution.

Quantisation: where accuracy meets physics

A 70B parameter model in FP16 precision requires 140 GB of VRAM simply to hold the weights. Quantising weights to 4 bits (such as GGUF Q4_K_M or AWQ) compresses that footprint down to ~40 GB, fitting comfortably in a 64GB or 128GB unified memory pool.

Modern 4-bit quantisation does not naively round floats. Advanced methods identify outlier feature channels (which carry disproportionate semantic entropy) and preserve them in full 16-bit precision while quantising the remaining 99% of weights with per-group scaling factors:

# llama.cpp quantization and server invocation example
# Convert and quantize to 4-bit Medium K-quant
./llama-quantize ./models/Meta-Llama-3-70B/ggml-model-f16.gguf \
    ./models/Meta-Llama-3-70B-Q4_K_M.gguf Q4_K_M

# Run high-throughput server pinned to Apple Silicon Metal backend
./llama-server \
    -m ./models/Meta-Llama-3-70B-Q4_K_M.gguf \
    -c 8192 \
    -ngl 99 \
    --threads 8 \
    --host 127.0.0.1 --port 8080

In autoregressive decoding, token generation speed is bounded by memory bandwidth: $\text{Tokens/sec} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$.

Unified memory vs. Discrete PCIe GPUs

Apple Silicon's M-series chips feature unified memory architectures offering between 400 GB/s (Max) and 800 GB/s (Ultra) of memory bandwidth shared directly between CPU and GPU cores. A 40 GB Q4 model streaming across an 800 GB/s bus theoretically yields $\frac{800}{40} = 20$ tokens/sec.

Discrete GPU setups (like two RTX 3090s via PCIe) offer higher peak bandwidth per card (936 GB/s), but face two constraints:

VRAM ceilings: Consumer cards top out at 24 GB, requiring tensor parallelism across cards over PCIe lanes.
Inter-card latency: Without NVLink bridges, inter-GPU activation transfers over PCIe Gen 4 introduce noticeable micro-stutters during prompt ingestion.

When local inference makes sense

Local hosting is not about saving money on trivial workloads; API calls to hosted providers are cheaper for light usage. Local inference wins on data sovereignty (confidential financial or medical codebases), zero network latency for agent loops, and absolute resilience against cloud API rate limits or policy deprecations.

Benchmarking local model inference on custom hardware? Share your setup.