Running Local LLM Inference on a Cloud Mac Mini: Memory Planning and Quantization

A reader running edge-AI experiments asked me last week: he wanted to try Llama 3 8B and a 13B code model, but his laptop didn't have enough memory, and he was hesitant to buy a Mac Studio without first knowing whether he'd actually need that much power. His question was whether he could rent a cloud machine first to figure out the right tier. This is actually a pretty common scenario — so this post walks through the full process of running local LLM inference on a cloud Mac mini: memory planning, environment setup, and quantization selection.

Why Reverse-Engineer Model Size from Unified Memory Bandwidth

Apple Silicon's CPU, GPU, and Neural Engine share the same pool of unified memory, so there's no overhead from shuffling data between VRAM and system RAM. That's a natural advantage for inference workloads — but it also means memory capacity is the hard ceiling on how large a model you can run. A rough rule of thumb: a model trained in FP16, once quantized to 4-bit, occupies roughly 0.55–0.6× its parameter count in GB, plus whatever the KV cache and system overhead add on top.

This table is just a starting reference — how large a model you can actually run depends on whether Xcode, a browser, or other memory-hungry apps are running at the same time. Before your inference process crashes, check how much free memory is left in top — don't wait until an OOM error to start troubleshooting.

Environment Setup: llama.cpp vs. MLX

The two paths don't conflict with each other — you can install both on the same machine and switch depending on the task.

llama.cpp: Prioritizing Compatibility

brew install cmake
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j
./build/bin/llama-cli -m models/llama-3-8b-q4_k_m.gguf \
  -p "写一段解释递归的代码示例" -n 256 --ctx-size 4096

-DGGML_METAL=ON is the key flag here — enabling the Metal backend is what lets you use the full GPU. Without it, llama.cpp falls back to CPU-only inference, which is several times slower.

MLX: Apple's Native Framework

python3 -m venv mlx-env
source mlx-env/bin/activate
pip install mlx-lm
mlx_lm.generate --model mlx-community/Llama-3-8B-Instruct-4bit \
  --prompt "解释一下 KV cache 的作用" --max-tokens 256

MLX hosts quantized weights directly under the mlx-community namespace, so there's no need to convert formats yourself — getting started is faster than with llama.cpp. That said, the ecosystem and community troubleshooting resources around llama.cpp are still more mature at this point.

Choosing a Quantization Format

Lower isn't always better when it comes to quantization — the right choice depends on the task:

• Q4_K_M: Sufficient for everyday Q&A and code completion. It has the smallest memory footprint and is the default choice for most use cases.
• Q5_K_M: For long-chain reasoning or multi-step logical tasks, bump up to this tier — output stability improves noticeably.
• Q8_0: Near-original precision, suited for accuracy-critical evaluation scenarios. Memory usage roughly doubles, so only consider it when you have plenty of headroom (24GB+).

The way to decide is simple: run a batch of questions you'd actually ask using Q4 first, and manually check the answer quality. If you notice obvious logical leaps or numerical errors, step up one tier at a time — there's no need to max out precision (and memory) from the start.

Disk Space and Snapshot Management

Model files can range from a few GB to tens of GB, so plan your cleanup before your rental period ends:

1. Keep all downloaded models under a single ~/models/ directory, so you can clean up or migrate them as a whole.
2. Use du -sh ~/models/* to regularly check disk usage, and delete outdated quantized versions promptly to avoid filling up the system disk and slowing down inference.
3. If you need to keep your experiment environment configuration (virtual environments, lists of downloaded weights), package it into a tarball and download it locally before your rental expires — the system disk won't retain any data once the rental period ends.

du -sh ~/models/* | sort -rh | head -5
tar czf mlx-env-backup.tar.gz mlx-env models/*.json

Benchmarking and Monitoring

Keep a terminal window open to watch resource usage while running inference — it lets you catch memory pressure before it becomes a problem:

sudo powermetrics --samplers gpu_power -i 1000 -n 5

Watch the GPU utilization and power curve — if it stays high for a long time while responses are still slow, that's usually a sign the KV cache is saturating memory bandwidth. In that case, shorten --ctx-size or switch to a smaller quantization tier, rather than blindly increasing batch size.

Frequently asked questions