I share these notes publicly so others can examine my workflows. They are records of what I do, not tutorials or instructions. I do not recommend replicating anything without doing your own due diligence.
I run llama.cpp as my local inference server and I prefer it over everything else. I have run it on my laptop before and loved the speed, but without Homebrew the installation was never straightforward. Then the llama.cpp team released precompiled binaries and I switched right away. Looking back, it was worth it. With the right flags, llama.cpp is magic.
To install it on my Mac Studio I:
- downloaded the macOS Apple Silicon arm64 binary from the llama.cpp GitHub releases page
- extracted the llama-b10816-bin-macos-arm64.tar.gz archive into my Downloads directory
- ran xattr -dr com.apple.quarantine . in the terminal. This told macOS not to flag it as malicious software
Notes
llama.cpp GitHub releases: https://github.com/ggml-org/llama.cpp/releases
File name: llama-b10816-bin-macos-arm64.tar.gz
Version: I am running llama-b10816 (version 0.4). The developers release new versions constantly.
xattr -dr com.apple.quarantine .
macOS attaches a quarantine attribute to downloaded executables. Without this command, Apple's security system would block the files and prevent them from running.