The fastest way to run Mixtral 8x7B on Apple Silicon Macs

Show HN: The fastest way to run Mixtral 8x7B on Apple Silicon Macs

The fastest way to run Mixtral 8x7B on Apple Silicon Macs operates in show hn launch.

Profile

I’d originally launched my app: Private LLM[1][2] on HN around 10 months ago, with a single RedPajama Chat 3B model. The app has come a long way since then. About a month ago, I added support for 4-bit OmniQuant quantized Mixtral 8x7B Instruct model, and it seems to outperform Q4 models at inference speed and Q8 models at text generation quality, while consuming only about 24GB of RAM[3] at 8k context length. The trick is: a) to use a better quantization algorithm and b) to use unquantized embeddings and the MoE gates (the overhead is quite small).Other notable features include many more downloadable models, support for App Intents (Siri, Apple Shortcuts), on-device grammar correction, summa

Its activity falls under show hn launch; the profile below is assembled from public directory data.

The company is listed at the Show HN launch stage of development.

Website: privatellm.app/[2]: