How We Run 60 Hugging Face Models on 2 GPUs

Show HN: How We Run 60 Hugging Face Models on 2 GPUs

How We Run 60 Hugging Face Models on 2 GPUs is a company in show hn launch.

Profile

Most open-source LLM deployments assume one model per GPU. That works if traffic is steady. In practice, many workloads are long-tail or intermittent, which means GPUs sit idle most of the time.We experimented with a different approach.Instead of pinning one model to one GPU, we: •Stage model weights on fast local disk •Load models into GPU memory only when requested •Keep a small working set resident •Evict inactive models aggressively •Route everything through a single OpenAI-compatible endpointIn our recent test setup (2×A6000, 48GB each), we made ~60 Hugging Face text models available for activation. Only a few are resident in VRAM at any given time; the rest are restored when needed.Col

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

It is recorded at the Show HN launch stage.

Website: m.youtube.com/watch?v=IL7mBoRLHZkLive