Olla – Lightweight LLM Proxy for Homelab and OnPrem AI Inference

Show HN: Olla – Lightweight LLM Proxy for Homelab and OnPrem AI Inference

Olla – Lightweight LLM Proxy for Homelab and OnPrem AI Inference is a business in show hn launch.

Profile

We’ve been running distributed LLM infrastructure at work for a while and over time we’ve built a few tools to make it easier to manage them. Olla is the latest iteration - smaller, faster and we think better at handling multiple inference endpoints without the headaches.The problems we kept hitting without these tools:* One endpoint dies > workflows stall* No model unification so routing isn't great* No unified load balancing across boxes* Limited visibility into what’s actually healthy* Failures when querying because of it* We'd love to merge all them into OpenAI queryable endpointsOlla fixes that - or tries to. It’s a lightweight Go proxy that sits in front of Ollama, LM Studio, vLLM or O

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

Directory data places it at the Show HN launch stage.

Website: thushan.github.io/olla/usage/#development-tools-juni