Source available
Ollama Cluster
Ollama Cluster (ocluster) sits in front of however many machines you have running Ollama and gives you one Ollama-compatible endpoint to talk to. It doesn't replace Ollama, it wraps it: model-aware routing, automatic failover, health checks, and a dashboard, so clients never need to know which box actually served the request.
I built it to put some old laptops and a spare mini PC to work. None of them can touch a proper GPU on their own, but AI inference queues in seconds, not milliseconds, so the real constraint is how many requests you can run at once, not how fast any single one finishes. Wire a few small boxes together and you get more usable throughput than chasing another expensive upgrade.
Quick start
Install, register a node, serve
No Ollama Cloud account, no sign-up. Point it at Ollama instances you already run.
# Binary from GitHub Releases, or build with Cargo (Rust 1.88+)
cargo install --git https://github.com/levi-putna/ollama-cluster --bin ocluster
# Point it at one Ollama node, then start the controller
ocluster init --node gpu-01@http://127.0.0.1:11434
ocluster serve
# From another terminal, register more nodes as you find them
ocluster node add gpu-02 --url http://192.168.1.20:11434Why bother
One node, one queue
A web request is measured in milliseconds. An Ollama request is measured in seconds, sometimes a full minute, depending on the model and the prompt. That's fine for one person chatting in a browser tab. It falls apart the moment more than one thing wants the GPU at once: a couple of apps, an agent running in the background, a household sharing the same machine. Requests don't fail gracefully, they queue, and everything behind the first one waits while the GPU sits fully loaded.
A bigger GPU helps the latency of a single request. It does nothing about the fact that you've still got one queue on one node. The other option is horizontal: add more nodes, spread the load, and route each request to whichever one is actually free. That's the whole job of Ollama Cluster.
How it works
A pool of Ollama nodes behind one address
Each node is a normal Ollama install, the same binary you'd download from ollama.com. The cluster discovers whatever models exist on each one, tracks health with passive failure detection and periodic probes, and picks a target per request using model-aware routing: which nodes actually have the model loaded, and which one is least busy right now.
Clients
curl, Open WebUI, agents: anything speaking the Ollama API
ocluster proxy
:11434, one Ollama-compatible endpoint
Routing engine
model-aware, least-busy node wins
Node A
Ollama
Node B
Ollama
Node C
Ollama
Clients don't know or care how many nodes sit behind that address. They talk to :11434 exactly like they would a single Ollama instance, through the same /api/generate, /api/chat, and /api/embed routes.
Failover without touching the client
If a node dies before it starts streaming a response, the pre-stream retry sends the same request to another node instead. A circuit breaker ejects the failed node from rotation until it starts passing health checks again.
Request routed to Node A
Looks healthy at the time
Node A fails before streaming starts
Circuit breaker ejects it from rotation
Retried on Node B
Client sees a slower response, not an error
What you get
0.1.0: the operating MVP
Static node registration, routing, proxying, and enough tooling to actually run the thing day to day.
| Capability | What it does |
|---|---|
| Unified endpoint | One Ollama-compatible address for every client |
| Model-aware routing | Prefers nodes that already have the model loaded, then the least-busy one |
| Failover | Pre-stream retry on another node when a backend fails |
| Health monitoring | Passive failure detection, circuit breakers, node probes |
| Management API | REST endpoints for nodes, models, requests, events, config |
| Terminal dashboard | Live monitoring and node actions without leaving the terminal |
| Web admin panel | Browser dashboard on :11602 for node and model CRUD |
| Metrics | Prometheus endpoint for the usual observability stack |
| Persistence | SQLite storage, so state survives a controller restart |
Who it's for
Anyone with more than one box running Ollama
A home lab stitched together from old laptops and mini PCs. A small team sharing GPU workstations. An app that has outgrown a single node's queue and needs routing without touching client code. It's not aimed at Ollama Cloud, it clusters the ollama serve processes you already run yourself.
Licence and status
Free below a revenue threshold
Ollama Cluster is source-available under a fair-source, two-tier licence. It's free, commercial use included, for individuals, non-profits, and for-profit organisations with annual gross revenue of AUD $100,000 or less. Above that threshold you need a commercial licence.
Get the tool
Pre-built binary or Cargo
Grab a release for macOS or Linux from GitHub, or install with Cargo if you already have Rust 1.88 or later. Full install steps, CLI reference, and the terminal dashboard walkthrough live in the repo's README.
More
Browse the rest of the toolkit
All tools and utilities
SDLC, Agent Kit, Ollama Cluster, and whatever else I've built to make working with AI less painful.