Ollama Load Balancer

You are managing an Ollama load balancer that distributes inference requests across multiple Ollama instances with automatic discovery, health monitoring, and failover. The load balancer handles all routing decisions transparently.

What the load balancer solves

Ollama has no built-in load balancing. One machine goes down, your app gets errors. No health checks, no failover, no queue management. You're manually pointing clients at specific machines and hoping they stay up.

This load balancer auto-discovers Ollama instances via mDNS, monitors their health continuously, and distributes load based on real-time scoring. The load balancer automatically retries on failure. Zero config files. Zero Docker. pip install ollama-herd, run two commands, and load balancing is active.

Deploy the load balancer

CODEBLOCK0

Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd

Load Balancer Endpoint

The load balancer runs at http://localhost:11435. Drop-in replacement for direct Ollama connections — same API, same model names, with load balancing built in.

CODEBLOCK1

Load Balancer Health Monitoring

Fleet-wide load balancer health check (15 automated checks)

The load balancer checks: offline nodes, degraded nodes, memory pressure, underutilized nodes, model thrashing, request timeouts, error rates. Each load balancer check returns severity (info/warning/critical) and recommendations.

Load balancer node status and metrics

Returns per-node: status (online/degraded/offline), CPU utilization, memory usage, loaded models with context lengths, and load balancer queue depths (pending/in-flight/done/failed).

Load balancer queue depths

Load Balancer Auto-Recovery

• - Load balancer auto-retry — if a node fails before the first response chunk, the load balancer re-scores and retries on the next-best node (up to 2 retries, configurable via FLEET_MAX_RETRIES)
• Load balancer zombie reaper — background task detects in-flight requests stuck longer than 10 minutes and cleans them up
• Load balancer context protection — strips dangerous num_ctx parameters that would trigger model reloads
• Load balancer VRAM-aware fallback — routes to an already-loaded model instead of triggering a cold load
• Load balancer auto-pull — optionally pulls missing models (disabled by default, toggle via settings)
• Load balancer holding queue — when all nodes are busy, requests wait (up to 30s) rather than failing

Load Balancer API Endpoints

Models available through the load balancer

Load balancer request traces

Load balancer usage statistics

Load balancer model recommendations

Load balancer settings (runtime toggles)

Load balancer model management

Load balancer per-app analytics

Load Balancer Dashboard

Web dashboard at http://localhost:11435/dashboard with eight tabs: Fleet Overview, Trends, Model Insights, Apps, Benchmarks, Health, Recommendations, Settings. All load balancer data updates in real-time via Server-Sent Events.

Load Balancer Operational Queries

Recent load balancer failures with error details

Load balancer retry frequency by node

Load balancer requests per hour

Test load balancer inference

Load Balancer Guardrails

• - Never restart or stop the load balancer or node agents without explicit user confirmation.
• Never delete or modify files in ~/.fleet-manager/ (contains load balancer latency data, traces, and logs).
• Do not pull or delete models on load balancer nodes without user confirmation — downloads can be 10-100+ GB.
• If a load balancer node shows as offline, report it rather than attempting to SSH into the machine.
• If all load balancer nodes are saturated, suggest the user check the dashboard.

Load Balancer Failure Handling

• - Connection refused → load balancer may not be running, suggest herd or INLINECODE8
• 0 nodes online → suggest starting herd-node on load balancer backend devices
• mDNS discovery fails → use INLINECODE10
• Load balancer requests hang → check for num_ctx in client requests; verify with INLINECODE12
• Load balancer API errors → check INLINECODE13