Distributed Inference

A coordination layer for distributed inference across heterogeneous machines. Each node is autonomous — it runs its own Ollama, manages its own models, and works fine standalone. The distributed inference coordinator routes requests to the optimal node using a multi-signal distributed inference scoring function and records every distributed inference decision for analysis.

Install Distributed Inference

CODEBLOCK0

Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd

Distributed Inference Architecture

CODEBLOCK1

Distributed inference nodes discover the coordinator via mDNS (_fleet-manager._tcp.local.) or connect explicitly with --router-url. Each distributed inference node sends heartbeats every 5 seconds containing: CPU utilization, memory usage and pressure classification, disk metrics, loaded models with context lengths, available models, and an optional capacity score from the behavioral model.

Distributed Inference Scoring Function

The distributed inference coordinator evaluates every online node for every request using 7 weighted signals:

Distributed inference nodes with insufficient memory, critical pressure, or missing models are eliminated before scoring. The highest-scoring distributed inference node wins.

Adaptive Distributed Inference Capacity

Distributed inference nodes optionally learn usage patterns and constrain their availability:

• - 168-slot behavioral model — one slot per hour of the week, learns when the distributed inference machine is typically free
• Dynamic memory ceiling — maps availability score to how much RAM the distributed inference coordinator can use

Enable with FLEET_NODE_ENABLE_CAPACITY_LEARNING=true on the distributed inference node agent.

Context-aware Distributed Inference Model Placement

The distributed inference coordinator protects against a known Ollama behavior where changing num_ctx at runtime triggers a full model reload. For an 89GB model, this causes multi-minute hangs.

• - num_ctx ≤ loaded context → stripped from the distributed inference request
• INLINECODE6 > loaded context → searches loaded models across all distributed inference nodes for sufficient context
• Configurable: INLINECODE7

Distributed Inference API

Distributed Inference Coordinator State

Distributed Inference (OpenAI-compatible)

Distributed Inference (Ollama-native)

Distributed Inference Model Fallback Chains

Distributed Inference Trace Analysis

Distributed Inference Node Performance

Distributed Inference Health and Capacity

Distributed Inference Model Lifecycle

Distributed Inference Fault Tolerance

Distributed Inference Data Model

All distributed inference state is in SQLite at ~/.fleet-manager/latency.db:

CODEBLOCK10

Structured distributed inference logs at ~/.fleet-manager/logs/herd.jsonl — daily rotation, 30-day retention.

Distributed Inference Dashboard

INLINECODE10 — eight tabs covering distributed inference fleet overview, trends, model insights, per-app analytics, benchmarks, health checks, model recommendations, and settings.

Distributed Inference Constraints

• - Never restart distributed inference services or modify ~/.fleet-manager/ without explicit user confirmation.
• Distributed inference model pull/delete operations require user confirmation (10-100+ GB transfers).
• If the distributed inference coordinator is unreachable, suggest herd or uv run herd.
• If no distributed inference nodes are online, suggest herd-node on target machines.