Models¶

Load a Local GGUF¶

Choose Settings → Models → Choose GGUF File and select one complete local GGUF. The Models pane loads it with a 4096-token context and reports admission, deployment progress, and any failure.

The local API also accepts a model name, an absolute file path, and a context size:

curl http://localhost:7258/v1/models/load \
  -H "Content-Type: application/json" \
  -d '{"model":"my-model","path":"/absolute/model.gguf","context_size":4096}'

Automatic loading requires the complete verified b11960 Apple-silicon runtime. The legacy b10516 two-executable development overrides are no longer accepted. Context sizes range from 1 through 131072, subject to the native estimator. Loading uses one request slot, f16 KV, a batch size of 512, and a microbatch size of 128. Client-only mode disables local loading. If this node is serving as a worker, its coordinator must stop that deployment before it can load a model of its own. Downloads, split GGUFs, and MTP drafts are not supported by automatic loading.

Salut tries the local Metal device first, then healthy paired Metal or Windows CUDA devices at the same llama.cpp commit. Fresh signed memory reports and the pinned native no-allocation fitter determine whether those devices have enough capacity. Remote estimation uses the real encrypted RPC forwarders. Plans exceeding measured memory budgets are rejected before the model controller starts, and Salut keeps the verified GGUF descriptor open from preparation through deployment.

Model Identity¶

An immutable execution plan records at least:

• model ID exposed through the OpenAI-compatible API

• GGUF repository, revision, filename, byte size, and SHA-256

• pinned backend source revision and executable hashes

• context, batch, microbatch, parallel request count, and KV types

• selected peer identities, device order, and allocation weights

Local files use their content digest as identity, with local as the repository and no remote revision. Memory reports are refreshed during admission.

The current plan accepts one complete GGUF file. Split-GGUF filenames are rejected. Salut opens the verified file, passes an inherited descriptor to llama-server, and monitors the leased inode for size, timestamp, or identity changes while the deployment is active.

The Apple-silicon app verifies its outer signature, the locked b11960 source manifest, the final post-sign manifest, and all 39 runtime files before copying the closure into an owner-only content-addressed directory. The complete staged closure is reverified immediately before launch. Executable-only development overrides are no longer accepted. Backend processes run under parent-liveness watchdogs with a strict environment allowlist.

Development Deployment¶

An explicitly allowed paired operator can still submit one immutable plan to the coordinator over peer HTTPS:

uv run python -m salut.inference.operator deploy \
  --host coordinator.local \
  --coordinator-fingerprint <full-fingerprint> \
  --plan plan.json

The coordinator must list that operator’s full fingerprint in SALUT_LLAMA_OPERATOR_FINGERPRINTS. The empty default denies every remote operator. This operation verifies a supplied plan. It does not choose a model, calculate admission, or place devices automatically.

A ready deployment appears in /v1/models. The public chat endpoint can serve it locally or proxy to a paired peer that advertises the exact model ID. The Models pane follows every manager state transition, including early runtime verification before the controller process exists, controller cleanup, and worker-only participation.

Device Plans¶

Protocol version 2 identifies Darwin arm64 Metal as MTL0, Darwin x86_64 Accelerate as BLAS, and Windows AMD64 CUDA as CUDA0, all at one source commit. Local-only N=1 uses the coordinator device without an RPC worker. A mixed plan uses one or more remote workers and may also select the coordinator device. An all-remote plan requires at least two workers.

llama-runtime.toml locks official b11960 archives for all three targets. Apple-silicon macOS has a runtime resolver and Briefcase packaging. Windows CUDA workers running from source verify the locked archives and supervise native processes with Windows Job Objects. An Apple-silicon Mac coordinates inference.

Intel macOS still needs BLAS system-memory admission, placement, package integration, signing, and physical validation. Windows model coordination, app packaging, and a production-path CUDA0 hardware run remain unfinished. Developer ID signing and notarization of the macOS package are also unfinished.

Optional MTP Draft¶

An immutable plan may identify a separate exact MTP draft GGUF for native llama.cpp speculative decoding. The draft has its own repository, revision, path, size, hash, device, KV types, and token acceptance settings. Salut leases and monitors the draft artifact independently from the target model. A draft is never selected implicitly.

Memory Admission¶

Filename, parameter count, quantization label, and GGUF byte size are not sufficient memory admission evidence. A safe plan must account for:

• model buffers on every selected device

• CPU-side buffers retained by the controller

• KV cache for the exact context and parallel request count

• compute buffers

• RPC tensor-cache creation and other cold-load transients

• operating-system and application headroom

Automatic admission uses the pinned native estimator to bound shared host and device memory, including controller working memory for loading and long prompts and RPC tensor transfers. On macOS, the host budget counts free and speculative pages, plus a discounted file-cache allowance. That allowance is three quarters of the smaller of inactive pages and file-backed pages after subtracting speculative pages. Speculative pages are counted only once. Anonymous, purgeable, and compressed totals add no capacity.

Admission requires normal system memory pressure and keeps at least 2 GiB or 10% of total RAM, whichever is larger, in reserve. The cache allowance is an estimate, not a measurement of clean cache. File-backed pages may still be active or dirty, and other applications can allocate memory after the check. Admission does not guarantee zero swap or monitor a running deployment for new swap use.

Windows CUDA workers also keep 256 MiB of currently free GPU memory for runtime allocations made after the capacity check. Their device budget is the smaller of current free memory minus that reserve and total memory minus the existing 2 GiB or 10% floor. The two reserves are not added together. This reduces the risk of admitting a model onto a nearly full GPU, but other applications can still allocate memory after admission.

GGUF inspection and hashing request uncached reads on macOS. File cache can still refill while the native runtime uses the model. Before admission, Salut also asks macOS to release cached pages of the selected GGUF on local filesystems. It skips file portions with modified or unknown page state, checks that the model has not changed, and measures memory again. Reclamation can be incomplete, and other applications can refill the cache. It does not bypass the memory checks.

Placement prioritizes capacity, with the local device first. It does not optimize from measured bandwidth.

Two-Mac development runs compared native estimates with actual memory use and monitored swap through teardown. These results cover the tested model and hardware. A run proving capacity for a model that requires the combined memory of the cluster remains pending.

Distributed batch-1 inference is a capacity feature. It is not a promise that a model which fits safely on one device will run faster across several devices.

Storage and Cleanup¶

The complete GGUF remains at the selected local path on the coordinator. Remote workers may retain a documented llama.cpp tensor cache after deployment teardown. Every accepted run records cache location and size. Processes, forwarders, listeners, active requests, temporary copies, and file leases must be gone after explicit deployment cleanup.