Models¶

Preload Model¶

Set a model to load automatically when Salut starts. This is useful for machines that always serve the same model.

Enter the full Hugging Face model ID, for example:

• mlx-community/Qwen3-8B-MLX-4bit

• mlx-community/Llama-3.3-70B-Instruct-4bit

• mlx-community/Phi-4-mini-instruct-4bit

The model downloads from Hugging Face on first use and is cached in the ~/.cache/huggingface/ directory. Subsequent loads use the cached version.

Set via SALUT_PRELOAD_MODEL environment variable.

Custom Models¶

Add model IDs that appear in your local model selector alongside auto-detected ones. This is useful for models hosted on private Hugging Face repositories or custom fine-tunes.

Each model ID should be on its own line in the custom models field.

How Loading Works¶

When a model is requested (either via preload or an API request):

1. Salut checks if the model is already loaded in memory.

2. If not, it downloads the model weights from Hugging Face (if not cached).

3. The model is loaded into GPU memory using MLX.

4. The tokenizer is initialized.

5. The model is ready to serve requests.

Salut keeps only one user-selected target model loaded at a time; loading another target unloads the previous one. Distributed greedy decoding may also retain one hidden, audited draft model when its exact pinned snapshot is already cached and the coordinator can reserve it safely.

Model Sizing¶

The amount of GPU memory (VRAM) a model needs depends on:

• Parameter count — larger models need more memory (7B, 13B, 70B, etc.)

• Quantization — 4-bit models use roughly half the memory of 8-bit models

• Context length — longer contexts use more memory during inference

Rough guidelines for 4-bit quantized models on Apple Silicon:

Distributed Models¶

When a model is too large for the requester’s selected memory budget, Salut can distribute it across paired peers. Each peer handles a subset of the model’s transformer layers (a “shard”). Models remain whole only when their weights, fixed recurrent state, and request cache fit one ready node’s budget; batch-1 sharding is a capacity feature, not a speed claim.

Salut does not treat all advertised unified memory as equally fast model space. bounded-working-set-v2 starts with MLX’s device-specific recommended working set, keeps 1/40 of it as coordinator headroom or 1/20 as worker headroom, and subtracts a further 512 MiB on the coordinator or 256 MiB on a worker from the layer-placement budget. During load and inference Salut temporarily applies the corresponding MLX memory limit, then restores the previous limit. This is conservative allocator guidance, not a residency guarantee: model pages remain pageable, and Salut never changes Metal’s wired-memory limit.

When MLX reports a positive working-set recommendation, Salut does not replace that whole-model budget with the larger physical-memory estimate. The whole model remains prepared; chat either forms a distributed layout or fails with insufficient_cluster_capacity instead of materializing oversized weights on one node. Whole-model compatibility uses guarded physical memory only when no working-set recommendation is available.

Placement reserves the model-derived batch-1 cache for the current request on every assigned layer. Ordinary attention models pre-size one standard key/value cache per layer. Qwen’s hybrid layouts reserve their fixed recurrent state and periodic key/value caches. LongCat keeps its global n-gram embedding cache on the coordinator and maps each transformer layer to its paired compressed MLA caches. Positive token capacity is rounded to MLX’s 256-token allocation block before prefill so caches allocate once instead of repeatedly growing and copying.

A positive-size model whose cache geometry or layer mapping cannot be proven is not eligible for distributed placement. Unknown composite layouts and rotating caches remain rejected until Salut can pre-size their memory demand.

Before the model is materialized, Salut renders the model’s own chat template and tokenizes it with the prepared tokenizer, then adds the requested output-token budget. It derives exact batch-1 FP16 K/V bytes per token and transformer layer from the model’s key/value head count and head dimension. Long contexts can therefore correctly make an otherwise fitting model ineligible without relying on generic prompt or architecture-width estimates.

After every worker has joined successfully, Salut retains the exact snapshot, ordered topology, layer counts, and the largest context that layout can hold. Shorter later requests reuse those layer counts instead of alternating between nearly identical shards and paying another multi-minute materialization. A request above the retained envelope may upgrade the layout once; a failed upgrade leaves the last successful envelope intact. Each worker still validates the new request’s exact context and memory plan.

Explicit preload runs one real prefill position and one real decode position so the selected weights and kernels are exercised before the first user chat. It reports both the exercised warmup context and the layout’s context-capacity envelope. It reports bounded-working-set-v2 when every shard fits the conservative working-set budgets, or guarded-physical-v2 and working_set_warm: false when the layout relies on the larger pageable physical-memory budget. A warm working-set layout means its evaluated arrays are retained under bounded memory guidance; it does not mean the OS cannot page them. Salut does not fill a dummy KV cache to the full envelope, which would make startup unnecessarily expensive.

When enough paired Macs are available, Salut compares every bounded working-set peer subset and chooses the one with the lowest estimated decode cost, including memory-bandwidth work and a penalty for each network hop, before considering the larger guarded-physical-v2 fallback. The fallback leaves the existing MLX memory limit unchanged and keeps the larger of 4 GiB or 10% of physical memory outside the placement budget, but it can page and run slowly under pressure. Add another peer or use smaller weights when preload reports a guarded layout and stable decode latency matters. The -v2 capacity values deliberately prevent unsafe older peers from joining these plans.

Before accepting an automatic model pull, a worker verifies both its configured model-cache budget and current filesystem space. Already-cached shards are not charged twice; any missing weight bytes must fit while leaving 4 GiB of free disk reserve.