Network¶

Salut’s clustering controls determine how your machine participates in the peer network. You can be a full participant, a client that offloads work, or completely standalone.

Participation Modes¶

Cluster Mode (Default)¶

Your machine both advertises itself on the network and accepts inference work from paired peers.


Advertise	Yes — other peers can discover you
Browse	Yes — you discover other peers
Accept inbound	Yes — paired peers can send you work
Route outbound	Yes — you can distribute work to peers

This is the default and the right choice for most setups. Every machine contributes its GPU to the shared pool.

Client-Only Mode¶

Your machine discovers and routes to other peers but doesn’t advertise itself or accept inbound work.


Advertise	No
Browse	Yes
Accept inbound	No
Route outbound	Yes

Use this for machines you want to use as API clients without contributing their GPU. For example, a laptop that sends queries to more powerful desktops.

Enable in Settings → General by toggling Client-Only mode.

Set via SALUT_CLIENT_ONLY=1 environment variable.

Solo Mode¶

Your machine operates standalone — no advertising, no browsing, no peer communication.


Advertise	No
Browse	No
Accept inbound	No
Route outbound	No

Toggle clustering off in Settings → General or from the menu bar. All inference runs locally on your machine only.

Set via SALUT_CLUSTER_ENABLED=0 environment variable.

Distributed Inference¶

Salut uses whole-model inference only when the requester’s selected memory budget can hold the weights, fixed recurrent state, and request cache. If an already-ready paired peer can hold the whole model inside its bounded working-set budget, normal routing sends the request there instead. Salut shards automatically when the model does not fit on the requester and needs pooled cluster capacity. If MLX cannot report a working-set recommendation, Salut retains the guarded physical estimate as a whole-model compatibility fallback; a positive working-set budget is never replaced by that larger estimate for a whole-model load.

Batch-1 shards execute sequentially across the network. Sharding is therefore a capacity feature, not a promise that a model which already fits on one node will run faster. SALUT_FORCE_DISTRIBUTE=1 bypasses the local-capacity gate for correctness tests and benchmarks; it is not a normal performance setting. Automatically preparing an unloaded whole model on a remote peer is not yet part of this routing policy.

Salut first plans against a bounded working set before considering the larger pageable physical-memory budget. Each Mac advertises MLX’s recommended working-set size. Salut keeps 1/40 of that recommendation as coordinator headroom or 1/20 as worker headroom, then keeps another 512 MiB on the coordinator and 256 MiB on each worker out of the layer-placement budget for transient request allocations. Load, warmup, and decode temporarily apply the matching MLX memory limit and restore the prior limit afterward. This is allocator guidance rather than a residency guarantee, and Salut never changes Metal’s wired-memory limit.

The request reserves the model’s exact batch-1 cache layout on every assigned layer. Ordinary attention models use one pre-sized key/value cache per layer. Qwen hybrid models combine fixed recurrent state with periodic key/value caches. LongCat keeps its global n-gram embedding cache on the coordinator and assigns each transformer layer its paired compressed MLA caches. Salut rounds positive token capacity up to MLX’s 256-token allocation block before the first forward pass to avoid repeated concatenate-and-grow allocations during prefill.

The coordinator must additionally hold embeddings, final normalization, and the language-model head. Longer prompts or output limits therefore reduce the weight capacity available for that request. Salut rejects unknown composite cache layouts, unprovable cache geometry, and rotating caches whose memory demand cannot be pre-sized.

Rendezvous searches every peer combination that fits bounded-working-set-v2 and chooses the layout with the lowest estimated decode cost. The estimate combines each shard’s weight stream at that Mac’s memory bandwidth with a penalty for every network hop, then placement fills the fastest bounded capacity first. A guarded-physical-v2 fallback remains available when the cluster cannot form a bounded layout; it leaves MLX’s previous memory limit unchanged, can page under pressure, and is reported with working_set_warm: false. That fallback makes an otherwise impossible model runnable, but it is not the performance result to target.

A successful layout carries a context envelope and its capacity kind. For every request inside that envelope, Salut first computes the best current plan, then keeps the warm layout only when its estimated serial decode cost is no more than 10% higher. This small hysteresis avoids reloading for equal or near-equal layouts while allowing a materially faster new peer, or a shorter pipeline with fewer 25 ms worker round trips, to take over. A model revision, peer restart or endpoint change, working-set-capacity or bandwidth change, or explicit unload still invalidates the envelope outright.

How It Works¶

A request comes in for a model (e.g., a 70B model).
The coordinator node builds a rendezvous plan — mapping transformer layers to peers based on available VRAM and bandwidth.
Each peer loads its assigned shard (a contiguous range of layers).
During inference, activations flow between peers in sequence: peer 1 processes layers 0–19, sends the result to peer 2 for layers 20–39, and so on.
The final peer returns the output to the coordinator, which sends the response to the client.

Layer Assignment¶

Layers are assigned based on each peer’s reported and estimated capability:

Available VRAM — more memory means more layers
Request length — each layer must leave room for its estimated KV cache
GPU type — Metal (Apple Silicon) performance characteristics
Memory bandwidth — the Apple chip name distinguishes Pro, Max, and Ultra memory systems instead of guessing from RAM size

The best assignment is recalculated for every request. A compatible warm assignment is reused inside its context envelope only while it remains within the 10% decode-cost hysteresis.

Static Headless Clusters¶

Headless daemons can use SALUT_PEERS when mDNS discovery is not suitable. List every node in rank order using its full certificate fingerprint, hostname, and TLS peer port:

SALUT_PEERS="<riker-fingerprint>@riker.local:17258,<scotty-fingerprint>@scotty.local:17258"

Every listed remote fingerprint must already be paired on each node. Salut refuses to start static distributed mode when an entry omits its fingerprint or peer port, or when a remote fingerprint is not paired locally. Static peer traffic always uses the pinned TLS endpoint; it never falls back to the public HTTP API.

Static topology reuses the mDNS capability report for each fingerprint in SALUT_PEERS, keeps that list’s rank order, and applies the same working-set-first, request-aware placement as dynamic rendezvous. If the fixed ranks cannot fit the bounded plan, static placement uses the same explicitly reported guarded fallback. Salut refuses to load an oversized model when any static rank lacks a current capacity report or when the resulting assignment would exceed guarded memory. Smaller models that fit the coordinator may retain the legacy even split when a report is temporarily unavailable.

Activation Transport¶

Activations (the intermediate tensors passed between peers) can be transported in two modes:

Passthrough (default): Full-precision fp16 activations. Best quality, higher bandwidth usage.
INT8 Blockwise: Activations are quantized to 8-bit integers using a blockwise scheme (similar to Petals). Cuts bandwidth in half with minimal quality loss.

Configure in Settings → General or via SALUT_QUANT_ACTIVATIONS=1.

Decode Acceleration¶

Greedy distributed requests use exact prompt-lookup speculation by default on every supported cache layout, including LongCat. Repeated prompt or committed-output token sequences are only proposals: the target model verifies the block while Salut checkpoints the coordinator and worker caches, then commits accepted target state or restores and corrects rejected state. Set SALUT_PROMPT_LOOKUP_SPECULATION=0 to disable it.

Salut also implements external draft speculation for audited Qwen pairs whose exact target and draft revisions match. The draft must already be fully cached and fit the coordinator’s explicit placement reservation; automatic policy can decline it when that reservation would worsen the target layout or when measured savings do not cover draft work. LongCat has no audited external draft pair and uses prompt lookup only.

Cluster Tokens¶

For lab or office setups with many machines, manual pairing is tedious. Cluster tokens provide automatic pairing.

How It Works¶

Choose a shared secret (any string) and set it on all machines.
Set it in Settings → General → Cluster Token or via SALUT_CLUSTER_TOKEN environment variable.
When peers discover each other, they compare HMAC-SHA256 hashes of their tokens and fingerprints.
If the HMACs match, the peers are automatically paired — no manual acceptance needed.

Security Considerations¶

The token itself is never sent over the network — only the HMAC.
Tokens should be treated like passwords. Use a strong, unique value.
Peers with mismatched tokens go through the normal manual pairing flow.
You can use both: cluster token for trusted lab machines, manual pairing for ad-hoc guests.

Tip

A simple way to distribute the token: set it as an environment variable in a shared dotfile, configuration management tool, or launch script.