Troubleshooting

Salut is a pre-release llama.cpp and GGUF system. Automatic local loading supports Apple-silicon Macs. Windows CUDA workers running from source verify locked runtimes and use native process supervision. Windows GPU hardware validation and final signed packages are still in progress.

Peers Do Not Appear

  • Confirm both nodes are on a network that permits mDNS on UDP port 5353.

  • Confirm clustering is enabled.

  • A client-only node browses but does not advertise itself.

  • Allow incoming connections to Salut in the operating-system firewall.

  • Keep configured API and peer ports consistent with SALUT_PORT and SALUT_PEER_PORT.

The local API uses TCP port 7258 and peer HTTPS uses TCP port 17258 by default. A native worker also uses an operating-system-assigned incoming port for its deployment-scoped encrypted forwarder. A firewall restricted to only the fixed ports blocks distributed computation.

A Peer Is Visible but Cannot Participate

Open Settings → Peers and check that both nodes accepted the pairing. A discovered peer is not trusted automatically. If the certificate fingerprint changed, remove the old trust record, verify the new machine identity, and pair again.

Solo and client-only nodes reject inbound deployment controls. An unhealthy peer is also excluded from model routing.

/v1/models Is Empty

The operation lists ready local deployments and models advertised by healthy paired peers. It does not list arbitrary GGUF files on disk.

Choose Settings → Models → Choose GGUF File, or use /v1/models/load, to load an existing local model. Confirm that the deployment reports ready and that its model ID matches the request exactly.

/v1/models/load Fails

  • 400, invalid_gguf: check that the selected path points to one complete, readable GGUF. Split GGUFs are not supported by automatic loading.

  • 409, deployment_busy: stop the active deployment or compute contribution before loading another model. Local loading is disabled in client-only mode.

  • 422, insufficient_capacity: the measured memory budgets cannot fit the requested model and context. Free memory, reduce the requested context, or make another compatible paired Mac available.

  • 503, backend_unavailable: check the native runtime and deployment failure details. Automatic loading requires the complete verified b11960 Apple-silicon runtime. The legacy two-executable development overrides cannot provide it.

Native HTTPS Deployment Fails

Status 403

  • Confirm the operator is a distinct paired Salut identity.

  • Put its full lowercase certificate fingerprint in SALUT_LLAMA_OPERATOR_FINGERPRINTS on the coordinator.

  • Restart the coordinator after changing the allowlist.

  • Confirm clustering is enabled and the coordinator is not client-only.

Pairing alone does not authorize deployment control. An empty allowlist denies every remote operator.

Status 409

The coordinator rejects a request ID reused within ten minutes. Let python -m salut.inference.operator generate a new ID for each poll or retry. A conflicting deployment must be stopped by its exact deployment ID and plan SHA-256.

Status 413

The MessagePack control exceeds 64 KiB. A plan contains identities and metadata, not embedded model or executable bytes.

Status 503

  • In the Apple-silicon app, confirm the outer app signature is valid and that runtime-final.json still matches all 39 b11960 runtime files.

  • For native inference from a development checkout, build and launch the app with its complete verified runtime, or set SALUT_LLAMA_APP to its absolute path. Remove the old SALUT_LLAMA_SERVER and SALUT_GGML_RPC_SERVER overrides.

  • Confirm the coordinator and all workers use the same runtime release.

  • Confirm the GGUF path, byte size, and SHA-256 match the plan.

  • Check whether a previous cleanup failure retained deployment ownership for an exact stop retry.

The operator command controls an already running and paired Salut node. It does not install binaries, copy a GGUF, or start Salut remotely.

Distributed Deployment Fails

  • Confirm every selected worker is still paired with the coordinator.

  • Confirm raw RPC listens only on worker loopback.

  • Allow the deployment-scoped encrypted forwarder through the firewall.

  • Check for a failed nonce acknowledgment, expired reconnect grace period, or certificate mismatch.

  • Confirm the plan uses a device supported by its exact runtime identity. Protocol version 2 identifies Apple-silicon Metal as MTL0, Intel macOS Accelerate as BLAS, and Windows CUDA as CUDA0. Only the Apple-silicon runtime resolver and package path are implemented today.

  • Confirm the GGUF inode did not change while it was leased.

One remote worker cannot be the sole device in an all-remote N=1 plan. Local N=1 uses the coordinator device, and an all-remote plan requires at least two workers.

Slow Performance

Distributed batch-1 inference pools capacity. It can be slower than a safe single-device deployment because every RPC device adds network and scheduling work.

  • Use wired Ethernet for distributed measurements.

  • Run one warm-up before comparing generation rates.

  • Record prompt throughput, time to first token, generation throughput, and traffic for each repetition.

  • Check memory pressure, swap growth, swapouts, and pageouts from immediately before load through explicit deployment teardown.

  • Account for cold RPC tensor-cache creation separately from warmed requests.

Do not treat swap-induced completion as safe admission. The production plan must reject an unsafe deployment before it starts backend processes.

The App Stops Responding

Check Console.app for messages from the Salut process. Increase the log level to Debug, reproduce the problem once, and record the exact deployment and plan identities. If a backend process remains after the app exits, capture the process and listener evidence before terminating it so cleanup can be fixed.

Reset Local State

Salut identity, trust, and settings are stored outside the application bundle. Move the data directory to a backup location before starting again. Removing it creates a new identity and invalidates every existing pairing.

GGUF and llama.cpp tensor caches are separate. Delete them only after resolving the exact managed paths and confirming that the retained cache is not needed for evidence.