Models¶

Preload Model¶

Set a model to load automatically when Salut starts. This is useful for machines that always serve the same model.

Enter the full Hugging Face model ID, for example:

• mlx-community/Qwen3-8B-MLX-4bit

• mlx-community/Llama-3.3-70B-Instruct-4bit

• mlx-community/Phi-4-mini-instruct-4bit

The model downloads from Hugging Face on first use and is cached in the ~/.cache/huggingface/ directory. Subsequent loads use the cached version.

Set via SALUT_PRELOAD_MODEL environment variable.

Custom Models¶

Add model IDs that appear in your local model selector alongside auto-detected ones. This is useful for models hosted on private Hugging Face repositories or custom fine-tunes.

Each model ID should be on its own line in the custom models field.

How Loading Works¶

When a model is requested (either via preload or an API request):

1. Salut checks if the model is already loaded in memory.

2. If not, it downloads the model weights from Hugging Face (if not cached).

3. The model is loaded into GPU memory using MLX.

4. The tokenizer is initialized.

5. The model is ready to serve requests.

Only one model is loaded at a time. Loading a new model unloads the previous one to free GPU memory.

Model Sizing¶

The amount of GPU memory (VRAM) a model needs depends on:

• Parameter count — larger models need more memory (7B, 13B, 70B, etc.)

• Quantization — 4-bit models use roughly half the memory of 8-bit models

• Context length — longer contexts use more memory during inference

Rough guidelines for 4-bit quantized models on Apple Silicon:

Distributed Models¶

When a model is too large for a single machine, Salut can distribute it across paired peers. Each peer handles a subset of the model’s transformer layers (a “shard”).

This happens automatically during the rendezvous process — you don’t need to configure sharding manually. See Clustering for more details.