Lab
Projector Tuning (Multimodal)
Projector tuning is the third training route. It trains the projection layer that connects a frozen modality encoder to the language model, so workspace models gain grounded image or audio understanding.
Where it fits
The Lab has three distinct training routes. They solve different problems and are never interchangeable:
| Route | What trains | Use it for |
|---|---|---|
| PEFT Tuning (LoRA) | LoRA adapter weights (rank/alpha, optionally GRPO) | Task and behavior specialization from examples |
| Doc-to-LoRA (D2L) | A hypernetwork that emits adapters from documents | Context internalization: serve context from weights |
| Projector tuning (this page) | The encoder-to-LLM projection layer, optionally plus a LoRA | Multimodal grounding: image or audio inputs |
How it works
A frozen pretrained encoder turns the modality input into embeddings. The projector maps those embeddings into the language model's token space. Training aligns that mapping on paired examples from a workspace data asset. The encoder itself never trains.
| Parameter | Type | Description |
|---|---|---|
modality | image | audio | Input modality. Image uses a vision tower, audio uses a speech encoder. |
encoder | string | Frozen encoder tap. Defaults per modality: SigLIP for image, Whisper for audio. |
stages | stage1 | stage1+2 | stage1 trains the projector only. stage1+2 also jointly tunes a LoRA adapter on the language model. |
lora_rank / lora_alpha | integer | LoRA hyperparameters for stage 2. Only used with stage1+2. |
Each run is a durable record with status, encoder, stage progress, and HybrIE job linkage. Runs execute on the HybrIE engine's multimodal training route and report through the same job stream as other Lab training.
CLI
# Create an image projector run (SigLIP encoder, projector only)
stimulir lab projectors create -m image --stages stage1 --dataset <data-asset-id> -n "vision alignment v1"
# Create an audio run that also jointly tunes a LoRA (stage 2)
stimulir lab projectors create -m audio --stages stage1+2 --lora-rank 16 --lora-alpha 32 --dataset <data-asset-id>
# Inspect runs
stimulir lab projectors list --modality image --status running
stimulir lab projectors get <run-id>
# Delete a run record (in-flight HybrIE jobs run to completion)
stimulir lab projectors delete <run-id> --yesConsole
The same runs live in the Console under Lab, in the Projectors tab. The view has modality and stage switchers, run detail with HybrIE job linkage, and run deletion.