Skip to content

Inference

Python

from decision_models import config, models
from decision_models.data import schema

encoder = models.load(config.ModelRef(kind="encoder", path="runs/ultrabert_380m_lr1.5e-5_sigmoidnorm"))
decoder = models.load(config.ModelRef(kind="decoder", path="Athekunal/llm-jev-qwen3-0.6b"))
lora    = models.load(config.ModelRef(kind="decoder", path="Qwen/Qwen3-0.6B", adapter="runs/q/adapter"))

row = schema.Example(id="q:1", source="mine",
                     parts=(schema.Part("Document", "The supplier delivers by May 1."),),
                     question="When must the supplier deliver?",
                     options=("By May 1", "By June 1", "Not stated"), gold=0)
s = encoder.score([row])[0]        # tensor [K], one score per option
k = encoder.predict(row)           # argmax s: index into row.options

gold is required by Example but not used for scoring; set it to any valid index.

ModelRef field meaning
kind encoder or decoder
path encoder: training run directory (its config.yaml gives the prompt); decoder: Hub id or path
adapter decoder LoRA adapter, merged on load
attention encoder: override the run's attention (sdpa on pre-Ampere GPUs)
dtype encoder weights, float32 (default, as trained) or bfloat16

A split, to JSONL

decision-models-predict model.kind=encoder model.path=runs/x rows.source=data/splits/mine out=runs/predict/mine.jsonl
decision-models-predict model.kind=decoder model.path=Athekunal/llm-jev-qwen3-0.6b limit=100 citation={} out=runs/predict/cited.jsonl

Each line is {id, source, answer, option, q, gold, correct}, plus the top evidence sentences with citation={}.

key meaning
rows.source, rows.split data folder or Hub id, and split (default eval)
rows.datasets, rows.per only these datasets, at most N rows each
limit first N rows
citation {} adds ContextCite evidence (citations)

Local page

uv sync --extra serve
decision-models-serve model.kind=encoder model.path=runs/x evidence=runs/evidence_x    # evidence head optional

Open http://127.0.0.1:8767/, pick a row, then Cite. The same server answers GET /api/sample?source=musique, /api/row?id=<id> and /api/attribute?id=<id>.

SGLang (decoders)

make sglang-install
make sglang-serve BASE=Qwen/Qwen3-4B ADAPTER=runs/qwen3_4b/adapter GPUS=0 PORT=30000   # merges the adapter first
decision-models-sglang tokenizer=runs/qwen3_4b/merged "rows.datasets=[contractnli]" rows.per=5
make sglang-stop PORT=30000
from transformers import AutoTokenizer
from decision_models.inference import sglang

tok = AutoTokenizer.from_pretrained("runs/qwen3_4b/merged")
answer, s = sglang.predict("http://127.0.0.1:30000", tok, row)

Merge an adapter on its own: decision-models-merge model=Qwen/Qwen3-4B adapter=runs/qwen3_4b/adapter out=runs/qwen3_4b/merged.