decision-models¶
Models that pick one option for a question about a document in a single forward pass, and show the sentences behind the pick. Results and figures are in the blog post.
| family | class | score of option k | loss |
|---|---|---|---|
| decoder | TrieDecoder |
log p("k]" | prompt … "Best answer: [") | L_tree + KL anchors (LLM-as-Jev) |
| encoder | MaskedEncoder |
ℓ(" yes") [− ℓ(" no")] at the mask in front of option k | softmax, sigmoid-norm or sqrt-softplus over the options |
Install¶
uv sync --extra dev # add --extra qwen35 (Qwen3.5), serve (local page), spans (labels), profile
Python 3.11+ and an NVIDIA GPU for training. Or use Docker: make docker-build TARGET=encoder && make docker-run TARGET=encoder.
Quickstart¶
make data # or use data.source=Athekunal/llm-jev-reason-long
make train-encoder CONFIG=configs/train/ultrabert_380m_sigmoid_norm.yamlGPUS=0
make train-decoder CONFIG=configs/train/qwen3_0.6b_lora.yamlGPUS=0,1,2,3
scripts/evaluate.sh encoder runs/ultrabert_380m_lr1.5e-5_sigmoidnorm 0
decision-models-predict model.kind=encoder model.path=runs/ultrabert_380m_lr1.5e-5_sigmoidnorm out=runs/predict/eval.jsonl
from decision_models import config, models
from decision_models.data import storage
model = models.load(config.ModelRef(kind="decoder", path="Athekunal/llm-jev-qwen3-0.6b"))
rows = storage.load_examples("Athekunal/llm-jev-reason-long", split="eval")[:8]
choices = [int(s.argmax()) for s in model.score(rows)]
Every command takes --config file.yaml and key=value overrides. The keys are the dataclass fields in src/decision_models/config.py.
Guides¶
| page | covers |
|---|---|
| Data | the row format, what the encoder and decoder see, building the mix, your own data |
| Training | configs, every argument, outputs, evaluation |
| Inference | Python, JSONL predictions, local page, SGLang |
| Citations and evidence | ContextCite, the evidence head, generating evidence labels |
| Docker | images per model family, GPU generations |
Releases¶
- Dataset: Athekunal/llm-jev-reason-long (ReClor withheld)
- Encoders (UltraBERT-380M): softmax "yes", softmax "yes" − "no", sigmoid-norm, sqrt-softplus
- Decoders: Qwen3-0.6B, Qwen3-4B