Skip to content

Data

Every dataset is stored as rows of one type, schema.Example (src/decision_models/data/schema.py). Encoders and decoders train on the same rows and differ only in how they turn a row into tokens.

The row

field type meaning
id str unique within its dataset
source str dataset name (per-dataset metrics group by it)
parts list of Part(name, text) the input text, in prompt order
question str what to decide
options list of str at least 2; option i is shown as [i+1]
gold int 0-based index of the correct option
spans list of Span(part, start, end) optional: characters [start, end) of parts[part] that support the gold
group, subset, meta str, str, JSON optional: leakage unit, stratum, provenance

A short row from the eval split:

Example(
    id="anli:451999a5f0cc679f", source="anli",
    parts=(Part("Observation 1", "Noah had a cold."),
           Part("Observation 2", "After two days, Noah felt better.")),
    question="Which hypothesis best explains what happened between the two observations?",
    options=("Noah ate cold cuts and got a lot of rest.",
             "Noah took his prescribed medicine and stayed home."),
    gold=1,
)

A row with evidence (ContractNLI). The span points into the contract text:

row.question  # 'Statement: "Receiving Party shall not disclose the fact that Agreement was agreed or negotiated." ...'
row.options   # ('The contract entails the statement.', 'The contract contradicts the statement.', 'The contract does not mention it.')
row.gold      # 0
row.spans     # (Span(part=0, start=1354, end=1637),)
row.parts[0].text[1354:1637]
# 'Listing Broker, Prospect, and Prospect’s Broker agree not to disclose to any other person the fact that ...'

What the decoder sees

models/decoder.py:build_prompt renders prompts/decision.j2 in the chat template and prefills the answer up to Best answer: [ (Qwen3, thinking off):

<|im_start|>user
State: Noah had a cold.

After two days, Noah felt better.

Which hypothesis best explains what happened between the two observations?
Options:
[1] Noah ate cold cuts and got a lot of rest.
[2] Noah took his prescribed medicine and stayed home.
Select one option. Answer only with its bracketed numeric identifier.<|im_end|>
<|im_start|>assistant
<think>

</think>

Best answer: [

Option k is the token suffix k]. All suffixes form a trie; the score of option k is

s_k = Σ_t log p(suffix_k[t] | prompt, suffix_k[<t])

For this row the suffixes are ['1', ']'] and ['2', ']'], and training teacher-forces the gold one (2]). With 12 options, Qwen splits digits, so the trie branches twice:

(root) -> 1 2 3 4 5 6 7 8 9
"1"    -> 0 1 2 ]          # "1]", "10]", "11]", "12]"
"2".."9", "10".."12" -> ]
from decision_models.models import decoder

enc = decoder.encode(row, tokenizer)            # options in release order
enc.input_ids                                   # prompt + "2]"
enc.path                                        # [(82, legal ['1','2'] ids, gold '2' id), (83, [']'], ']')]

What the encoder sees

models/encoder.py:build_prompt renders prompts/decision_mlm.j2 with one mask in front of every option (UltraBERT):

State: Noah had a cold.

After two days, Noah felt better.

Which hypothesis best explains what happened between the two observations?
Mark the correct option with yes and every other option with no.
Options:
[1] <|mask|> Noah ate cold cuts and got a lot of rest.
[2] <|mask|> Noah took his prescribed medicine and stayed home.

One forward pass gives the MLM logits ℓ at every mask, and the score of option k is

s_k = ℓ_k(" yes")  [ − ℓ_k(" no")  when format.no_token: " no" ]

The loss normalises s over the K options (training).

from decision_models import config
from decision_models.models import encoder

enc = encoder.encode(row, tokenizer, config.FormatConfig(no_token=" no"))
enc.masks     # [52, 68]: token position of each option's mask
enc.gold      # 1

During training both models shuffle the options per row (encode(..., rng)), so the gold position is uniform.

Build the training mix

make data # download the pinned sources, then build data/splits/reason_long_nimble
decision-models-build-data --configconfigs/data/reason_long_nimble.yaml\
datasets=[musique,contractnli] per_dataset=500 out=data/splits/small

argument default meaning
out data/splits/reason_long_nimble writes train.parquet, eval.parquet, manifest.json
datasets all 13 subset of clinc150 massive bitext reclor logiqa2 cosmosqa piqa anli proofwriter contractnli musique quality nimble
per_dataset 1000 training rows per dataset
eval_rows 500 held-out rows per dataset
max_tokens 10240 longer prompts are dropped, never truncated
length_tokenizer Qwen/Qwen3.5-4B measures max_tokens
download_only false only fetch the sources

Or skip the build and train on the released mix (ReClor withheld): data.source=Athekunal/llm-jev-reason-long.

Your own data

Write Examples to <folder>/train.parquet and <folder>/eval.parquet. For example, from a JSONL file with context, question, choices, answer and an optional evidence quote:

import json
from decision_models.data import schema, storage

def to_example(i: int, r: dict) -> schema.Example:
    context = r["context"]
    start = context.find(r.get("evidence") or "\0")
    spans = (schema.Span(0, start, start + len(r["evidence"])),) if start >= 0 else ()
    return schema.Example(
        id=f"mine:{i}", source="mine",
        parts=(schema.Part("Document", context),),
        question=r["question"],
        options=tuple(r["choices"]),
        gold=r["choices"].index(r["answer"]),
        spans=spans,
    )

for split in ("train", "eval"):
    with open(f"mine_{split}.jsonl") as f:
        rows = [to_example(i, json.loads(line)) for i, line in enumerate(f)]
    storage.save(storage.to_dataset(rows), f"data/splits/mine/{split}.parquet")

Example checks the invariants (≥ 2 options, gold in range, spans inside their part) and raises ValueError otherwise. Check how a row renders before training:

from transformers import AutoTokenizer
from decision_models.models import decoder

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
text, _ = decoder.build_prompt(rows[0], range(len(rows[0].options)), tok)
print(text)

Then train or evaluate on it with data.source=data/splits/mine (training) or rows.source=data/splits/mine (evaluation, prediction).

Add it to the build instead

To mix your dataset into make data (deduplication, benchmark-overlap filter, sampling), register a source. sources/multiple_choice.py is the pattern: base.make_example gives stable ids and groups, and spans are given as (part name, start, end).

# src/decision_models/data/sources/mine.py
from decision_models.data.sources import base

@base.register("mine")
def mine() -> base.Split:
    def row(r):
        return base.make_example(
            "mine", key=r["id"], subset="", group=r["doc_id"],
            parts=[("Document", r["context"])], question=r["question"],
            options=r["choices"], gold=r["label"],
            spans=[("Document", a, b) for a, b in r.get("spans", [])],
        )
    return {"train": [row(r) for r in my_reader("train")],      # your file reader
            "validation": [row(r) for r in my_reader("dev")]}

Import the module in sources/__init__.py, then decision-models-build-data ... datasets=[mine].