Skip to content

Citations and evidence

ContextCite

from decision_models import config, models
from decision_models.citation import cite

model = models.load(config.ModelRef(kind="decoder", path="Athekunal/llm-jev-qwen3-0.6b"))
settings = config.CiteConfig(removal="full")      # decoders: reader | full | delete
cite.prepare(model, settings)                     # packed attention + fused kernels, once
result = cite.cite(model, row, settings)
result.answer, result.q, result.weights           # one weight per sentence
cite.top_sentences(row, result, k=3)
removal model how equals deletion
reader decoder document once, ablated readers packed (FlexAttention) no (leaks)
full decoder kept tokens packed, shared prefixes once (FA2 varlen) yes
delete any one scored row per ablation (encoders pack them on FA2) yes

Lower level:

from decision_models.citation import context_cite, masked
from decision_models.models import decoder, flash_attention

flash_attention.use_decoder(model.network); decoder.fuse(model.network)
masked.attribute(model, row, removal="full")
context_cite.attribute(model, row)                # any DecisionModel, by deletion
context_cite.attribute_with(my_scorer, row)       # your own (example, sentences, keep) -> scores

Measure ContextCite against gold spans

scripts/attribution.sh Athekunal/llm-jev-qwen3-0.6b runs/attribution/qwen3_0.6b 0 50
decision-models-cite-eval model.kind=encoder model.path=runs/ultrabert_380m_lr1.5e-5 citation.removal=delete out=runs/attribution/ub

Writes rows.jsonl and summary.json: span AP / top-1, drop@k, LDS, seconds.

Evidence head (encoders)

make evidence CONFIG=configs/evidence/ultrabert_380m.yamlGPUS=0
decision-models-train-evidence --configconfigs/evidence/ultrabert_380m.yamldata.labels=gold out=runs/ev_gold

from decision_models.evidence import head as head_lib
from decision_models.models import encoder

model = encoder.MaskedEncoder.from_run("runs/ultrabert_380m_lr1.5e-5_sigmoidnorm")
evidence = head_lib.load("runs/evidence_ultrabert_380m_sigmoidnorm", model)   # head + dev-tuned threshold
pred = head_lib.predict(model, evidence.head, row)
pred.answer, pred.tokens.where, pred.probabilities      # (part, start, end) and p per token

Generated evidence labels (best-of-N, SelfCite reward)

uv sync --extra spans
GPUS="0 1" PORTS="8011 8012" scripts/serve_labeller.sh
decision-models-spans --configconfigs/evidence/spans.yaml\
"base_url=[http://127.0.0.1:8011/v1,http://127.0.0.1:8012/v1]" out=runs/spans/train.jsonl