# Closed-candidate ranking of five post-training holdouts — 2026-10-12

Frozen before ranking inference. Four original base models and five v2 adapters; no training.
Five cases: proprietà emergenti -> palafitte; Natural Uranium Decay -> lead generation;
PALAFITTE -> proprietà emergenti; NON POSSO PARLARE -> servo muto; UNA DONNA ORRIBILE
MANIPOLA L'OPINIONE PUBBLICA -> una brutta influenza. Exact prompts match earlier probes.

Candidate pool: unique lowercase whitespace-collapsed answers from all v2 corpus splits,
plus the five holdout answers. Accent/punctuation retained. All candidates offered to the scorer
for all prompts; no filtering by enumeration, language or semantic relevance. Candidate list is
NEVER provided to the model as part of its prompt. Each answer has one canonical spelling.
This is closed-list retrieval, not free-generation rank or token-vocabulary rank.
Training answers in the pool may favor fine-tuned models. English has mostly Italian distractors,
and palafitte is an unusually short answer: both are major limitations. The original and inverse
case are related. Five exploratory cases do not estimate general solver accuracy.

Primary rank: teacher-forced sum of natural log probabilities of candidate tokens plus EOS,
T=1, no top-k/top-p, no length normalization. Include EOS so a prefix isn't treated as a complete
answer. Also store answer-only logprob and prefix-rank for comparison to prior scoring.
All scores in BF16 model with FP32 log-softmax. Length-sorted fixed batches; same batches for each
base/LoRA pair. Slight differences vs prior single-example logprobs are expected from BF16 batching.
Exact ties use pessimistic rank for aggregate metrics; report optimistic rank and +/-0.05 nat
neighbor sensitivity interval too. This interval is a numerical sensitivity diagnostic, not CI.

Recall@k is fraction of five gold answers ranked <= k. k50 is third-smallest rank (>=50%, actually
60% on five items). Also report Italian-only k50 (second-smallest of four) because English distractors
are unbalanced. Record recall@1/3/5/10/20/50/100/500. Same fixed pool for all models.

Store every candidate score, top20 lists, source/plan/protocol/adapter hashes. After adapter unload,
rescore one fixed batch per prompt and require base restoration differences <1e-4. Fixed evaluation
batch sizes 16 for 4B, 8 otherwise. A100-80GB for 4B/9B, H100 for 27B; model revisions as v2.
Do not alter training corpus or add results to the existing v2 test aggregate.