Scoring system

Each reviewer report includes 1-5 numerical scores on two dimensions, “Correctness” and “Value & insight”, and each is accompanied by a 1–5 confidence rating.

Correctness

The claims in the manuscript are worthy of belief.

• Mathematical computations must be accurate, proofs must be true, methodologies must be rigorous, and arguments must be sound.
• Caveats and countervailing considerations must be presented alongside the corresponding claim with prominence appropriate to their seriousness (e.g., in the abstract).
• Correctness is logically independent of importance; papers can be correct but trivial or incorrect but stimulating.

Options:

• 5 — All claims are correct
• 4 — Has minor, easily fixable mistakes
• 3 — Significant errors exist; major revisions necessary
• 2 — Contains serious errors; unlikely to be fixable
• 1 — Irreparably wrong

Value & insight

The manuscript advances our principled understanding of alignment of intelligent systems.

• Improved benchmark performance per se does not generally imply insight, but the statistical behavior of models can of course provide important information about their internals.
• Empirical evidence of improved performance on non-alignment capabilities is insufficient.
• Scientific novelty in the traditional strict sense is not necessarily required: insightful translational and synthesis work will be accepted so long as it satisfies all criteria.

Options:

• 5 — Seminal. Likely the most important alignment paper of the year.
• 4 — Deeply insightful. Worthy of special recognition and wide readership.
• 3 — Valuable. At least a few researchers should read this paper immediately.
• 2 — Possibly useful: worth having in the literature, but there is no one who ought to read it immediately.
• 1 — Negligible insight or value

Confidence

Both scores (Correctness and Value & Insight) carry a confidence rating on the same 1–5 scale:

• 5 — Absolutely certain the evaluation is correct and very familiar with the relevant literature
• 4 — Confident, but not absolutely certain, that the evaluation is correct
• 3 — Fairly confident that the evaluation is correct
• 2 — Willing to defend the evaluation, but likely not to have understood central parts of the paper
• 1 — The evaluation is an educated guess

Visibility

Scores are visible to all participants in the review discussion, and they remain confidential with the discussion, even after publication.