Guide for Reviewers
Last updated: 29 October 2026
The Alignment Journal is a peer-reviewed journal for work on the scientific foundations of understanding, predicting, and steering artificial intelligent systems. It is free to read and free to publish in. This guide tells you what we ask of reviewers and what to expect.
How is reviewing different in the Alignment Journal?
Most of this will be familiar. How we differ:
- One continuing discussion. After the initial reports are in, reviewers, authors, and the action editor discuss the paper in a single thread with no fixed rounds. The editor proposes a decision once they have enough to go on.
- A published “review summary”. Each accepted paper is published with a short summary to show potential readers what the reviewers learned (see Review summary).
- The paper is listed publicly, the review is confidential. Papers that pass desk review appear on Papers under review, where researchers can nominate themselves as reviewers. Reports, scores, and the discussion stay confidential during and after review. So does your identity, unless you choose to sign the review summary.
- Mutual anonymity. The action editor knows who you are, but the authors and the other reviewers do not. The paper is anonymized.
- Paid reviewing. You can choose to be paid, with bonuses for timely and outstanding reviews, or waive the fee and recommend a nonprofit for the Journal to donate to. The pay will not depend on whether your assessment is positive or negative (see Compensation).
- We aim to be fast. Submissions are rolling, and we aim for under four months from submission to publication.
Before you accept
Before accepting, tell the action editor about any conflict of interest you are aware of, as defined in our conflict of interest policy. As anonymity during review relies on good faith effort by the participants, please do not try to identify the authors.
If you would like to involve a student, give the action editor the student’s email; if the editor agrees, the student is invited as an additional reviewer and reviews in their own name, credited and paid like any other reviewer. You are also welcome to suggest a colleague who could review the paper well.
If you have any questions before you accept, please reply with them to your invitation email, or contact us at contact@alignmentjournal.org.
What happens step-by-step?
- Invitation. The action editor emails you the title, anonymized abstract, the terms of the review, and a link. You accept or decline on the website. If you nominated yourself on the Journal’s website, the email telling you the editor accepted your nomination serves as your invitation.
- Initial report. Until you submit, you will not see the other reports, the discussion, or any AI-generated material the Journal holds on the paper. Aim to submit your initial report within four weeks of accepting; a report within 15 days of the invitation earns a timeliness bonus (see Compensation). If you need more time, or find you cannot continue, tell the action editor as early as you can; they may set a new deadline, invite another reviewer, or proceed without your report.
- Discussion. When the authors reply, please defend, clarify, or concede the points in your reports, and help settle factual disputes. You may also weigh in on points raised by the other reviewers. Try to reply within a few days when addressed, and let the editor know if you will be away.
- Decision. The action editor proposes accept, revise and resubmit, or reject. On revise and resubmit, the discussion pauses until the authors resubmit, usually within one to three months; the editor may ask you to look at the revision or may decide on their own. You are told the outcome.
- Review summary. If the paper is accepted, the editor invites one reviewer to write the review summary.
- Close. The paper leaves review when it is accepted, rejected, or withdrawn. Compensation is settled after.
Your report
You will be judging the paper for whether its claims are worthy of belief (“correctness”) and for whether it is the most valuable thing to read right now for at least some alignment researchers (“value and insight”): does it advance principled understanding of intelligent systems?
You review the PDF. You may consult supplementary material if it helps, but you are not required to. Per the AI-usage policy, the main text must be written by the authors, though light AI editing is allowed; the appendices and supplementary material may be AI-written if this is disclosed.
Judge the value and insight of the main text, which must carry motivation, main results, and how they were obtained; judge correctness on the whole paper, appendices included. Strict novelty is not required, and reproducibility factors into a paper’s value. Read the Journal’s scope and acceptance criteria before your first review.
Check the paper carefully enough that you’d be comfortable publicly vouching for its overall correctness. Say what you couldn’t reasonably check, as this work can be shared with other reviewers.
Structure, scoring, and style
The review box contains suggested but optional headings for your report. Distinguish between issues that (1) must be fixed before publication, (2) are not required, but will be named as deficiencies in the review summary if not addressed, or (3) are helpful suggestions for the authors. Require only what the acceptance criteria demand; reservations short of this belong in the review summary, where readers can weigh them.
You give two scores, each from 1 to 5: Correctness and Value & insight. Each comes with a confidence rating, also from 1 to 5. The scoring system explains this in detail. Scores are visible to everyone in the discussion and stay confidential with it. Correctness is independent of importance: a paper can be correct but trivial, or wrong but stimulating.
Markdown and LaTeX are supported.
Review summary
Each accepted paper is published with a “review summary” of roughly 150 to 500 words. It is a reader-oriented account of what the review process learned: strengths, weaknesses, caveats, historical context, and who should read the paper. The action editor invites one reviewer to write it for additional pay, and the writer may draw on what reviewers and the action editor wrote in the discussion. The authors need not endorse the summary, but they must find the wording acceptable before the paper can be published.
Once the summary is final, each reviewer and the action editor may sign it publicly. Reviewers are strongly encouraged to sign; if you prefer not to, tell the editor why. Any reason for declining is accepted. The summary is citable and will carry a DOI once the Journal’s Crossref registration is complete.
Confidentiality and anonymity
The paper is public: it is a preprint, and its anonymized PDF is listed on Papers under review. Your report, your scores, the discussion, and your identity are confidential during and after review, whatever the outcome. Your name appears only if you sign the review summary. The text you write in the discussion may be reused, verbatim or adapted, in the review summary.
The authors and other reviewers do not know who you are, and you should preserve this anonymity. If you refer to your own work, do so in the third person.
AI tools
AI tools are welcome during the review, for example, to search the document, define terms, check a derivation, or find related work. What you post must be written by you, and you must disclose your AI use in your report. One disclosure covers later comments involving the same kind of use. We strongly encourage you to form your own overall view of the paper before asking a tool for its opinion.
Please see the Journal’s AI usage policy for details.
Compensation
Your invitation states the base fee for the paper. On top of the base fee you can earn:
- $300 for an initial report within 15 days of the invitation;
- $500 if the action editor judges your review “outstanding”: one that substantially improves the manuscript or the reviewers' understanding of it;
- $400 for writing the review summary, if you are invited to.
Your pay does not depend on whether your assessment is positive or negative.
You may accept or waive the fee, and must decide before submitting your initial report. A waiver is final once you confirm it.
If you waive, you may recommend a 501(c)(3) nonprofit for the Journal to donate the same amount to. The recommendation is not binding: whether, how much, and to whom the Journal donates is its decision. You can add or change your recommendation until the paper closes.
No fee is earned until the paper leaves review. Payment covers the whole review process, so a reviewer who leaves before the paper closes is not paid. If the authors withdraw the paper, you are paid as normal.
The payments will be handled by Principles of Intelligence, the Journal’s fiscal sponsor. The Journal holds only the details our fiscal sponsor requires to initiate the payment process.
Questions and concerns
If you have questions or concerns related to the review process (e.g., conflicts of interest, process ambiguity, suspected plagiarism or AI-writing in the main text), send a message to the editors. For operational questions (e.g., payments), send a message to the journal staff.