Transparent random draw to select two $500 prize winners from the five honorable mentions in The Unjournal's 2024–25 Evaluator Prize.

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

How does The Unjournal evaluate research? This video walks through our six-step process for rigorous, transparent peer review. THE PROCESS: 1. Research curation/identification/submission, prioritization by field specialists on our team 2. Evaluation manager recruits expert evaluators (reviewers) 3. Detailed evaluation reports with quantitative ratings 4. Author response, evaluators can adjust 5. Manager synthesis and summary 6. Public dissemination (pubpub, academic databases) with DOIs 🔑 KEY FEATURES: • Evaluators are compensated (~avg. $350-$450 per review) • 3-5 week evaluation timelines • Benchmarked ratings; overall across criteria • Anonymous or signed evaluations • All materials publicly accessible with DOIs The Unjournal provides open, rigorous evaluation of quantitative research — especially economics, policy, and social science relevant to global priorities. 📚 Learn more: https://unjournal.org 📊 See our evaluations: https://unjournal.pubpub.org 📖 Full documentation: https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/ #OpenScience #PeerReview #AcademicResearch #Economics #ResearchEvaluation

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

The Unjournal is collaborating on Metaculus' first animal-focused forecasting tournament — harnessing collective intelligence to inform decisions that affect billions of animals.

Please aim to write a report up to the standards of a high-quality referee report for a traditional journal. Consider standard guidelines as well as The Unjournal's emphases. Remember to address any specific considerations mentioned by the evaluation manager, including in our bespoke evaluation notes. Please provide a concise summary of your evaluation below. Otherwise, please write your evaluation here, provide a link to it, or let us know you have emailed it. If you are linking or sending a file, we prefer the 'native format' (word, latex, markdown, bibtex, etc.) and not a pdf (but if necessary we can handle a pdf). These are percentile ratings relative to a reference group: serious research in the same area that you have encountered in the last few years. A rating of 50 means the paper is at the median of this reference group; 80 means it is in the top 20%; 20 means only 20% of comparable work is worse. See the guidelines on quantitative metrics for details. Tip: Use the Calibrate button above to practice rating sample papers and check your calibration. Guidance Judge the quality of the research heuristically. Consider all aspects of quality, credibility, importance to future impactful applied research, and practical relevance and usefulness, importance to knowledge production, and importance to practice. Benchmark: serious research in the same area encountered in the last three years. Guidance Consider the following: Are the methods well-justified and explained? Are they a reasonable approach to answering the question(s) in this context? Are the underlying assumptions reasonable? Are the results and methods likely to be robust to reasonable changes in assumptions? Does the author demonstrate robustness? Did the authors take steps to reduce bias from opportunistic reporting and questionable research practices? Guidance To what extent does the project contribute to the field or to practice, particularly in ways relevant to global priorities and impactful interventions? Focus on "improvements that are actually helpful" (applied stream) Originality and cleverness should be weighted less than typical journals — we focus on impact More weight on "contribution to global priorities" than "contribution to academic field" Do the paper's insights inform beliefs about important parameters and intervention effectiveness? Does the project add useful value to other impactful research? Sound, well-presented null results can also be valuable Guidance Are goals and questions clearly expressed? Are concepts clearly defined and referenced? Is the reasoning transparent? Assumptions explicit? Are all logical steps clear and correct? Does the writing make arguments easy to follow? Are conclusions consistent with the evidence presented? Do authors accurately characterize evidence and its support for main claims? Are data and analysis relevant to the arguments? Are tables, graphs, diagrams easy to understand (no major labeling errors)? Guidance This covers several considerations: Replicability, reproducibility, data integrity: Would another researcher be able to perform the same analysis and get the same results? Are methods explained clearly enough for credible replication? Is code provided? Is data source clear and as available as reasonably possible? Consistency: Do numbers in the paper and code output make sense? Are they internally consistent throughout? Useful building blocks: Do authors provide tools, resources, data, and outputs that might enable future work and meta-analysis? Reference: COS TOP Guidelines — a framework for evaluating transparency across 8 dimensions including data, code, materials, and preregistration. Guidance Is the topic and approach useful to global priorities, cause prioritization, and high-impact interventions? Does the paper consider real-world relevance, policy, and implementation questions? Are the setup, assumptions, and focus realistic? Do authors report results relevant to practitioners? Do they provide useful quantified estimates (costs, benefits) for impact quantification? Do they communicate in ways policymakers can understand without misleading oversimplification? Guidance Where should this paper be published based on merit alone? Imagine a journal process that is fair, unbiased, and free of noise — where status, connections, and lobbying don't matter. 0: Won't publish / little to no value 1: OK / Somewhat valuable journal 2: Marginal B-journal / Decent field journal 3: Top B-journal / Strong field journal 4: Marginal A-journal / Top field journal 5: A-journal / Top journal Non-integer scores encouraged (e.g., 4.6, 2.2). 0 Won't publish 1 OK 2 Marginal B 3 Top B 4 Marginal A 5 Top A Guidance Where will this research actually be published? If already published and you know where, report the prediction you would have given absent that knowledge. 0: Won't publish / little to no value 1: OK / Somewhat valuable journal 2: Marginal B-journal / Decent field journal 3: Top B-journal / Strong field journal 4: Marginal A-journal / Top field journal 5: A-journal / Top journal Non-integer scores encouraged (e.g., 4.6, 2.2). 0 Won't publish 1 OK 2 Marginal B 3 Top B 4 Marginal A 5 Top A This section is meant to help practitioners use this research to inform their funding, policymaking, and other decisions. It is not intended as a metric to judge the research quality per se. This is mainly relevant for empirical research. If 'claim assessment' does not make sense for this paper, please consult the evaluation manager, or skip this section. I. Identify the most important and impactful factual claim this research makes — e.g., a binary claim or a point estimate or prediction. III. [Optional] What additional information, evidence, replication, or robustness check would make you substantially more (or less) confident in this claim? IV. [Optional] Identify the important implication of the above claim for funding and policy choices. To what extent do you believe this implication? How should it inform policy choices? + Add Another Claim We generally incorporate this into the 'abstract' of your evaluation (see examples at unjournal.pubpub.org). Your comments here will not be public or seen by authors. Please use this section only for comments that are personal/sensitive in nature. Please place most of your evaluation in the public section. Please disclose your use of AI/LLM tools in this evaluation. AI tools may be used for specific tasks (literature search, methodology checks, writing assistance) but not for generating overall evaluations or ratings. You must independently verify any AI-generated suggestions. See recommended tools → Responses to these will be public unless you mention in your response that you want us to keep them private. Feedback (responses below will not be public or seen by authors) Would you be interested in discussing this with other evaluators and writing a joint report? See bit.ly/UJevalcollab. It will come with some additional compensation. If you and other evaluators are interested, we may follow up to arrange an (anonymous) discussion space or synchronous meetings. Tick here if you would like to be part of our 'evaluator pool' in the future. To be contacted for compensated evaluation work when research comes up in your area. To expedite this, fill out the EOI form at Join the Unjournal. Do you have any other suggestions or questions about this process or The Unjournal? (We will try to respond, and incorporate your suggestions.)

Follow The Unjournal on social media, subscribe to our newsletter, and stay up to date on open evaluation of impactful research.

Evaluation Summary and Metrics: "A review of GiveWell’s discount rate" for The Unjournal.

Knowledge Futures mission is to make information useful.

We've evaluated 55+ papers/projects across global health, animal welfare, AI governance, environmental risks, and development economics; with more in the pipeline. We've built tools, databases , and partnerships. We’re now focusing on understanding and enhancing our impact — “ is this actually u

Suppose you are convinced that public expert evaluation and rating of publicly-hosted research is better than traditional journal accept/reject peer-review. You are considering using, submitting, or supporting The Unjournal, but you want to know more about how it works. Here's a talk-through explanaiton.

Another "unjournal in a nutshell" produced with the help of NotebookLM (content at https://notebooklm.google.com/notebook/bc4d4d2b-efd7-4b28-80e1-55b6b740f92f) and some overdub. Planning to produce a few of these explainer videos coming soon. Let me know if you find this type video compelling or useful, and if so, what would you like to see next. And If you want AI only: https://notebooklm.google.com/notebook/bc4d4d2b-efd7-4b28-80e1-55b6b740f92f?artifactId=7859c271-8f03-44c5-bdb8-08f125700cd7

Evaluation Summary and Metrics: "Adjusting for Scale-Use Heterogeneity in Self-Reported Well-Being" for The Unjournal.

Longer discussion with Professor Larisa Cioaca of the Freeman School of Business, on Unjournal.org and her experience with their evaluation of her co-authored paper "The Effect of Public Science on Corporate R&D" (see https://unjournal.pubpub.org/pub/evalsumpublicscience/). Interviewer: Anthony Rowett (Unjournal Field Specialist) Further response: David Reinstein, Founder and Co-Director of The Unjournal

We've evaluated 55+ papers/projects across global health, animal welfare, AI governance, and development economics; and more in the pipeline . We've built tools, databases, and partnerships. We’re now focusing on understanding and enhancing our impact — “ is this actually useful, who is using it,