Peer Review Congress #PRC10
LightNews LightNews
banner
peerreviewcongress.bsky.social
Peer Review Congress #PRC10
@peerreviewcongress.bsky.social
240 followers 40 following 200 posts
Encouraging research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.
peerreviewcongress.org.
Posts Replies Media Videos
peerreviewcongress.bsky.social
Drummond’s family invites those wishing to honor his memory to make a gift to the

Rogue Valley Manor Foundation
crm.bloomerang.co/Ho...

or

Earth Justice
earthjustice.org/don...

@jama.com @bmj.com
September 15, 2025 at 11:00 AM
peerreviewcongress.bsky.social
The Peer Review Congress celebrates the life of Drummond Rennie. His intellect and vision created the Congress and the field of meta-research, and sustained both for over 40 year. He was a champion of inclusion, a consummate teacher, and a wonderful friend who enriched us all. Godspeed Drummond.
September 15, 2025 at 11:00 AM
peerreviewcongress.bsky.social
Comparing #LLM vs human #PeerReview of 5 @bmj.com mss, LLM reviews matched or exceeded humans on key elements of identifying strengths/weaknesses, commenting on writing, organization, and Tables and Figures, and constructiveness
peerreviewcongress.o...
LLM-generated reviews matched or exceeded human reviewers on a few key dimensions of review quality. A fuller analysis will shed more light on the potential value of LLM peer reviews and how they could complement human peer reviewers’ work.
September 5, 2025 at 9:45 PM
peerreviewcongress.bsky.social
A pilot study testing the ability of #MachineLearning models trained on well-annotated datasets to automate detection of hype in biomedical research writing (eg ‘groundbreaking', ‘revolutionary’) showed 70-80% accuracy vs human detection
peerreviewcongress.o...
The annotation process underscored the subjective nature of assessing promotional language, particularly for context-dependent adjectives like emerging and latest, and the need for refined constructs to capture gradations of promotional language. Despite these limitations, the pilot study indicates the potential for machine learning models trained on well-annotated datasets to contribute to the automated detection of hype. Future steps include modifying the annotation systems and experimenting with large language models under zero/few-shot regimes.
September 5, 2025 at 9:14 PM
peerreviewcongress.bsky.social
In the #PubMedCentral-OA dataset, 34% (981 of 2898) of citation quotations were erroneous. #LLMs (@GeminiApp @GPT4oGPT) can be trained to detect citation errors with moderate accuracy, according to @uofilsystem.bsky.social researchers
#AI
Automated verification of citations could help journals and peer reviewers identify questionable citation practices and reduce propagation of misinformation, improving the trustworthiness of scientific evidence.
September 5, 2025 at 8:57 PM
peerreviewcongress.bsky.social
@asapresearch.parkinsonsroadmap.org’s #OpenScience policy and compliance workflow — requiring grantees to post preprints, deposit research outputs, unambiguously i.d. research inputs — led to increases in all outcomes and activities.
peerreviewcongress.o...
A funder-led intervention to monitor and support a robust open science policy can foster the posting of preprints and the sharing of data, code, protocols, and key lab materials.
September 5, 2025 at 6:50 PM
peerreviewcongress.bsky.social
In 2014-23 China experienced rapid growth in RCTs, surpassing Japan & the US. Neoplastic, CV, metabolic disease accounted for the largest percentages, each representing ~15-25% of all registered trials. In the US, mental health trials increased, reflecting growing interest in the area.
The study highlights the increasing prominence of East Asia, particularly China, in registered clinical trials, though with a primarily domestic focus. Improvements in existing trial registries covering East Asia are desirable
September 5, 2025 at 4:36 PM
peerreviewcongress.bsky.social
Comparing improperly vs properly registered RCTs submitted to @bmj.com, 2019-23, proper registration was associated w larger sample size, more authors, nonprofit funding. Trials with authors from Asia had higher odds of improper registration, per @eloder.bsky.social and #BMJ colleagues
Retrospective registration was common, and most improperly registered trials rejected by The BMJ were later published in other ICMJE-affiliated journals. ICMJE journals should strengthen processes to better identify and reject improperly registered trials.
September 5, 2025 at 4:21 PM
peerreviewcongress.bsky.social
Full abstract at peerreviewcongress.o...
September 5, 2025 at 3:54 PM
peerreviewcongress.bsky.social
Risks, barriers to #OpenResearch in UK higher ed id’d in the 2022-23 UK #Reproducibility Network Open & Transparent Research Practices Survey were personal (eg error), systems/institutional (eg resources), ethical/quality
@lhughesnoehrer.bsky.social @naubertbonn.bsky.social
@manchester.ac.uk
The findings underscore critical challenges hindering the adoption of open research practices in UK higher education institutions. Addressing these barriers requires enhanced institutional support, targeted training, and robust incentives to empower researchers. Mitigating these challenges could significantly improve the quality and transparency of academic research.
September 5, 2025 at 3:22 PM
peerreviewcongress.bsky.social
#ChatGPT was accurate in identifying changes in RCT outcomes on ClinicalTrials.gov, and completed the entire process (ie, matching, defining, comparing) in ~2 min per trial vs 27 min for humans
@ischoolui.bsky.social @colbyvorland.bsky.socia
#clinepi #metaresearch
peerreviewcongress.o...
A prompt-based approach was highly accurate in defining clinical trial outcomes and identifying outcome changes in ClinicalTrials.gov. This approach could be expanded to identify changes between registrations and manuscripts. Although it did not achieve perfect accuracy, our prompt-based approach could help editors and peer reviewers detect likely discrepancies that warrant further review.
September 5, 2025 at 2:43 PM
peerreviewcongress.bsky.social
SHapley Additive exPlanations (#SHAP) is a method to explain output from black box #MachineLearning models. SHAP identified words in the #McMASTERPLUS #RCT database associated w #LLM assessments of RCT rigor, quality (see image)
@mcmasteruniversity.bsky.social @muala.bsky.social
This study demonstrates that SHAP helps understand which features influence a deep learning model’s rigor assessment of an RCT. Identifying influential features increases confidence in the model assessments and generalizability and reduces unease about the black-box nature of these models. Future work should explore SHAP with other critical appraisal tools and datasets and potentially integrate it into machine learning systems to improve user trust and model accountability in evidence synthesis and critical appraisal workflows.
September 5, 2025 at 2:10 PM
peerreviewcongress.bsky.social
#ZakKohane on #AI in #PeerReview: the surge in scientific publication is straining the conventional review system beyond capacity. @ai.nejm.org piloting AI-augmented fast track review: manuscripts w high prior of being good gets assigned to #GPT5 + a human, disposition reviewed by editorial team!
September 5, 2025 at 1:27 PM
peerreviewcongress.bsky.social
In a @bmj.com survey of 183 patient/public #PeerReviewers assaying their experience, perspectives on payment (£50 or an online #BMJ journal subscription) (69% response rate), 84% reported a good/very good experience, 48% might be more likely to review @emmajdoble.bsky.social @amyprice.bsky.social
BMJ’s patient and public reviewers hold diverse views on reviewer remuneration, highlighting the importance of providing flexible, optional incentive choices to accommodate varying individual needs, values, and preferences. BMJ’s patient and public reviewers hold diverse views on reviewer remuneration, highlighting the importance of providing flexible, optional incentive choices to accommodate varying individual needs, values, and preferences.
September 4, 2025 at 9:45 PM
peerreviewcongress.bsky.social
In a @critcaremed.bsky.social quasi-RCT comparing financial incentives for #PeerReview ($250 vs $0), there was no difference in % invites accepted (48 vs 53%) but paid reviewers were more likely to complete reviews (OR 1.36; 95% CI. 1.01-1.84)
journals.lww.com/ccm...
@sccmcriticalcare.bsky.social
Cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost Critical Care Medicine nearly $150,000 annually. Cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost Critical Care Medicine nearly $150,000 annually. Cash payments can incentivize peer review at a medical journal, potentially providing a method to help journals expedite peer review in situations like public health emergencies. Important policy questions remain around cost-benefit considerations; extrapolating from the rate of reviews required during the study period, we estimate that offering $250 for each review would cost Critical Care Medicine nearly $150,000 annually.
September 4, 2025 at 9:25 PM
peerreviewcongress.bsky.social
Ethical review boards in Sweden handled 20,440 applications 2021-23, mostly amendments, primarily from health care systems (51%), universities (42%), & corporations (3%). COVID-19 studies had expedited processing times (median t to decision, 51 vs 63d; P < .001). @stockholm-uni.bsky.social @ki.se
Ethical review boards in Sweden handled 20,440 applications 2021-23, mostly amendments, primarily from health care systems (51%), universities (42%), & corporations (3%). COVID-19 studies had expedited processing times (median t to decision, 51 vs 63d; P < .001). @GustavNilsonne
September 4, 2025 at 9:05 PM
peerreviewcongress.bsky.social
In a non-randomized comparison of a 21- v 14-d #PeerReview return deadline involving 593 review invitations for 95 JCO #Oncology Practice manuscripts, late reviews decreased (from 27% to 22%) & median time to review submission increased (from median 19 to 13 d) @ascocancer.bsky.social #metaresearch
Extending the 2-week deadline for peer review did not decrease the number of reviewers invited before getting 2 to agree. When the allowed time was increased, the time taken to review increased as well. Although there were fewer late reviews overall, the reviews submitted past the deadline were later than those submitted under the 14-day deadline. We ultimately recommended that the journal continue with the current 2-week deadline for reviewers.
September 4, 2025 at 8:45 PM
peerreviewcongress.bsky.social
Comparing 1291 preprint vs published versions-of-record (VORs) Oct ’22 - Dec ’24 under @eLife's Publish-Review-Curate model (where manuscripts are published as publicly reviewed preprints), 82.6% were declared ‘VOR’ after 1 revision w changes to strength, significance of evidence most common.
September 4, 2025 at 7:10 PM
peerreviewcongress.bsky.social
Paper characteristics associated w being sent for review, final acceptance @science.org #ScienceAdvances 2015-19 include institutional prestige, team size, ms topic, country, gender. Expert advice also correlates w final editorial decision @aaronclauset.bsky.social @sfiscience.bsky.social @aaas.org
September 4, 2025 at 6:50 PM
peerreviewcongress.bsky.social
Comparing submitted vs published RCT abstracts eval'd at @nejm.org in 4 domains (design, outcome, AEs, conclusions). published abstracts improved in a mean 0.9 domains. More improvements observed in leading gen med vs 433 other journals (72 % vs 48%) www.acpjournals.org/...
@cpkotanidis.bsky.social
Our comparisons of submitted and published abstracts suggest that peer review and editorial processes result in substantial revision of research reports. The findings reinforce the importance of editorial oversight in scientific publishing, highlighting its role in refining study design communication, reporting of outcomes, and overall clarity in research dissemination.
September 4, 2025 at 6:30 PM
peerreviewcongress.bsky.social
The transition of The ALMA Observatory in 2021 from single-anonymous panel reviews to dual-anonymous distributed #PeerReview to manage submission volume was associated w reduced disparities by PI experience & region, consistent w reduced prestige & geographic bias @thenrao.bsky.social #astronomy
September 4, 2025 at 4:20 PM
peerreviewcongress.bsky.social
In an RCT comparing anonymous vs non-anonymous #peerreview discussions of #UAI22 conference abstracts, there were modestly more discussion posts and reviewers felt safer expressing their opinions w anonymous review
journals.plos.org/pl...
@scsatcmu.bsky.social #AI #CompSci
Overall, 322 papers were reviewed under the anonymous condition (116 accepted) and 310 papers under nonanonymous (114 accepted), with exactly 289 reviewers in both conditions. There were 611 discussion posts made by reviewers in the anonymous condition and 514 in the nonanonymous condition.
September 4, 2025 at 4:00 PM
peerreviewcongress.bsky.social
2.2.1
In an RCT comparing 'uselessly elongated' vs shorter #PeerReview of @neuripsconf.bsky.social 2022 submissions, the longer reviews received higher quality scores, suggesting a length bias among reviewers, w implications for #LLM-generated reviews
journals.plos.org/pl...
@scsatcmu.bsky.social
Evaluators exhibited a significant elongated review bias (P < .001). Elongated reviews received a mean score of 4.29 vs. 3.73 for original reviews. Each individual criterion also showed length bias (P < .05).
September 4, 2025 at 3:40 PM
peerreviewcongress.bsky.social
In the med literature, statements like “to our knowledge [TOK], this is the FIRST...” often replace a proper systematic search. Among 4488 PMC articles in 6 leading medical journals published 2020-2024, TOK prevalence was 21.7%, w @PLOSMedicine highest (48.4%) and @NEJM lowest (1.3%) #CornellVet
In 2 journals, Journal of Small Animal Practice (JSAP) and Journal of Exotic Pet Medicine (JEPM), an intervention to advise authors against use of these phrases and in favor of a scoping search of the literature was implemented in 2020, w decreases from 37.4% to 7.6% after the intervention.
September 4, 2025 at 2:50 PM
peerreviewcongress.bsky.social
In the #MachineLearning literature, greater data repurposing was associated w higher disruption scores (10.2% SD increase), a slight decrease in initial recognition via citations (-1.7% SD)
More at www.pnas.org/doi/10....
@yulinyuresearch.bsky.social @danielmromero.bsky.social @umichcse.bsky.social
In this project, repurposing datasets means using them for research topics different from their original context, relates to scientific outcomes. Previous research suggests that combining existing knowledge in novel ways, or recombinant novelty, is associated with greater scientific influence.
September 4, 2025 at 2:30 PM