A limited demonstration that AI reviews can do the job of human reviewers at improving papers. Deciding what to publish still needs an accountable editor with human reviewers they trust.
Excellent topic. And 100% agree. In my own experience I’ve had very good success with a “Reviewer #2” agent I’ve developed for my own papers prior to submission to peer review. It’s been incredibly useful in identifying blind spots and biases in my own papers over the last year or so. The key improvement was to go beyond just a long prompt and instead iteratively developed an agent.md and set of skills.md files for a Claud agent, that also is provided with a set of assets as well specific for the draft it’s going to review. If it doesn’t have access to the right information or sites, it will prompt me to provide additional files for its context window to take in.
I haven’t used this as part of my own reviewing of manuscripts - although it’s temping - mainly because nearly all journals prohibit the use of AI for assisting in peer review. Flipping this around though, I agree and I think the journals need to start acknowledging the power of these models and agents developed around them.
Also agree with what I think your sort of pointing at here; the role of peer reviewers in the future, and of scientists in general - will be more about our context, experience, creativity and insight more than specific technical skills. It’s like that Rick Rubin clip about music and being a producer with no technical skills whatsoever.
So… what then becomes of early career scientists, recent grads, and postdocs who don’t have the experience that you and I may have (largely due to years of work)? I’ve been thinking a lot about how new reviewers (and, while off topic, early career scientists in general) will get the experience they need if all of these “basic” technical tasks are just plain “better” when a robust agent does the work. I feel like this is a crisis in our field (probably all scientific / technical fields). How are academics handling this?
You should write about your own agent[.]md and skill[.]md files on your Substack. Just sayin :)
In our paper linked in the post, we talk about how AI is already being used for peer review, but clandestinely, without any control. With receipts.
To your last question. It's a good one. I haven't seen a good answer yet. Yes, you and I can do all the "higher level" things that scientists do, from writing to reviewing to code review and architecture decisions. But only because we've been doing it for decades. I wrote some thoughts on this here. Not encouraging. https://blog.stephenturner.us/p/fewer-rungs-on-the-ladder
Hard agree. If find that the bots do a great job of actually reading the work put in front, plus are incredibly good at checking methods/analysis when they are written. I've had frontier models point out very nieche , but hugely important, errors in my work that 1) were not picked up by coauthors or 2) likely would ever be looked at by a peer reviewer since who actually has time or energy to re-run or read the various raw scripts. I'm so curious to see how Journals pick this up, and how interesting it'll be to have public prompts from Journals, but I think it'll be so useful
Excellent topic. And 100% agree. In my own experience I’ve had very good success with a “Reviewer #2” agent I’ve developed for my own papers prior to submission to peer review. It’s been incredibly useful in identifying blind spots and biases in my own papers over the last year or so. The key improvement was to go beyond just a long prompt and instead iteratively developed an agent.md and set of skills.md files for a Claud agent, that also is provided with a set of assets as well specific for the draft it’s going to review. If it doesn’t have access to the right information or sites, it will prompt me to provide additional files for its context window to take in.
I haven’t used this as part of my own reviewing of manuscripts - although it’s temping - mainly because nearly all journals prohibit the use of AI for assisting in peer review. Flipping this around though, I agree and I think the journals need to start acknowledging the power of these models and agents developed around them.
Also agree with what I think your sort of pointing at here; the role of peer reviewers in the future, and of scientists in general - will be more about our context, experience, creativity and insight more than specific technical skills. It’s like that Rick Rubin clip about music and being a producer with no technical skills whatsoever.
So… what then becomes of early career scientists, recent grads, and postdocs who don’t have the experience that you and I may have (largely due to years of work)? I’ve been thinking a lot about how new reviewers (and, while off topic, early career scientists in general) will get the experience they need if all of these “basic” technical tasks are just plain “better” when a robust agent does the work. I feel like this is a crisis in our field (probably all scientific / technical fields). How are academics handling this?
You should write about your own agent[.]md and skill[.]md files on your Substack. Just sayin :)
In our paper linked in the post, we talk about how AI is already being used for peer review, but clandestinely, without any control. With receipts.
To your last question. It's a good one. I haven't seen a good answer yet. Yes, you and I can do all the "higher level" things that scientists do, from writing to reviewing to code review and architecture decisions. But only because we've been doing it for decades. I wrote some thoughts on this here. Not encouraging. https://blog.stephenturner.us/p/fewer-rungs-on-the-ladder
Hard agree. If find that the bots do a great job of actually reading the work put in front, plus are incredibly good at checking methods/analysis when they are written. I've had frontier models point out very nieche , but hugely important, errors in my work that 1) were not picked up by coauthors or 2) likely would ever be looked at by a peer reviewer since who actually has time or energy to re-run or read the various raw scripts. I'm so curious to see how Journals pick this up, and how interesting it'll be to have public prompts from Journals, but I think it'll be so useful