Before anything else: I write this as a tenured professor who loves this job. Academia has been good to me, and I plan to spend the rest of my career here.1 So this essay isn’t a criticism or indictment. Quite the contrary. I only bother critiquing the people and institutions I actually care deeply about and want to see improve. That’s the reason I want us to be more honest about how we measure and evaluate each other in academia. So read what follows as the feedback you might give a trusted colleague whose work you respect, maybe in a lab meeting, before the reviewers see it.
Last week Ran Blekhman published a short essay in PLOS Biology asking what biology research is for. I read it over the weekend. You can read the whole thing in 5 minutes. It starts off with something I think a lot of us feel but might not say out loud in a promotion and tenure meeting (emphasis added):
Put bluntly, it is now straightforward to generate a plausible scientific paper without doing any actual research. Scientists have to come to terms with what this means: the paper, the artifact of research that the biology research community has used to assess, fund, hire, and promote for a century, has come loose from the science it was meant to represent.
After receiving tenure last year I now serve on the P&T committee. I also interview every faculty candidate that comes through this place. I think most people sitting on hiring and P&T committees know that paper counts were always a flawed proxy long before ChatGPT. And yet. We still bean-count publications and note which ones landed in high-impact journals (while we all know and agree that impact factor is a deeply flawed measure of an individual researcher’s impact).
Blekhman argues the proxy might have at one time been useful, because a paper took effort, and that effort was evidence a student learned something or a dataset got built or some other groundwork was laid for future work.
Generative AI has broken this already-flawed proxy completely.
Generating a plausible manuscript now takes an afternoon or less. And plenty of journals will publish it without much scrutiny for a fee. Without a deliberate response, Blekhman predicts, we’ll desperately cling to our old ways and keep using the same old evaluation metrics like impact factor and journal prestige.
Some of our other ways of measuring “impact” don’t hold up much better. I don’t want anyone reading this to think I’m calling them out, dearest gentle reader, so I’ll talk about myself for a moment. On more than zero occasions, I have torched thousands of dollars of taxpayer money to fly across an ocean and give a 20-minute talk on an already-published paper to a room where half the audience was on their phones, then skipped half the remaining sessions to walk around Istanbul or Barcelona or Honolulu or Krakow. I enjoyed it, but in no world should we call such practice “impact.” And yet, we tally up these invited presentations as if they were.
On writing. In March 2025 I published a paper in PLOS ONE that has been viewed 409 times in the year and a half since.2 A blog post explaining it, written so people wouldn’t have to read the paper, passed 5,000 views in its first 24 hours. A popular science piece I wrote for The Conversation on the biosecurity risks of AI-run labs was read more than 40,000 times in the first week, yet an academic paper I wrote on AI and biosecurity has been viewed 167 times since I published the preprint months ago. I’m not saying page views are the right measure of impact. They are not, and they can be gamed. But the academic paper that no one ever reads is the item that counts in P&T or hiring, and the writing that reached tens of thousands of readers counts for little or nothing.3
Blekhman proposes a first pass at the goals of biology research: explaining how living systems work, improving human health, building resources like datasets and software, training the next generation, and keeping the trust of the public that pays for nearly all of it. Look closely. Publishing papers isn’t on the list. For the last hundred years a paper advanced at least one of those goals, and so maybe counting papers was a reasonable shortcut. It is no longer. I don’t have a new metric to propose. And the right response also isn’t “form a committee and write a report in a year.”
What to do? Maybe start by piping up in the appropriate setting (P&T committees, hiring and recruitment committees, etc.). If you’re on a P&T or a search committee, your voice is as equal as anyone else’s in that room, no matter how junior you are or senior they are. For each item a candidate lists on a CV, ask which of those goals it advanced and what evidence shows it. Do this instead of oooo-ing and ahhh-ing about the number of Science, Nature, and Cell papers someone lists on their CV. Asking that aloud at the next committee meeting requires no new policy. Yes, this takes time and effort. But consider that awarding tenure is one of the most consequential decisions you’ll ever make as a faculty member, essentially cementing the other party as a permanent colleague you’ll work with until you retire.
Blekhman closes the essay with a call to really think hard about what (biology) research is for. Read it again, but substitute “biology” for whatever field you’re in.
Goodhart’s law is often stated as the observation that, when a measure becomes a target, it stops being a good measure. ‘Publish more papers’ has been a target in biology for decades. The biology research community needs to define new ways to evaluate science that are aligned with its goals. This is work for the entire field: journals, funders, universities, and institutions. We, as a field, have got to find a better goal. Maybe, as a starting point, we should start by asking ourselves: what is biology research for?
And I love my own university, so much so that I left and came back after a 6-year Rumspringa.
PLOS also reports about 700 PDF downloads with no matching page view. Some are likely bots or scrapers, though Google Scholar often links directly to the PDF, so some are probably people. Even counting all of them, the paper hasn’t matched what the blog post got in a day.
We’re already thinking more broadly at my own school. In our own publicly available P&T policies we recognize things like “programming libraries, software products, curated data sets, and patents.” We encourage “use of diverse dissemination platforms—such as blogs, GitHub, YouTube, and other public-facing media—to share research insights, methods, or tools in accessible ways that contribute to scholarly and societal impact.” I think we’re really progressive on this front. But we, and others, have further to go.
I absolutely applaud this. In looking at this less from the academic impact perspective and more from the societal impact perspective but having contributed to Oxford's thinking around embedding engagement and impact into recruitment and progression criteria, the one thing I would say is that meaningful assessment of impact is hard - not because it is beyond us in principle but exactly because it's just not incentivised, and requires more than just looking at the easily gotten metrics. Why spend time actually trying to capture and understand whether anything you've done has made a difference (yourself or buying in the support) when it's not the thing that will get you or keep your job? Obviously in the UK we have REF but that incentivised the institution, not the individual really, and I can already hear the groans about the disproportionate effort putting together a solid impact case study can take...
Appreciate this essay! As someone figuring out what I’m going to do with my life after my postdoc, it’s pretty clear to me that counting pubs is not going to be useful going forward. That incentive structure has overwhelmed the desks of reviewers/editors everywhere as well.
Sadly, I’m not sure how many places are making real efforts to create new incentives that are legible to candidates and people up for promotion. Time will tell what schools land on a new system that takes us away from the path we are heading down…