Anthropic published a paper and a blog post a few days ago. Seems like every tech podcast I listen to talked about this paper this week (Hard Fork, Machine Gods, others). In the extra hour we got here in the US falling back to standard time1 I got a chance to read over the paper and accompanying blog.
Blog post: How AI agents shape experts’ judgment calls
Paper: Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls
Top line result: Professional analysts followed an AI agent’s randomly assigned preferences on almost all of the analytical decisions they made. Later, about half of the surveyed analysts rated the final recommendation as more their own ideas than those of the AI.
They asked 50 analysts to use Claude Code to evaluate an advisory program for a fictional farming cooperative. The different analytical choices all involved reasonable alternatives, things like measuring crop yield per hectare or yield per dollar spent. They used random draws to decide which options the agent would favor, and they did this without telling participants.
When the participants wrote up their explanations, over 3/4 repeated the agent’s arguments, and in some cases essentially word for word. Only 15% mentioned the AI in any explanation.
This suggests that asking people how much they relied on AI doesn’t really give us a complete picture of how much influence the AI had (see my previous post on AI disclosure skepticism). The result also speaks to this question of how much people accept whatever the AI suggests without much thought.
The concern here is that a data analysis can look sound and reasonable, while depending heavily on judgment calls its author barely considered. Deciding what to measure, or which outcomes deserve more weight, these can all change a recommendation even when the code and calculations are all correct.
I think with a carefully written AGENTS.md file you can get your AI assistant to make those choices explicit, and to then explain why you as the competent analyst might choose differently. Before putting my name on some analysis, I want be able to explain why those choices fit whatever research question I have, and how the conclusions might change under alternatives.
The superior time, better than DST in every way (if you disagree, you are incorrect).
This is why using AI on both how to succeed (get to a result) and how not to fail (review this, where are the gaps) is so important. It’s important to develop both as a human - you can use AI tooling to help with both. Your “AI as reviewer” articles are key here.