What if every scientific paper you read was just the “highlight reel” of a much longer, messier, and more complicated movie? You see the breakthrough, but you never see the hundreds of hours of footage showing what didn’t work.
Ultimately, the ARA marks a shift toward a future where “The Last Human-Written Paper” isn’t the end of science, but the beginning of a much deeper, machine-readable conversation.
However, this shift toward radical transparency comes with its own set of hurdles. While ARAs make AI agents more efficient, the study found a “prior-run box” effect where seeing a human’s past failures actually limited an AI’s ability to think outside the box and find creative new solutions. There is also a significant cultural and technical gap to bridge: the system relies on researchers being willing to expose their “messy” unfinished work, and even with better data, the jump in actual experiment reproduction was relatively modest. Furthermore, the reliance on “compilers” to translate old papers into this new format risks baking in errors or “hallucinations” if the original source was vague, proving that while machine-readable data is powerful, it isn’t a magic fix for the inherent complexities of scientific discovery.
We systematically evaluated the medical reasoning abilities of an LLM across six diverse experiments, comparing the model with hundreds of expert physicians. Overall, the model outperformed physicians across experiments, including in cases utilizing real and unstructured clinical data taken directly from the health record in an emergency department. These diagnostic touchpoints mirror the high-stakes decisions taken in emergency medicine departments, where nurses and clinicians make time-sensitive choices with limited information. Our results showed that humans, GPT-4o, and o1 all improved their diagnostic abilities as more information was available; o1 outperformed humans at multiple touchpoints, with the widest gap at initial ER triage, where there is the least information available.
The rapid pace of improvement in LLMs has substantial implications for the science and practice of clinical medicine. Although applying AI to assist with clinical decision support is sometimes viewed as a high-risk endeavor (22, 23), greater use of these tools might serve to mitigate the human and financial costs of diagnostic error, delay, and lack of access (24, 25). Our findings suggest the urgent need for prospective trials to evaluate these technologies in real-world patient care settings and for health care systems to prepare for investments for computing infrastructure and design for clinician-AI interaction that can facilitate the safe integration of AI tools into patient-care workflows. This includes the development of robust monitoring frameworks to oversee the broader implementation of AI clinical decision support systems (22), monitoring not just final diagnostic accuracy but other metrics crucial for successful deployment, including safety, efficiency, and cost.
We emphasize that our study addresses only text-based performance for both humans and machines; clinical medicine is multifaceted and awash with nontext inputs, including auditory (such as the patient’s level of distress) and visual information (for example, interpretation of medical imaging studies) that clinicians routinely use. Existing studies suggest that current foundation models are more limited in reasoning over nontext inputs (26, 27); future work is needed to assess how humans and machines may effectively collaborate (28) in use of nontext signals. This requires new benchmarks, trials, and technological solutions to more faithfully measure clinical encounters. Existing investment in increasingly pervasive ambient AI scribes and other passive monitoring technologies holds promise to serve as the basis for such investigations.
