Title:Scientific judgment drifts over time in AI ideation
Abstract:Scientific discovery begins with ideas, yet evaluating early-stage research concepts is a subtle and subjective human judgment. As large language models (LLMs) are increasingly tasked with generating scientific hypotheses, most systems assume that scientists’ evaluations form a fixed gold standard, and that scientists’ judgments do not change. Here we challenge this assumption. In a two-wave study with 7,182 ratings from 57 active researchers across six scientific departments, each participant repeatedly evaluated a constant “control” research idea alongside AI-generated ideas. We show that scientists’ rati…
Title:Scientific judgment drifts over time in AI ideation
Abstract:Scientific discovery begins with ideas, yet evaluating early-stage research concepts is a subtle and subjective human judgment. As large language models (LLMs) are increasingly tasked with generating scientific hypotheses, most systems assume that scientists’ evaluations form a fixed gold standard, and that scientists’ judgments do not change. Here we challenge this assumption. In a two-wave study with 7,182 ratings from 57 active researchers across six scientific departments, each participant repeatedly evaluated a constant “control” research idea alongside AI-generated ideas. We show that scientists’ ratings of the very same idea systematically drift over time: overall quality scores increased by 0.61 points on a 0-10 scale (P = 0.005), and test-retest reliability was only moderate across core dimensions of scientific value, revealing systematic temporal drift in perceived idea quality. Yet the internal structure of judgment remained stable, such as the relative importance placed on originality, feasibility, clarity. We then aligned an LLM-based ideation system to first-wave human ratings and used it to select new ideas. Although alignment improved agreement with Wave-1 evaluations, its apparent gains disappeared once drift in human standards was accounted for. Thus, tuning to a fixed human snapshot produced improvements that were transient rather than persistent. These findings reveal that human evaluation of scientific ideas is not static but a dynamic process with stable priorities and requires shifting calibration. Treating one-time human ratings as immutable ground truth risks overstating progress in AI-assisted ideation and obscuring the challenge of co-evolving with changing expert standards. Drift-aware evaluation protocols and longitudinal benchmarks may therefore be essential for building AI systems that reliably augment, rather than overfit to, human scientific judgment.
| Subjects: | Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2511.04964 [cs.HC] |
| (or arXiv:2511.04964v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2511.04964 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lingyu Zhang [view email] [v1] Fri, 7 Nov 2025 03:57:47 UTC (5,579 KB)