AI researchers flag bias risks in LLM judging (opens in new tab)

Several AI evaluation studies in the supplied sources focused on weaknesses in how large language models are tested, judged and selected. The arXiv listings examined LLM judges in reasoning and coding tasks, self-preference bias in automated evaluation and targeted underperformance affecting vulnerable users. Developer-facing posts highlighted tools for comparing local models and benchmarking visual generation. The research updates said large reasoning models can outperform non-reasoning LLMs...