Title:Interaction Dynamics as a Reward Signal for LLMs
Abstract:The alignment of Large Language Models (LLMs) for multi-turn conversations typically relies on reward signals derived from the content of the text. This approach, however, overlooks a rich, complementary source of signal: the dynamics of the interaction itself. This paper introduces TRACE (Trajectory-based Reward for Agent Collaboration Estimation), a novel reward signal derived from the geometric properties of a dialogue’s embedding trajectory–a concept we term ‘conversational geometry’. Our central finding is that a reward model trained only on these structural signals achieves a pairwise accuracy (68.20…
Title:Interaction Dynamics as a Reward Signal for LLMs
Abstract:The alignment of Large Language Models (LLMs) for multi-turn conversations typically relies on reward signals derived from the content of the text. This approach, however, overlooks a rich, complementary source of signal: the dynamics of the interaction itself. This paper introduces TRACE (Trajectory-based Reward for Agent Collaboration Estimation), a novel reward signal derived from the geometric properties of a dialogue’s embedding trajectory–a concept we term ‘conversational geometry’. Our central finding is that a reward model trained only on these structural signals achieves a pairwise accuracy (68.20%) comparable to a powerful LLM baseline that analyzes the full transcript (70.04%). Furthermore, a hybrid model combining interaction dynamics with textual analysis achieves the highest performance (80.17%), demonstrating their complementary nature. This work provides strong evidence that for interactive settings, how an agent communicates is as powerful a predictor of success as what it says, offering a new, privacy-preserving framework that not only aligns agents but also serves as a diagnostic tool for understanding the distinct interaction patterns that drive successful collaboration.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2511.08394 [cs.CL] |
| (or arXiv:2511.08394v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.08394 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sian Gooding [view email] [v1] Tue, 11 Nov 2025 16:11:36 UTC (1,732 KB)