A cheap specialist judge gets used by agents but fails to reduce alignment audit costs (opens in new tab)

TL;DRI gave AuditBench's investigator agents a lightweight (Gemma 2-2B) EM-toxicity-scorer (judge) as an additional audit tool, targeting a proof-of-concept for misalignment detection at low cost, looking to validate that a specialized judge would (1) get used by these investigator agents, (2) help audits, (3) reduce overall evaluation spendValidation results:(1) Yes - the judge was used in every audit run. ~7 calls/audit without a prompt-given mandate, ~16 per audit with a mandate(2) Partial...