Anthropic Proposes Embedded AI Risk Evaluators with Limited Enforcement Power
Anthropic CEO Dario Amodei has proposed embedding third-party safety evaluators within frontier artificial intelligence (AI) companies to monitor and assess risks associated with increasingly powerful models. The proposal draws a parallel to bank supervisors, who have the authority to compel action or shut down institutions if necessary. However, experts like Julie Andersen Hill argue that without similar enforcement powers—such as halting operations—the analogy is flawed. Current AI evaluation methods can identify significant model failures but may not detect rare combinations of behaviors leading to catastrophic outcomes, according to Albert Ziegler at XBOW. While the evaluators would have access comparable to internal risk teams and rights to publish findings without editorial control (subject to limited redactions), they lack legal authority over development or deployment decisions. Neither Anthropic's proposal nor OpenAI’s existing evaluation framework grants independent evaluators power to stop model training or release, raising concerns about the effectiveness of such oversight in preventing potential AI risks.
Topics: AI regulation banking supervision analogy third-party evaluators