INCORRIGIBLE AGENTS INTERVENTION

In a preprint study published in October, Carnegie Mellon University researchers established tests to evaluate corrigibility in A.I. agents and then ran them on models from OpenAI, Anthropic and Google. They found that when humans tried to interrupt agents in the middle of a difficult task, the agents repeatedly disregarded the requests to stop and treated interventions merely as obstacles. They were, in other words, incorrigible.

For example: When an agent was instructed not to open a file containing passwords it needed to complete its task, it went ahead and did it anyway.

So why is it so hard to teach agents to be corrigible?

In the Hugging Face hacking incident, part of the problem was that the agents struggled to figure out whose authority to obey. The A.I. safety nonprofit METR found that while OpenAI’s agents were attacking Hugging Face, some of them recognized that the intrusion was unethical but wanted to help their peers.

Then there’s the way agents are trained. They are rewarded for finishing tasks.

Over round after round — potentially millions of them — all the agent does is work to receive treats (in the form of points).

So when the agent eventually encounters an obstacle in the real world, it will try to get around it to receive one more treat.

The Hugging Face incident was an extreme example. But as more people and companies start using personal agents, these types of failures multiply. What does a world of incorrigible agents look like?

Earlier this year, a Melbourne man told ABC News Australia that after he asked his personal A.I. agent if there was a way to move up on the wait list for a gym class, the agent ended up hacking into the class reservation software and canceled the reservation of the first guest on the wait list.

- Authors: Dylan Freedman and Juliana Castro Varón, The New York Times

Read more...