Are we ready to hand AI agents the keys?

On May 6, 2010, at 2:32 p.m. Eastern time, nearly a trillion dollars evaporated from the US stock market within 20 minutes—at the time, the fastest decline in history. Then, almost as suddenly, the market rebounded.

After months of investigation, regulators attributed much of the responsibility for this “flash crash” to high-frequency trading algorithms, which use their superior speed to exploit moneymaking opportunities in markets. While these systems didn’t spark the crash, they acted as a potent accelerant: When prices began to fall, they quickly began to sell assets. Prices then fell even faster, the automated traders sold even more, and the crash snowballed.

The flash crash is probably the most well-known example of the dangers raised by agents—automated systems that have the power to take actions in the real world, without human oversight. That power is the source of their value; the agents that supercharged the flash crash, for example, could trade far faster than any human. But it’s also why they can cause so much mischief. “The great paradox of agents is that the very thing that makes them useful—that they’re able to accomplish a range of tasks—involves giving away control.

Agents are already everywhere—and have been for many decades. Your thermostat is an agent: It automatically turns the heater on or off to keep your house at a specific temperature. So are antivirus software and Roombas. Like high-¬frequency traders, which are programmed to buy or sell in response to market conditions, these agents are all built to carry out specific tasks by following prescribed rules. Even agents that are more sophisticated, such as Siri and self-driving cars, follow prewritten rules when performing many of their actions. 

But in recent months, a new class of agents has arrived on the scene: ones built using large language models. Operator, an agent from OpenAI, can autonomously navigate a browser to order groceries or make dinner reservations. Systems like Claude Code and Cursor’s Chat feature can modify entire code bases with a single command. Manus, a viral agent from the Chinese startup Butterfly Effect, can build and deploy websites with little human supervision. Any action that can be captured by text—from playing a video game using written commands to running a social media account—is potentially within the purview of this type of system.

LLM agents don’t have much of a track record yet, but to hear CEOs tell it, they will transform the economy—and soon. OpenAI CEO Sam Altman says agents might “join the workforce” this year, and Salesforce CEO Marc Benioff is aggressively promoting Agentforce, a platform that allows businesses to tailor agents to their own purposes. The US Department of Defense recently signed a contract with Scale AI to design and test agents for military use. 

Scholars, too, are taking agents seriously. “Agents are the next frontier,” says Dawn Song, a professor of electrical engineering and computer science at the University of California, Berkeley. But, she says, “in order for us to really benefit from AI, to actually [use it to] solve complex problems, we need to figure out how to make them work safely and securely.”  

That’s a tall order. Like chatbot LLMs, agents can be chaotic and unpredictable. In the near future, an agent with access to your bank account could help you manage your budget, but it might also spend all your savings or leak your information to a hacker. An agent that manages your social media accounts could alleviate some of the drudgery of maintaining an online presence, but it might also disseminate falsehoods or spout abuse at other users. 

As of now, there’s no foolproof way to guarantee that agents will act as their developers intend or to prevent malicious actors from misusing them. And though researchers like Bengio are working hard to develop new safety mechanisms, they may not be able to keep up with the rapid expansion of agents’ powers. “If we continue on the current path of building agentic systems,” Bengio says, “we are basically playing Russian roulette with humanity.”

show less