Bengio will lead a team of more than 15 researchers who are initially working to build a new technical solution called Scientist AI that’s meant to act as a guardrail for AI agents. — Photo by Igor Omilaev on Unsplash
Yoshua Bengio, an artificial intelligence pioneer, is creating a new nonprofit research organisation to promote an alternative approach to developing cutting-edge AI systems, with the aim of mitigating the technology’s potential risks.
The nonprofit, called LawZero, is set to launch June 3 with US$30mil (RM127.45mil) in backing from one of former Google chief executive officer Eric Schmidt’s philanthropic organisations and Skype co-founder Jaan Tallinn, among others.
Bengio will lead a team of more than 15 researchers who are initially working to build a new technical solution called Scientist AI that’s meant to act as a guardrail for AI agents.
OpenAI, Anthropic and other AI companies have increasingly focused on developing agents that can field more complex tasks on a user’s behalf, with little human involvement.
By contrast, Bengio said LawZero is designing a system that will act like "a selfless, idealised scientist” that learns to understand the world rather than act in it. The goal is for this model to be used in tandem with leading AI agents and provide oversight of these systems, minimising potential harms.
Bengio, a professor of computer science at the Université de Montréal, is considered one of the "godfathers” of AI, along with fellow academics Geoff Hinton and Yann LeCun.
In recent years, Bengio, Hinton and others have raised concerns about whether AI is progressing so rapidly that it might one day become impossible for humans to fully control.
While AI has become more adept at helpful tasks like research and coding, some systems have also demonstrated concerning capabilities, including deception, self-preservation and making up false information.
Anthropic recently said that during prerelease safety testing, its latest AI model tried to blackmail an engineer in order to avoid being replaced by another system.
"We don’t know how to design these very powerful AIs so that they will just follow our instructions,” Bengio said. "If we don’t figure it out in time – which could be a matter of years – we will be taking terrible risks.”
(LawZero is a nod to science fiction writer Isaac Asimov’s Zeroth Law for robotics, which places the protection of humanity as a whole above all else.)
Bengio said the most capable AI agents include a piece of code called a monitor that’s intended to act as a guardrail. However, this monitor is designed in a similar way to the underlying system, which can make it an inadequate check on the AI’s behaviour, Bengio said.
"You don’t give the keys to the prison to a criminal,” he said. "We want to put a trustworthy AI in charge of checking it.”
Bengio said he has held discussions with OpenAI, Google and Anthropic about his project, as well as with political leaders.
Bengio thinks that different actors can agree on the need to build AI safely – even as many in Washington and Silicon Valley appear more focused on the AI arms race with China. "Nobody wants to create a monster,” he said. – Bloomberg
