How the AI that drives ChatGPT will move into the physical world

Covariant’s AI-powered Robotic Putwall system autonomously sorts items at the company’s headquarters in Emeryville, California. By combining camera and sensory data with the enormous amounts of text used to train chatbots like ChatGPT, Covariant has built AI technology that gives its robots a much broader understanding of the world around it. — The New York Times

EMERYVILLE, California: Companies such as OpenAI and Midjourney build chatbots, image generators and other artificial intelligence tools that operate in the digital world.

Now, a startup founded by three former OpenAI researchers is using the technology development methods behind chatbots to build AI technology that can navigate the physical world.

Covariant, a robotics company headquartered in Emeryville, California, is creating ways for robots to pick up, move and sort items as they are shuttled through warehouses and distribution centers. Its goal is to help robots gain an understanding of what is going on around them and decide what they should do next.

The technology also gives robots a broad understanding of the English language, letting people chat with them as if they were chatting with ChatGPT.

The technology, still under development, is not perfect. But it is a clear sign that the AI systems that drive online chatbots and image generators will also power machines in warehouses, on roadways and in homes.

Like chatbots and image generators, this robotics technology learns its skills by analysing enormous amounts of digital data. That means engineers can improve the technology by feeding it more and more data.

Covariant, backed by US$222mil (RM1.03bil) in funding, does not build robots. It builds the software that powers robots. The company aims to deploy its new technology with warehouse robots, providing a road map for others to do much the same in manufacturing plants and perhaps even on roadways with driverless cars.

The AI systems that drive chatbots and image generators are called neural networks, named for the web of neurons in the brain.

By pinpointing patterns in vast amounts of data, these systems can learn to recognise words, sounds and images – or even generate them on their own. This is how OpenAI built ChatGPT, giving it the power to instantly answer questions, write term papers and generate computer programs. It learned these skills from text culled from across the internet. (Several media outlets, including The New York Times, have sued OpenAI for copyright infringement.)

Companies are now building systems that can learn from different kinds of data at the same time. By analysing both a collection of photos and the captions that describe those photos, for example, a system can grasp the relationships between the two. It can learn that the word “banana” describes a curved yellow fruit.

OpenAI employed that system to build Sora, its new video generator. By analysing thousands of captioned videos, the system learned to generate videos when given a short description of a scene, like “a gorgeously rendered papercraft world of a coral reef, rife with colourful fish and sea creatures”.

Covariant, founded by Pieter Abbeel, a professor at the University of California, Berkeley, and three of his former students, Peter Chen, Rocky Duan and Tianhao Zhang, used similar techniques in building a system that drives warehouse robots.

The company helps operate sorting robots in warehouses across the globe. It has spent years gathering data – from cameras and other sensors – that shows how these robots operate.

“It ingests all kinds of data that matter to robots – that can help them understand the physical world and interact with it,” Chen said.

By combining that data with the huge amounts of text used to train chatbots such as ChatGPT, the company has built AI technology that gives its robots a much broader understanding of the world around it.

After identifying patterns in this stew of images, sensory data and text, the technology gives a robot the power to handle unexpected situations in the physical world. The robot knows how to pick up a banana, even if it has never seen a banana before.

It can also respond to plain English, much like a chatbot. If you tell it to “pick up a banana”, it knows what that means. If you tell it to “pick up a yellow fruit”, it understands that, too.

It can even generate videos that predict what is likely to happen as it tries to pick up a banana. These videos have no practical use in a warehouse, but they show the robot’s understanding of what’s around it.

“If it can predict the next frames in a video, it can pinpoint the right strategy to follow,” Abbeel said.

The technology, called RFM, for robotics foundational model, makes mistakes, much like chatbots do. Though it often understands what people ask of it, there is always a chance that it will not. It drops objects from time to time.

Gary Marcus, an AI entrepreneur and an emeritus professor of psychology and neural science at New York University, said the technology could be useful in warehouses and other situations where mistakes are acceptable. But he said it would be more difficult and riskier to deploy in manufacturing plants and other potentially dangerous situations.

“It comes down to the cost of error,” he said. “If you have a 150-pound robot that can do something harmful, that cost can be high.”

As companies train this kind of system on increasingly large and varied collections of data, researchers believe it will rapidly improve.

That is very different from the way robots operated in the past. Typically, engineers programmed robots to perform the same precise motion again and again – such as pick up a box of a certain size or attach a rivet in a particular spot on the rear bumper of a car. But robots could not deal with unexpected or random situations.

By learning from digital data – hundreds of thousands of examples of what happens in the physical world – robots can begin to handle the unexpected. And when those examples are paired with language, robots can also respond to text and voice suggestions, as a chatbot would.

This means that like chatbots and image generators, robots will become more nimble.

“What is in the digital data can transfer into the real world,” Chen said. – The New York Times