A robot, trained for the first time by watching videos of seasoned surgeons, executed the same surgical procedures as skilfully as the human doctors.
The successful use of imitation learning to train surgical robots eliminates the need to programme robots with each individual move required during a medical procedure and brings the field of robotic surgery closer to true autonomy, where robots could perform complex surgeries without human help.
“It’s really magical to have this model and all we do is feed it camera input and it can predict the robotic movements needed for surgery,” said study senior author and Johns Hopkins University associate professor of mechanical engineering Dr Axel Krieger.
“We believe this marks a significant step forward toward a new frontier in medical robotics.”
The findings led by Johns Hopkins researchers in the United States were highlighted at the Conference on Robot Learning held Nov 6-9 (2024) in Munich, Germany, a top event for robotics and machine learning.
The team, which included Stanford University researchers, used imitation learning to train the da Vinci Surgical System robot to perform fundamental surgical procedures: manipulating a needle, lifting body tissue and suturing.
The model combined imitation learning with the same machine learning architecture that underpins ChatGPT.
However, where ChatGPT works with words and text, this model speaks “robot” with kinematics, a language that breaks down the angles of robotic motion into mathematics.
The researchers fed their model hundreds of videos recorded from wrist cameras placed on the arms of da Vinci robots during surgical procedures.
These videos, recorded by surgeons all over the world, are used for post-operative analysis and then archived.
Nearly 7,000 da Vinci robots are used worldwide, and more than 50,000 surgeons are trained on the system, creating a large archive of data for robots to “imitate”.
While the da Vinci system is widely used, researchers say it’s notoriously imprecise.
But the team found a way to make the flawed input work.
The key was training the model to perform relative movements rather than absolute actions, which are inaccurate.
“All we need is image input and then this AI system finds the right action,” said study lead author and Johns Hopkins robotics postdoctoral researcher Dr Ji Woong “Brian” Kim.
“We find that even with a few hundred demos, the model is able to learn the procedure and generalise new environments it hasn’t encountered.”
The team trained the robot to perform three tasks: manipulate a needle, lift body tissue and suture.
In each case, the robot trained on the team’s model performed the same surgical procedures as skilfully as human doctors.
“Here the model is so good, learning things we haven’t taught it,” Assoc Prof Krieger said.
“Like if it drops the needle, it will automatically pick it up and continue. This isn’t something I taught it to do.”
The model could be used to quickly train a robot to perform any type of surgical procedure, the researchers said.
The team is now using imitation learning to train a robot to perform not just small surgical tasks, but a full surgery.
Before this advancement, programming a robot to perform even a simple aspect of a surgery required hand-coding every step.
Someone might spend a decade trying to model suturing, Assoc Prof Krieger said.
And that’s suturing for just one type of surgery.
“It’s very limiting,” he said.
“What is new here is we only have to collect imitation learning of different procedures, and we can train a robot to learn it in a couple days.
“It allows us to accelerate to the goal of autonomy while reducing medical errors and achieving more accurate surgery.”
