For many surgeons, the possibility of going back into the operating room to review the actions they carried out on a patient could provide invaluable medical insights.
Using a mix of Facebook’s PyTorch framework and machine-learning platform Allegro Trains, med-tech company theator is now providing surgeons with a tool that lets them watch over and analyze in detail the past operations they have performed, and access video footage of procedures carried out by colleagues around the world.
Dubbed the “surgical intelligence platform”, theator’s platform uses computer vision technology to extract key information from videos taken during surgical operations. The data is annotated, compiled and organized to let doctors review specific content by simply typing in key words through the platform. Surgeons can use the tool to jump to a specific step, re-watch critical moments, or access analysis about the procedure, such as time taken to perform a given action.
theator’s data scientists use a number of sophisticated machine-learning models to index and catalogue the video content provided by surgeons – but they quickly realized, as they were faced with mounting content, that manually training the models was a task more challenging than anticipated.
“We realized that running all of these processes manually was infeasible and automating training pipelines was an absolute must,” said Omri Bar, research team lead at theator in a new blog post. “Now, when new data comes in, it’s immediately processed and fed directly into training pipelines – speeding up workflow, minimizing human error, and freeing up our research team for more important tasks.”
Machine-learning pipelines consist of many different processes including data collection, preparation and cleaning, feature extraction or model validation. These tasks are traditionally completed by data scientists, which is more costly and can take up to several months.
Automating some of the more time-consuming and repetitive tasks that are necessary for the development of a machine-learning model, therefore, enables organizations to deploy the technology more efficiently and with less resources.
theator’s data science team was faced with many different processes that needed to be automated in the machine-learning training pipeline. A flexible tool was required to make sure that experiments could be carried out with the company’s in-house practices, which are tailored for the surgical domain. Facebook’s PyTorch framework, an open-source machine-learning library that is often used for computer vision applications, soon emerged as the technology of choice.
“We identified PyTorch as the best framework to deliver the flexibility that we needed,” said Bar. “With PyTorch we can easily disassemble and reassemble the parts we want to focus on and it provides frictionless top to bottom debugging capabilities.”
For example, the team found PyTorch’s video handling modules useful to facilitate the processing of heavy content. A surgical video that lasts several hours is typically represented as a huge 4D tensor, which makes it difficult to digest; using PyTorch’s, theator’s team was able to write customized data loading functionalities that accelerated the training of models.
Bar and his team have now successfully automated the entire process of machine-learning training for the surgical platform, from model development to deployment – as well as model improvement. The platform effectively includes a feedback loop: the data that is delivered to end-users is re-integrated within existing datasets for continuous training of the models.
“We’ve evolved from a company that relied on traditional machine-learning practices to a fully automated, scalable organization,” said Bar. “Along the way, we’ve saved our researchers and engineers countless hours by allowing them to focus on where the true value-add lies.”
On top of PyTorch, theator is also leveraging a machine-learning platform called Allegro Trains, which manages the data that comes in through the different models’ pipelines and organizes them in a queryable way. Trains can orchestrate several pipelines at the same time, identifying double data or flagging content coming from a new source.
At the other end of this complex machine-learning process, surgeons can now access a new wealth of data at the click of a button. Knowing how many minutes were spent performing various elements of an operation could be key to improving their technique, and help them make informed decisions in future operations.
Crucially, surgeons can also use the platform to learn from other professionals’ experiences. According to theator, the company is already sitting on AI-annotated video content from over ten thousand procedures, which users can access and browse to gather better insights on any given operation.
The surgical intelligence platform has already been fitted to operation rooms in Ichilov Hospital in Tel Aviv, and the company said that the technology is also active in a number of hospitals, medical schools and research centers in North America, including public research university McGill in Montreal.
