A researcher receives an urgent call: a patient has just reported an unexpected side effect to an experimental treatment in a clinical trial. With such urgency, the researcher could spend hours, even days, combing through piles of medical data, patient records, and previous test results to find an explanation. Instead, he can use a multimodal artificial intelligence (AI) platform. Within seconds, the AI provides him with an in-depth analysis: it identifies not only the incident in question but also similar patterns observed in past clinical trials, while proposing avenues of adjustment to the treatment protocol. This level of speed and precision is revolutionary. Where human error and wasted time would once have delayed decision-making, multimodal AI steps in to offer immediate, targeted solutions. What was once a futuristic vision is now becoming common practice, thanks to the rise of multimodal AI in clinical trials.

Patient recruitment and management

One of the biggest challenges in clinical trials is recruiting the right patients. Around 53% of clinical trials fail in Phase II due to the low number of patients. This is mainly due to the difficulty of recruiting a sufficient number of participants who meet the strict criteria for participation, and almost all trials exceed their expected enrolment deadlines. Here, multimodal AI has a role to play by simultaneously analyzing millions of electronic medical records, internet search histories, and even data from social networks to identify the populations most likely to meet the inclusion criteria, enabling recruiting efforts to be targeted more effectively and reducing these under-recruitment failures.

A group of Stanford researchers led by biomedical data scientist James Zou has developed a system called Trial Pathfinder that analyzes a set of completed clinical trials and assesses how adjustment of participation criteria affects relative risks, or rates of adverse events such as serious illness or death among patients. This system was applied in one to clinical trials for lung cancer. They found that the criteria adjustment suggested by Trial Pathfinder would have doubled the number of eligible patients without increasing relative risk.

Real-time monitoring

Once patient recruitment is complete, the next crucial step is continuous monitoring of their health status. Thanks to connected devices such as smartwatches and biometric sensors, data on vital signs and activity levels are collected in real time. Multimodal AI uses this data to detect potential abnormalities well before they become critical. The Dana-Farber Cancer Institute, in collaboration with PathAI, conducted a clinical trial using multimodal AI to monitor patients with different types of cancer in real time. This study aimed to combine data from multiple modalities including pathological images, molecular biomarkers, and electronic clinical data to improve early detection of disease progression and personalize treatments being tested.

Optimization of clinical trial protocols

Multimodal AI does not only monitor, it also optimizes test protocols in real-time. Multimodal AI can simultaneously analyze this heterogeneous data to identify complex patterns and correlations that would not be obvious to human researchers – such as blood test results, biopsy images, and medical imaging reports – AI can then suggest adjustments to treatments based on individual patient responses. This dynamic flexibility helps to reduce patient risk and increase the likelihood of a successful clinical trial.

Roche, for example, has used multimodal artificial intelligence in its clinical trials to identify new biomarkers in patients with rare cancers. These cancers are often difficult to treat due to their biological heterogeneity and low incidence, making traditional clinical trials long and costly. To overcome these challenges, Roche has implemented a strategy combining multimodal AI technologies to analyze patient data more thoroughly and faster than conventional methods. By analyzing billions of data points, AI was able to identify specific biomarkers that indicate how a particular patient might respond to a certain therapy. By improving the accuracy of patient selection and dynamic treatment adjustments, Roche was able to reduce the time needed to reach meaningful clinical conclusions and lower the overall cost of trials.

Prediction and Management of Adverse Events

Adverse events are inevitable in clinical trials. However, thanks to multimodal AI, it is becoming possible to predict these effects even before they occur. At the AACR Annual Meeting 2022, researchers led by Dr. Bart Westerman of Cancer Center Amsterdam presented preliminary results of an AI model capable of predicting side effects resulting from new combination therapies. The model, called “the adverse event atlas”, uses multimodal data from the FDA’s FAERS database, which contains more than 15 million adverse event records.

Using a convolutional neural network algorithm, the model was trained to identify patterns of adverse effects and was able to differentiate additive from synergistic effects of drug combinations. Preliminary results showed that the model can accurately predict adverse effects in the clinic, although further validation is required. This approach could improve the risk management of combination therapies, offering a promising tool for the personalization of treatments in oncology.

“Clinicians are challenged by the real-world problem that new combination therapies could lead to unpredictable outcomes.” said Bart Westerman, lead author of the study and associate professor at the Amsterdam Cancer Center. “Our approach can help us understand the relationship between the effects of different drugs in relation to the disease context.”

Why Multimodal AI is the Key to the Future?

Reduce Costs and Accelerate Time to Market

Traditional clinical trials represent a considerable expense for the pharmaceutical industry. On average, the cost of a Phase III clinical trial exceeds $20 million, and can sometimes reach over $100 million, depending on the therapeutic area and the complexity of the study. The use of multimodal AI can significantly reduce these costs by automating complex tasks, such as analyzing clinical data, detecting abnormalities, and identifying the most suitable patients for a specific treatment.

In addition, multimodal AI accelerates decision times by offering real-time analyses that quickly detect trends or potential problems. This reduces the time needed to move from one trial phase to the next, which can shorten the treatment time-to-market by months or even years.

Improving Data Quality and Results Reliability

Data quality is a crucial factor in the success of clinical trials. Traditionally, the data collected in trials is often fragmented, coming from different, unintegrated sources, which can introduce bias and affect the reliability of results. Multimodal AI, by integrating data from multiple sources – such as medical imaging, biometric sensor records, genetic data, and textual clinical notes – offers a more complete and holistic view of each patient.

A Look Into the Future : Multimodal AI and Virtual Clinical Trials

Virtual clinical trials are one of the most eagerly-awaited developments on the horizon. Multimodal AI could be at the heart of this, integrating augmented and virtual reality technologies to simulate test environments or physical responses, and cross-referencing this data with that from physical trials to obtain even more accurate results.

Multimodal AI, combined with augmented reality (AR) and virtual reality (VR) technologies, can transform virtual clinical trials. These technologies can simulate realistic test environments and patients’ physical responses. For example, in controlled VR environments, participants can be exposed to simulations of real-life situations, such as changes in medical conditions, to test the efficacy and safety of treatments in real time. These environments capture multimodal data, such as physiological reactions, body movements, and even facial expressions, which can be analyzed in depth by AI algorithms to provide a more accurate picture of the patient’s response to treatment.