Abstract (ASEBIR 2023): Oocyte Image Analysis Artificial Intelligence Tool, MAGENTA, Validated On Both EmbryoScope And Geri Time-Lapse Systems

INTRODUCTION:

MAGENTA is a non-invasive oocyte image analysis artificial intelligence (AI) tool that assesses 2-dimensional images of mature (metaphase II, MII) oocytes and provides a score from 0-10 that positively correlates with subsequent blastocyst development and quality. For workflow simplicity, images of MII oocytes can be obtained directly following ICSI from time-lapse (TL) systems. There are multiple TL systems in use in the IVF lab, most notably EmbryoScope (Vitrolife) and GERI (Genea Biomedx). It is important to validate MAGENTA‘s performance on not only various microscope cameras, but also different TL systems to ensure generalizability prior to clinical use, as well as provide greater accessibility for individual labs. Area-under-the-curve (AUC) is an important metric that can be used to assess AI model performance as it evaluates the classification (blastocyst positive or negative) by the tool at various thresholds and accounts for the sensitivity (true positive rate) and specificity (true negative rate) of the model.

GOALS:

To validate the performance of a non-invasive oocyte image analysis AI tool (MAGENTA) on images of MII oocytes obtained from two different time–lapse systems immediately post-ICSI, prior to clinical implementation.

MATERIALS AND METHODS:

American lab locations between the years 2014-2021 and from a GERI TL system at a single lab location in Spain between the years 2016-2020.  

Baseline performance of MAGENTA is calculated to have an AUC of 0.64 at a 95% CI [0.62, 0.65], a sensitivity of 0.65, and a specificity of 0.55. Performance of the AI tool in predicting blastocyst development on images of MII oocytes obtained from the TL systems (EmbryoScope and GERI) was assessed by calculating the AUCs. 95% confidence intervals of the AUCs were compared by the DeLong’s test for two receiver-operating-curves (ROC). AUC comparisons were conducted for performance on 6133 images of MII oocytes from EmbryoScope and 1194 from GERI, all previously unseen to the AI model.  

RESULTS:

The AUC was 0.65 at a 95% CI [0.64, 0.66] for EmbryoScope, with a sensitivity of 0.71 and a specificity of 0.51. Similarly, for GERI the AUC of the model was 0.65 at a 95% CI [0.62, 0.68], with a sensitivity of 0.52 and a specificity of 0.67.  DeLong’s test for two ROC curves displayed no significant difference between the performance of MAGENTA on the MII oocyte images obtained from EmbryoScope and those obtained from GERI (p = 0.91). Additionally, this indicates no significant difference on model performance between patient populations from North America or Spain. 

CONCLUSIONS:

Validation studies ensuring similar performance of AI image analysis tools on different image and data sources are crucial to ensure clinical applicability prior to integration and patient use. MAGENTA displays similar performance in predicting blastocyst development from images of denuded MII oocytes post–ICSI obtained from different TL systems, displaying consistency in assessment of oocyte quality for greater implementation. The results highlight the diversity and generalizability of the MAGENTA AI model across both various image sources (Embryoscope and GERI) and patient demographics (North America and Europe).