Images and population

We have used several echocardiographic tests in an attempt to represent a wide range of zoom, depth, zone of interest width, focus, gain, and image quality. These tests were performed in 2000 healthy subjects, previously included in the WASE study (World Alliance Societies of Echocardiography normal values study). The objective of the study was to determine normal values for multiple quantifiable echocardiographic parameters. ¹² Subjects came from 19 sites in 15 countries and included a large group of adults of both sexes, and various races, ethnic and age groups. These tests used different marketed echocardiographic systems available and followed the latest recommendations from the American Society of Echocardiography (ASE). ¹³ Images were rated by an expert according to the type of image (2D, pulsed wave Doppler, etc.) and echocardiographic view (4 chambers apical, long axis parasternal, etc.). Regular measurements were performed by a central echocardiography expert (core laboratory) with strict adherence to the latest recommendations. ¹⁴ For our study, we used images from 1800 subjects to develop and train ML algorithms (training group), while the remaining 200 were used to evaluate these algorithms (testing group).

Study design

The study was divided into 3 protocols. Protocol 1 was designed to develop ML algorithms and evaluate accuracy for automated identification of the type of image and echocardiographic view, similarly to the approach used in recent studies ^{9), (10), (11}, as well as to classify these images in stacks according to certain anatomical structures or reference “physiological events” (for example, mitral valve, diastolic function), which would eventually guide image review and interpretation. Protocol 2 was designed to develop and evaluate ML accuracy for automatic measurement of conventional echocardiographic parameters, similar to those used by Zhang et al. ¹⁰ Finally, Protocol 3 was designed to determine the feasibility and effectiveness resulting from the combination and integration of these protocols into echo reading, assessing efficiency (time saving) and reproducibility (measurement variability) parameters, and comparing these results to current interpretation and measurement methods.

The evaluation of Protocol 1 included developing and training a convolutional neural network (CNN) using measurements provided by an independent reader (TM) unfamiliar with development of the ML algorithm. Protocol 2 was evaluated by comparing measurements provided by the ML algorithm to those supplied by a second reader (KA), coming from a site different from that of the first reader. Finally, Protocol 3 compared the performance of 2 readers (KA and KK) who did not take part in the ML algorithm development and training phase.

Development of ML software, as well as the study design, were the result of the collaboration between the investigators involved (MedStar and University of Chicago) and TOMTEC Imaging Systems (Unterschleissheim, Germany). Principal investigators (RML, FMA) had unrestricted data access and analyzed data independently from TOMTEC, assuming full responsibility for the findings presented here.

Convolutional Neural Network algorithm

Training to detect every parameter of interest was conducted through a cascade strategy including 3 subdetection stages. This initial setup was supplemented by an image pre-processing mechanism in order to standardize any resulting information. This included detection of the ultrasound bundle area, deletion of any additional information inserted in echocardiographic images, and intensity and color adjustment for DICOM files.

Initially, for an approximate estimation of the position, size, and orientation in space of the relevant structure, every CNN subdetector was regressively fed with a subsample of images having a fixed resolution of less than 20% the number of pixels in the original image. In a second stage, every detector received the area and structure of interest from each image in order to reduce any changes in terms of scale and position. Image resolution for this second stage was standardized to average resolution (mm/pixel) for all images to make up for any potential change in resolution of individual images. In this case, relevant structures were identified using regressive CNN, as in the first stage of the process. Finally, for the third stage, we used a more refined set of images with improved localization of relevant structures (as compared to the previous stage). As before, regressive CNN made this possible.

Training in every subdetection stage was performed using a subset of measurements such as the ground truth. During training, every detector could analyze information input based on these measurements assumed as the ground truth, together with 32 augmented versions of such input. The increase was possible thanks to mild rotations and scale variations in the area of interest, leading to effective identification of a large number of variants and structures.

Protocol 1: automated image identification and stacking

The CNN was trained to identify 18 standard echocardiographic views: 6 corresponding to two-dimensional echocardiography (2D), and 12 corresponding to tissue Doppler imaging (TDI), pulsed wave Doppler (PW), and continuous wave Doppler (CW). 2D images included apical views in 2, 3 and 4 chambers (A2C, A3C and A4C, respectively), as well as long and short axis parasternal views (PLAX and PSAX, respectively), and subcostal view (SC). TDI included septal and lateral sampling from the mitral annulus and lateral portion of the tricuspid annulus. PW images were collected across the left ventricular outflow tract (LVOT) on apical view, the mitral inflow tract on four-chamber view, and the right ventricular outflow tract (RVOT) on PSAX view. CW showed aortic flow evaluation on apical view, as well as mitral regurgitation flow on four-chamber apical view, tricuspid regurgitation flow on four-chamber apical view and PSAX, and pulmonary flow on PSAX view. All these images were used as CNN input to create our library with different types of images and echocardiographic views.

Then, the CNN was trained to identify views and types of images in an independent set of test images for subsequent allocation or classification of each of these images in 8 stacks based on the following parameters or structures: left ventricular (LV) systolic function and size, LV diastolic function, right ventricular (RV) and right atrial (RA) function and size, valves (mitral, aortic, tricuspid and pulmonary), and pericardium. Table 1 shows a list of the images required to complete each stack. Finally, the results of automatic image identification and classification were compared to the “ground truth” predetermined by the expert reader.

Protocol 2: automatic analysis accuracy and reproducibility

After every image was automatically classified according to type and view, and after selecting the cardiac cycle phase for measurements, a ML monitored algorithm was trained to measure 16 standard parameters regularly quantified in clinical practice. These parameters included interventricular septum diastolic thickness, diastolic and systolic LV internal diameter, LV posterior wall diastolic thickness, LVOT midsystolic diameter, LVOT velocity time integral (VTI), left atrial (LA) maximum volume on A2C and A4C, LV end-diastolic and end-systolic volumes , mitral valve inflow E wave and A wave , and mitral lateral and septal annulus tissue velocity. All chamber volumes were measured using the biplane Simpson method.

The accuracy of these automatic measurements was compared to that of manual measurements collected by an expert reader from the test group of 200 subjects. Also, in a subset of 30 randomly selected tests, differences between automatic ML measurements and those provided by a reference expert reader were compared to inter- and intraobserver human variability by repeated blinded measurements. To assess interobserver variability, measurements were independently performed by two echocardiography expert cardiologists using conventional interpretation and measurement methods. Finally, to determine intraobserver variability, one of these experts, again fully blinded to previous measurements, re-read the same echocardiographic tests at least two weeks after the first analysis.

Protocol 3: efficiency and reproducibility of the ML assisted approach

In this protocol, both experts independently repeated interpretation of the 30 tests at least 2 weeks later, this time, using ML assistance. In this case, experts could use any stacks and measurements that were automatically established by algorithms, making any necessary adjustments to improve accurate identification and measurement of relevant structures or events. For both sessions, the time required to complete analysis of the echocardiographic test depended on the workstation being used, and intra and interobserver variability was evaluated for all previously described parameters. Finally, times of analysis with and without ML assistance were compared.

Statistical analysis

The accuracy of the automated method for type of image identification and echocardiographic view, as well as comparison versus experts, was quantified as the percentage of correct classifications based on the total number of analyzed image sequences. For every parameter, consistency between ML measurements and reference values collected by experts was evaluated using linear regression methods, and bias and limits of agreement tests using the Bland-Altman method. In order to compare different parameters, biases were expressed as the percentage of the average value of each parameter (% bias). Intra and interobserver variability for every parameter was quantified using variation coefficients, defined as the absolute difference among repeated measurements expressed as mean percentages. The statistical significance of variations in the time of analysis with and without ML assistance was determined by the Student’s t test using two-tailed paired measurements. Pvalues < 0.05 were considered significant.

Ethical considerations

The study was approved by the ethics committees of each institution and patient consent was obtained according to local regulations.

Protocol 1

The CNN performed automatic classification of echocardiographic views with an average of 0.87 seconds per test. The CNN was able to identify types of images and echocardiographic views with a 90% accuracy for 2D images and a 94% accuracy for Doppler images. Figure 1 shows data corresponding to agreement between ML and the expert’s reader “ground truth” classification for each echocardiographic view. While agreement was excellent in most cases, the largest number of errors occurred when classifying non-standard sub-optimal views, wrongly identified as A3C (and noted as “Other”). The CNN was able to separate and group images in stacks with a 91% accuracy. When counting the number of views required to complete every stack, ignoring those that were considered unnecessary, the final composition of each image stack showed a 99% agreement between ML and expert readers (Table 1).

Fig 1. Identifying the type and view of echocardiographic images: agreement between automated ML measurements and manual expert measurements. Each cell represents the percentage of agreement for every type of image and view being evaluated. The cells have different colors depending on the percentage of agreement: red (0%), orange (0.1%-50%), yellow (50.1%-90%) and green (90.1%-100%). Abbreviations: A2C, A3C and A4C = 2, 3 and 4 chamber apical, respectively; PLAX, PSAX, SC = long axis parasternal, short axis parasternal, and subcostal, respectively; MVann (l), MVann (s), TVann (l) = lateral mitral annulus, septal mitral annulus, and lateral tricuspid annulus, respectively; Api LVOT = apical view of the left ventricular outflow tract; Api AV = apical view of the aortic valve; MVinfl, TV infl, PVinfl = mitral, tricuspid and pulmonary valves inflow tracts, respectively; CW = continuous wave Doppler; PW = pulsed wave Doppler 

Protocol 2

Automatic ML measurements showed excellent agreement versus reference values collected by experts, as shown by high correlation rates, small biases, and narrow limits of agreement for most parameters under analysis (Table 2). The largest relative biases occurred for left atrial volumes both in A4C and A2C (18% and 25% of reference values), followed by left ventricular volumes (6.5% and 7.8% of reference values). For the remaining parameters being assessed, relative biases were very small, with values lower than 3%. The analysis of the subset of 30 tests showed that differences between ML and expert measurements were similar to differences between manual measurements by two human technicians (Table 3).

Protocol 3

Conventional interpretation of a complete echocardiographic study required an average of 11 minutes and 33 seconds (9:29 for reader #1 and 13:36 for reader #2). On the other hand, when using ML assistance through stacking and automated measurements, the average time needed to complete evaluation of a test was 6 minutes and 48 seconds (4:50 for reader #1 and 8:45 for reader #2). This means that using ML may reduce an average 40% the time required to interpret an echocardiographic test using current conventional methods (49% for reader #1 and 36% for reader #2). Please note that, as compared to the manual method, ML reduced inter-observer variability in 15 of 16 measurements (Table 3), except for LVOT diameter, which had a highly reproducible measurement using both methods.