Crowdsourcing and Automated Retinal Image Analysis for Diabetic Retinopathy

Introduction Screening for diabetic retinopathy (DR) is a straightforward and cost-effective method to detect disease. Two key elements of successful screening for DR include (1) capturing images of the retinal fundus and (2) accurate grading of these images. This screening paradigm adapts well to telemedicine, where the image grader may not be in close geographic proximity to the patient; however, the patient is still assured of receiving expert care. A natural progression from telemedicine is the use of innovative tools such as artificial intelligence (AI) and crowdsourcing to speed up the process of grading images as well as to further reduce costs of screening. Crowdsourcing is a novel method for processing information and generating data that is gaining popularity for its cost-effectiveness and fast results. AI, where a machine mimics cognitive functions such as, learning and problem-solving [1], has also been employed for rapid and cost-effective processing of information, such as automated grading of images. Two general approaches to using AI and crowdsourcing to grade images for DR may be taken. One is to use them as a first pass to screen out completely normal images, leaving the ungradable or borderline and abnormal images for review by a human expert. The second, is to replace experts altogether. A hybrid approach uses crowdsourced annotations to help develop training sets for deep learning algorithms. This manuscript reviews the current literature surrounding AI and crowdsourcing for retinal image analysis, with specific focus on how these tools may be applied to clinical care and research in the field of DR. Distributed Human Intelligence Crowdsourcing, defined by Daren Brabham as “an online, distributed, problem-solving, and production model that uses the collective intelligence of networked communities for specific purposes” [2]. Distributed human intelligence is a one form of crowdsourcing, where individuals complete only small, manageable tasks so that collectively, the larger goal is achieved [3]. Amazon Mechanical Turk (AMT) is a popular distributed human intelligence platform, where a user (requestor) uploads discrete tasks (human intelligence tasks, HITs) that a worker (Turker) can complete in a matter of seconds for a small reward (such as US$0.10). Turkers work anonymously; however, they do receive feedback on their work. AMT is a dynamic and complex ecosystem, where Turkers and requestors can rate each other; requestors can mandate that only highly rated Turkers complete their uploaded HITs and highly rated Turkers may avoid HITs from newer or lower-rated requestors. The Amazon brand and AMT’s relatively easy-to-use platform make it a popular option for those wanting to crowdsource; however, AMT is not the only platform available for distributed human intelligence; others, including Zooniverse, Volunteer Science, and Kaggle are available. Zooniverse and Volunteer Science are two platforms aimed towards “Citizen Science,” where crowdsourcing is used to further scientific knowledge. Kaggle was designed as a platform for crowdsourcing predictive modeling and analytics and has been used to generate predictive models for seizures [4]. Kaggle also hosted a competition in 2015, to develop an automated detection system for DR, with the first prize winner receiving $50,000. Apart from these publically available platforms, requestors can also create their own crowdsourcing platforms to meet their needs. Crowdsourcing for Image Analysis Table 1 summarizes three key studies where AMT and customized crowdsourcing platforms have been utilized to grade images for DR. In a proof-of-concept study, workers on AMT performed 190 gradings of 19 retinal images (10 repetitions per image) of patients with diabetes and correctly classified the images as normal or abnormal 81.3% of the time [5••]. The average time to grade each image was 25 s (including time to review training images), for a total cost of $1.10 per image. Subsequently, workers were asked to grade images in three or four categories of DR severity. This reduced the percent correctly graded to 64.4% for a three-category grading, and 50.9% for a four-category grading. However, in both trials, workers were able to maintain 100% sensitivity for abnormal images. Table 1 Studies Evaluating Crowdsourcing to Grade Images for Retinal Abnormalities* Study Platform Grading scheme Worker experience Worker compensation Correct (%) Sensitivity (%) Specificity (%) AUROC (95% CI) Time Brady (2014) [5••] AMT Abnormal vs normal ≥ 100 prior tasks with 97% approval $0.10 per image 81.3 93.6 67.8 20 min 3 category of DR severity 64.4 96.3 66.7 3 h 4 category of DR severity 50.9 96.3 66.7 2 h Mitry (2013) [6] AMT Normal vs severely abnormal None $0.03 per image 96 98 74 0.87 (0.85–0.89) < 1 day Normal vs severely abnormal None $0.05 per image 92 99 68 0.83 (0.81–0.85) < 1 day Normal vs severely abnormal ≥ 500 prior tasks with 90% approval $0.03 per image 90 98 85 0.92 (0.90–0.93) 1–2 days Normal vs severely abnormal ≥ 5000 prior tasks with 99% approval $0.03 per image 99 99 64 0.82 (0.80–0.84) 15 days Mitry (2016) [7••] Custom# Abnormal vs normal None, no training $0.05 per image 81.4 82.9 77.8 0.93 (0.90–0.96) 48 h Abnormal vs normal None, mandatory training $0.05 per image 80.6 91.3 54.2 0.94 (0.91–0.97) 11 days Abnormal vs normal Experienced workers^, no training $0.05 per image 86.3 87.5 83.3 0.89 (0.87–0.91) 13 days *Brady (2014) [5••] had workers grade images specifically for DR, while Mitry (2013 and 2016) [6, 7••] used images with a panoply of retinal conditions including DR. #AMT was used to recruit workers, but http://www.retinalannotations.com/ was used to complete the image grading task ^In this trial, “experienced” workers had a “masters qualification”, which is determined and assigned by AMT for a small group of elite workers AUROC area under receiver operator curve; AMT Amazon Mechanical Turk; 95% CI 95% confidence interval; DR diabetic retinopathy Mitry [6] used AMT to perform grading of images for a panoply of different retinal conditions, including signs associated with DR. Workers had to distinguish between normal and abnormal fundus images, and different compensation ($0.03 vs $0.05 payment per image) and different requirements of Turkers experience and reputation were trialed. The Turkers worked quickly and cheaply; 2000 gradings of 100 images (20 repetitions per image) were performed in under 24 h for $60. The authors reported a sensitivity of 96% for normal vs severely abnormal in all trials and the area under receiver operating characteristic curve (AUROC) ranged from 0.68–0.94 depending on the grading categories, worker experience and reimbursement. Interestingly, increased payment to workers did not improve grading accuracy. In a more recent study by Mitry [7••], workers had to classify images as normal or abnormal, and annotate by simple rectangle localization of pathology. The workers had to classify and annotate 84 images with a range of retinal pathology from vascular occlusion to DR to age-related maculopathy. In contrast to their prior studies, the authors required each worker to grade all 84 images; however, they gave each worker a unique code to log in to the site and allowed them to complete the task over multiple sessions. The workers also had a training area with a practice set of images. After each image was annotated, the workers received live feedback; the consensus annotation performed by other users was displayed to them. The authors ran several trials of this method varying the worker experience, and one trial where the completion of training images was mandatory. The less experienced workers completed their trial most quickly in 48 h, compared with the 13 days taken by the more experienced workers. In the mandatory training experiment, less experienced workers took 11 days to complete the task. The AUROC for classification of healthy vs abnormal images ranged from 0.89 (95% confidence interval, CI 0.87–0.91) for more experienced workers to 0.93 (95% CI 0.90–0.96) for the less experienced, and 0.94 (95% CI 0.91–0.97) for the less experienced with mandatory training. The Dice coefficient was reported to express the agreement between the workers and an expert annotation of pathology; this measure takes into account the size and location of annotated lesions, and ranges from 0 (no agreement) to 1 (perfect congruence). Values greater than 0.6 are generally considered good agreement. The Dice coefficient for the less experienced workers with and without training was 0.59 for both, and for the more experienced workers, the coefficient was 0.57. Harnessing the “Wisdom of the Crowd” Crowdsourcing is iterative by nature. At one level, there are the repetitions of different trials, where the same batch of images is posted under different conditions aiming to increase accuracy of grading (such as with and without training, varied levels of worker experience, and varied amounts of compensation). The literature suggests that worker accuracy is generally related to the workers’ understanding of the task prior to completing the task rather than to the amount of experience grading images [7••, 8, 9]. Payment for the task does not seem to be directly related to accuracy [6], although it may relate to participation and speed of completing the task [8]. Much crowdsourcing relies on multiple graders to try to obtain the group’s wisdom. In Mitry’s paper [6], the number of grading repetitions of each image needed to achieve maximal AUROC varied from 11 to 16, whereas in Brady’s paper [5••], 7 to 10 were needed to achieve maximal AUROC. The number of repetitions of grading for each image impacts the cost of the crowdsourced project as well as the accuracy achieved; AMT takes a greater commission for 10+ repetitions per image. From a statistical point of view, having multiple grades of the same image present an interesting exercise in deciding how best to choose the consensus grade. In many reports, majority votes, in which the grade rendered by 50% or more of graders is elevated to the consensus grade. Other methods that consider the variances between individual graders as well as the images have also been tried. Some studies, where crowdsourcing was used for applications outside of image grading, have evaluated alternative methods to harnessing the wisdom of the crowd. De Alfaro [10] compared the accuracy of using the average worker grading for a single item and of two different algorithms to arrive at a consensus grade. The authors found that the algorithm nicknamed “VariancePropagation” performed best. In brief, this method assumes all workers initially have the same accuracy, but then iteratively refines estimates of item quality and user accuracy, which are then used to weight the grading of an individual item by an individual worker. Crowdsourcing retinal image analysis does have inherent limitations. Almost by definition, control over who is performing the image analysis has been ceded to the abstract entity of “the crowd,” and efforts to exert choose or credential users of an image grading platform run counter to the spirit of crowdsourcing. Operationally, this means that any crowdsourcing implementation requires meticulous quality control to ensure the results seen in pilot-testing are maintained. Importantly, we are not aware of any large study comparing crowdsourced retinal grading with conventional clinical grading/reading centers or any automated artificial intelligence grading program. As such, we currently are unable to compare crowdsourcing with clinical telemedicine grading or automated grading systems. Moreover, while crowdsourcing appears to represent a low-cost method for retinal image interpretation, there may be opportunities for low-cost expert grading of images through a more traditional outsourcing paradigm, such as has been done with overseas nighthawk teleradiology. Artificial Intelligence Research in automated retinal image analysis (ARIA) and computer-aided detection/diagnosis (CAD) began around 40 years ago [11, 12], but these techniques continue to be of interest as another method to reduce costs associated with screening for DR. ARIAs have been used for both classification of image quality as well as identification and classification of image pathology in DR, although this report will focus on the latter. Many algorithms exist for ARIAs and varying sensitivities and specificities have been reported. Trucco [13] reviewed the literature on use of ARIAs for DR grading over the last 20 years and reported specificities ranging from 67 to > 95%, and sensitivities from 45 to 100%. More recently, Walton [14] reported the sensitivity of an algorithm known as IRIS in detecting sight-threatening diabetic eye disease compared with expert grading was 66.4% and specificity of 72.8%. In their study population, where the prevalence of any DR among diabetics was 15.8%, the positive predictive value of the IRIS algorithm was 10.8% and the negative predictive value was 97.8%. ARIAs for DR have advanced so far that there are now at least five commercially available [15] that are used in countries such as Scotland and Portugal. However, none is currently Food and Drug Administration approved, and only one has 501(k) pre-market notification. Tufail [16] compared expert grading of 20,258 images to 3 of these commercially available ARIAs: iGradingM (Medalytix LLC; Digital Healthcare, Scotland), Retmarker (Retmarker Ltd.; Portugual), and EyeArt (Eyenuk Inc.; Woodland Hills, CA). The authors reported sensitivities of 93.8 and 85.0% for referable DR for EyeART and Retmarker, respectively. The iGradingM ARIA classified all images as either ungradable or abnormal (Table 2). The authors also performed a cost analysis and reported the ARIAs saved more than USD $100,000 when used as a replacement for manual grading. They modeled the cost-effectiveness of using ARIAs as a first pass prior to human grading, interestingly, since these two strategies (ARIAs as a replacement for human grading and ARIAs as a first pass before human grading) performed nearly identically, using ARIAs as replacement was significantly more cost-effective. Table 2 Studies Evaluating Automated Retinal Image Analysis to Grade Images for Diabetic Retinopathy Product Study Grading scheme Sensitivity (95% CI) Specificity (95% CI) AUROC (95% CI) Retmarker Oliveira (2011) [20] Referable DR 96.1% (94.4%–97.9%) 51.7% (50.3%–53.1%) 0.849 Tufail (2017) [16] Any DR 73.0% (72.0%–74.0%) 20% Referable DR 85.0% (83.6%–86.2%) PDR 97.9% (94.9%–99.1%) iGradingM Philip (2007) [21] Any DR 86.5% (85.1%–87.8%) 67.4% (66.0%–68.8%) Tufail (2017) [16] All trials* 100% 0% EyeArt Bhaskaranand (2016) [22] Referable DR 90% (88.0%–92.0%) 63.2% (61.7%–64.6%) 0.879 (0.865–0.893) Tufail (2017) [16] Any DR 94.7% (94.2%–95.2%) 52.3% Referable DR 93.8% (92.9%–94.6%) PDR 99.6% (97.0%–99.9%) Google Gulshan (2016) [18••] All-cause referable DR 96.7% (95.7%–97.5%) 84.0% (83.1%–85.0%) 0.974 (0.971–0.978) *The iGradingM algorithm results all images as either “disease” or “ungradable” AUROC area under receiver operator curve; AMT Amazon Mechanical Turk; 95% CI 95% confidence interval; DR diabetic retinopathy; PDR proliferative diabetic retinopathy ARIAs do have systematic limitations. A study by MacGillvray [17] evaluated the suitability of a large UK population-based dataset of fundus photos and found that photos from older individuals were significantly less likely to be able to be processed and analyzed with automated programs. In their study, 84% of individuals in the large dataset had at least one image suitable for processing by ARIA. Trucco [13] reported another limitation of ARIAs as a lack of large-scale studies and evaluation of ARIAs in datasets with unique idiosyncrasies. Large, publicly available databases and competitions such as the Retinopathy Online Challenge and Kaggle DR competition have helped to allow for better comparisons of ARIAs by applying different algorithms to the same dataset. However, these large available datasets are also limited by a lack of coordination in how they are structured and annotated, and what “ground truth” information is provided. Deep learning is another type of AI gaining more attention. Deep learning allows an algorithm to “program itself by learning from a large set of examples that demonstrate the desired behavior, removing the need to specify rules explicitly” [18••]. The Google DeepMind project used deep learning on a dataset of 128,175 retinal images to develop an algorithm, which was then compared to expert grading for referable DR in two other large datasets [18••]. The AUROC for the deep learning algorithm was 0.99 for both datasets, the sensitivity ranged from 87.0–97.5%, and specificity ranged from 93.4–98.5%. An interesting study by Abramoff [19] compared a deep-learning enhanced algorithm and the same algorithm without deep learning components to expert grading, and reported a similar sensitivity of 96.8% for both algorithms, but an improved specificity with deep learning: the original algorithm had a specificity of 95% CI of 55.7–63.0%, while the deep-learning enhance algorithm had a 95% CI of 84.2–89.4%. The AUROC of the deep-learning enhance algorithm was 0.98. Conclusion Rapid and accurate grading of images is a key for successful DR screening, thus it is a logical progression for technology to be applied to further streamline and reduce costs of grading images. Crowdsourcing has been used grade fundus images for DR with high sensitivity for normal vs abnormal. As would be expected, the accuracy of grading seems to decrease when categories for level of severity of DR are added. There is some evidence to suggest that grading can be improved with the addition of brief training prior to completion of the task. The field of crowdsourcing for DR is still evolving with different techniques being trialed to improve the information generated, such as localizing pathology with annotation and using real-time feedback and interaction among workers. ARIAs are capable of grading images for DR with high sensitivity and specificity. So much so that there are now ARIAs in use in screening programs around the globe. Of note, one study suggested that ARIAs perform similarly whether used as a first pass to reduce burden on human grading or as a replacement to human grading, with the latter being more cost-effective in saving over USD $100,000 to screen approximately 20,000 individuals. ARIAs have also evolved with the new literature reporting improved sensitivities and specificities of ARIAs with deep learning enhancement. Moving forward, there may be opportunities for convergence of crowdsourcing and deep learning methods. To “learn” to identify disease on an image, deep learning algorithms require staggering amounts of gold standard graded images, with almost 130,000 images used in the training set by Gulshan et al. [18••]. As professional, expert grading would be time- and cost-prohibitive in such quantities, the authors, determined the gold standard grade for each image via crowdsourcing using a majority vote strategy. Although deep learning algorithms appear to perform better than crowdsourcing as a replacement for expert grading, the cost of deep learning algorithms may preclude their use in low resource settings. Additionally, any clinically implemented system will need to demonstrate real-world validation and efficacy as all automated systems are at risk of being accurate only within their training and test image sets if these do not reflect the diversity of the populations in which they are to be used. Overall, as the prevalence of DR continues to increase, utilization of time and cost saving methods such as crowdsourcing and ARIAs may be a key to decreasing the burden of this sight-threatening disease. Compliance with Ethical Standards Conflict of Interest Lucy I. Mudie, Xueyang Wang, David S. Friedman, and Christopher J. Brady declare that they have no conflict of interest. Human and Animal Rights and Informed Consent This manuscript does not include any studies with human or animal subjects performed by any of the authors. The Johns Hopkins University School of Medicine Institutional Review Board (IRB) has deemed the crowdsourcing work of the authors IRB-exempt as non-human subjects research. Disclaimer Disclaimer Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the Johns Hopkins ICTR, NCATS or NIH.

Medicine Matters Diabetes

Crowdsourcing and Automated Retinal Image Analysis for Diabetic Retinopathy

Abstract

Purpose of Review

Recent Findings

Summary

New additions to the Adis Journal Club