Designing an Intelligent Lesion Detection System Using Deep Architecture Neural Networks in the Lower Limb X-Ray Images
-
Sepideh Amiri
,
Mina Akbarabadi
,
Shahnaz Rimaz
,
Fatemeh Abdolali
,
Reza Ahadi
,
Mohsen Afshani
,
Zahra alaei askarabad
,
Tahereh Kowsarirad
,
Sohrab Sakinehpour
,
Nazila Eyvazzadeh
,
Susan Cheraghi
Abstract
Purpose: Diagnosis of musculoskeletal abnormalities is essential due to more than 1.7 billion people worldwide being affected by musculoskeletal disorders. In this study, we focus on diagnosing musculoskeletal abnormalities in the lower extremities within X-ray images by deep architecture neural networks.
Methods: Our dataset contains 61,098 musculoskeletal radiographic images, which includes 42658 normal images and 18440 abnormal images. Each image belongs to a single type of lower extremity radiography, including the toe, foot, ankle, leg, knee, femur, and hip joint. We proposed a new deep neural network architecture with two different scenarios that perform lower extremity lesion diagnosis functions with high accuracy. The core of the proposed method is a deep learning framework based on the Mask R-CNN and CNN. The model with the best results utilized the Mask R-CNN algorithm to generate the bounding box, followed by the CNN algorithm to detect the class based on that.
Results: The proposed model can detect different types of lower limb lesions by an AUC-ROC of 0.925, with an operating point of 0.859 sensitivity and a specificity of 0.893.
Conclusions: By comparing the different results, it can be concluded that the consecutive implementation of Mask R-CNN and CNN function better than Mask R-CNN and CNN separately.
1- A. D. Woolf and B. Pfleger, "Burden of major musculoskeletal conditions." (in eng), Bull World Health Organ, Vol. 81 (No. 9), pp. 646-56, (2003).
2- E. Yelin, S. Weinstein, and T. King, "The burden of musculoskeletal diseases in the United States." (in eng), Semin Arthritis Rheum, Vol. 46 (No. 3), pp. 259-60, Dec (2016).
3- Pranav Rajpurkar et al., "Mura: Large dataset for abnormality detection in musculoskeletal radiographs." arXiv preprint arXiv:1712.06957, (2017).
4- P. Rajpurkar et al., "Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists." (in eng), PLoS Med, Vol. 15 (No. 11), p. e1002686, Nov (2018).
5- Yee Liang Thian, Yiting Li, Pooja Jagmohan, David Sia, Vincent Ern Yao Chan, and Robby T Tan, "Convolutional neural networks for automated fracture detection and localization on wrist radiographs." Radiology: Artificial Intelligence, Vol. 1 (No. 1), p. e180001, (2019).
6- Macedo Firmino, Giovani Angelo, Higor Morais, Marcel R Dantas, and Ricardo Valentim, "Computer-aided detection (CADe) and diagnosis (CADx) system for lung cancer with likelihood of malignancy." Biomedical engineering online, Vol. 15 (No. 1), pp. 1-17, (2016).
7- Junji Shiraishi, Qiang Li, Daniel Appelbaum, and Kunio Doi, "Computer-aided diagnosis and artificial intelligence in clinical imaging." in Seminars in nuclear medicine, (2011), Vol. 41 (No. 6): Elsevier, pp. 449-62.
8- Kai-Lung Hua, Che-Hao Hsu, Shintami Chusnul Hidayati, Wen-Huang Cheng, and Yu-Jen Chen, "Computer-aided classification of lung nodules on computed tomography images via deep learning technique." OncoTargets and therapy, Vol. 8(2015).
9- Petros-Pavlos Ypsilantis et al., "Predicting response to neoadjuvant chemotherapy with PET imaging using convolutional neural networks." PloS one, Vol. 10 (No. 9), p. e0137036, (2015).
10- Ruben Pauwels et al., "Artificial intelligence for detection of periapical lesions on intraoral radiographs: comparison between convolutional neural networks and human observers." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, Vol. 131 (No. 5), pp. 610-16, (2021).
11- Gloria Gonella, Marco Paracchini, Elisabetta Binaghi, and Marco Marcon, "Breast Lesion Detection from Mammograms Using Deep Convolutional Neural Networks." in Proceedings of the 2020 European Symposium on Software Engineering, (2020), pp. 120-24.
12- Benedetta Savelli, Alessandro Bria, Mario Molinara, Claudio Marrocco, and Francesco Tortorella, "A multi-context CNN ensemble for small lesion detection." Artificial Intelligence in Medicine, Vol. 103p. 101749, (2020).
13- Joseph Antony, Kevin McGuinness, Noel E O'Connor, and Kieran Moran, "Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks." in 2016 23rd International Conference on Pattern Recognition (ICPR), (2016): IEEE, pp. 1195-200.
14- Lei Bi, Jinman Kim, Ashnil Kumar, and Dagan Feng, "Automatic liver lesion detection using cascaded deep residual networks." arXiv preprint arXiv:1704.02703, (2017).
15- Ruikai Zhang et al., "Automatic detection and classification of colorectal polyps by transferring low-level CNN features from nonmedical domain." IEEE journal of biomedical and health informatics, Vol. 21 (No. 1), pp. 41-47, (2016).
16- Varun Gulshan et al., "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs." Jama, Vol. 316 (No. 22), pp. 2402-10, (2016).
17- Hayit Greenspan, Bram Van Ginneken, and Ronald M Summers, "Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique." IEEE transactions on medical imaging, Vol. 35 (No. 5), pp. 1153-59, (2016).
18- Daniel S Kermany et al., "Identifying medical diagnoses and treatable diseases by image-based deep learning." Cell, Vol. 172 (No. 5), pp. 1122-31. e9, (2018).
19- Chaochao Yan, Jiawen Yao, Ruoyu Li, Zheng Xu, and Junzhou Huang, "Weakly supervised deep learning for thoracic disease classification and localization on chest x-rays." in Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics, (2018), pp. 103-10.
20- Yaniv Bar, Idit Diamant, Lior Wolf, Sivan Lieberman, Eli Konen, and Hayit Greenspan, "Chest pathology detection using deep learning with non-medical training." in 2015 IEEE 12th international symposium on biomedical imaging (ISBI), (2015): IEEE, pp. 294-97.
21- Jakub Olczak et al., "Artificial intelligence for analyzing orthopedic trauma radiographs: deep learning algorithms—are they on par with humans for diagnosing fractures?" Acta orthopaedica, Vol. 88 (No. 6), pp. 581-86, (2017).
22- Robert Lindsey et al., "Deep neural network improves fracture detection by clinicians." Proceedings of the National Academy of Sciences, Vol. 115 (No. 45), pp. 11591-96, (2018).
23- Hieu H. Pham, Tung T. Le, Dat Q. Tran, Dat T. Ngo, and Ha Q. Nguyen, "Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels." Neurocomputing, Vol. 437pp. 186-94, 2021/05/21/ (2021).
24- Stephen Waite et al., "Analysis of perceptual expertise in radiology–Current knowledge and a new perspective." Frontiers in human neuroscience, Vol. 13p. 213, (2019).
25- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems, Vol. 25(2012).
26- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, "Mask r-cnn." in Proceedings of the IEEE international conference on computer vision, (2017), pp. 2961-69.
27- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 770-78.
28- Fatemeh Abdolali, Jeevesh Kapur, Jacob L Jaremko, Michelle Noga, Abhilash R Hareendranathan, and Kumaradevan Punithakumar, "Automated thyroid nodule detection from ultrasound imaging using deep convolutional neural networks." Computers in Biology and Medicine, Vol. 122p. 103871, (2020).
29- Andrew G Howard et al., "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861, (2017).
30- Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation." in International Conference on Medical image computing and computer-assisted intervention, (2015): Springer, pp. 234-41.
31- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, "Rethinking the inception architecture for computer vision." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2818-26.
32- César Ferri, José Hernández-Orallo, and R Modroiu, "An experimental comparison of performance measures for classification." Pattern recognition letters, Vol. 30 (No. 1), pp. 27-38, (2009).
33- Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz, "Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation." in Australasian joint conference on artificial intelligence, (2006): Springer, pp. 1015-21.
34- Jacob Cohen, "A coefficient of agreement for nominal scales." Educational and psychological measurement, Vol. 20 (No. 1), pp. 37-46, (1960).
35- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large-scale hierarchical image database." in 2009 IEEE conference on computer vision and pattern recognition, (2009): Ieee, pp. 248-55.
36- Jianning Chi, Ekta Walia, Paul Babyn, Jimmy Wang, Gary Groot, and Mark Eramian, "Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network." Journal of digital imaging, Vol. 30 (No. 4), pp. 477-86, (2017).
37- Tianjiao Liu, Shuaining Xie, Jing Yu, Lijuan Niu, and Weidong Sun, "Classification of thyroid nodules in ultrasound images using deep model based transfer learning and hybrid features." in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2017): IEEE, pp. 919-23.
38- Sarang Narkhede, "Understanding auc-roc curve." Towards Data Science, Vol. 26 (No. 1), pp. 220-27, (2018).
39- Maya Varma et al., "Automated abnormality detection in lower extremity radiographs using deep learning." Nature Machine Intelligence, Vol. 1 (No. 12), pp. 578-83, (2019).
40- Nitesh Pradhan, Vijaypal Singh Dhaka, and Himanshu Chaudhary, "Classification of human bones using deep convolutional neural network." in IOP conference series: materials science and engineering, (2019), Vol. 594 (No. 1): IOP Publishing, p. 012024.
41- Yunxue Shao and Xin Wang, "A two stage method for abnormality diagnosis of musculoskeletal radiographs." in International Conference on Pattern Recognition and Artificial Intelligence, (2020): Springer, pp. 610-21.
42- Janick Rohrbach, Tobias Reinhard, Beate Sick, and Oliver Dürr, "Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks." Computers & Electrical Engineering, Vol. 78pp. 472-81, (2019).
43- Jesse C Rayan, Nakul Reddy, J Herman Kan, Wei Zhang, and Ananth Annapragada, "Binomial classification of pediatric elbow fractures using a deep learning multiview approach emulating radiologist decision making." Radiology: Artificial Intelligence, Vol. 1 (No. 1), p. e180015, (2019).
44- Gabriel Chartrand et al., "Deep learning: a primer for radiologists." Radiographics, Vol. 37 (No. 7), pp. 2113-31, (2017).
45- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson, "How transferable are features in deep neural networks?" Advances in neural information processing systems, Vol. 27(2014).
46- William Gale, Luke Oakden-Rayner, Gustavo Carneiro, Andrew P Bradley, and Lyle J Palmer, "Detecting hip fractures with radiologist-level performance using deep neural networks." arXiv preprint arXiv:1711.06504, (2017).
47- Elizabeth A Krupinski, Kevin S Berbaum, Robert T Caldwell, Kevin M Schartz, and John Kim, "Long radiology workdays reduce detection and accommodation accuracy." Journal of the American College of Radiology, Vol. 7 (No. 9), pp. 698-704, (2010).
2- E. Yelin, S. Weinstein, and T. King, "The burden of musculoskeletal diseases in the United States." (in eng), Semin Arthritis Rheum, Vol. 46 (No. 3), pp. 259-60, Dec (2016).
3- Pranav Rajpurkar et al., "Mura: Large dataset for abnormality detection in musculoskeletal radiographs." arXiv preprint arXiv:1712.06957, (2017).
4- P. Rajpurkar et al., "Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists." (in eng), PLoS Med, Vol. 15 (No. 11), p. e1002686, Nov (2018).
5- Yee Liang Thian, Yiting Li, Pooja Jagmohan, David Sia, Vincent Ern Yao Chan, and Robby T Tan, "Convolutional neural networks for automated fracture detection and localization on wrist radiographs." Radiology: Artificial Intelligence, Vol. 1 (No. 1), p. e180001, (2019).
6- Macedo Firmino, Giovani Angelo, Higor Morais, Marcel R Dantas, and Ricardo Valentim, "Computer-aided detection (CADe) and diagnosis (CADx) system for lung cancer with likelihood of malignancy." Biomedical engineering online, Vol. 15 (No. 1), pp. 1-17, (2016).
7- Junji Shiraishi, Qiang Li, Daniel Appelbaum, and Kunio Doi, "Computer-aided diagnosis and artificial intelligence in clinical imaging." in Seminars in nuclear medicine, (2011), Vol. 41 (No. 6): Elsevier, pp. 449-62.
8- Kai-Lung Hua, Che-Hao Hsu, Shintami Chusnul Hidayati, Wen-Huang Cheng, and Yu-Jen Chen, "Computer-aided classification of lung nodules on computed tomography images via deep learning technique." OncoTargets and therapy, Vol. 8(2015).
9- Petros-Pavlos Ypsilantis et al., "Predicting response to neoadjuvant chemotherapy with PET imaging using convolutional neural networks." PloS one, Vol. 10 (No. 9), p. e0137036, (2015).
10- Ruben Pauwels et al., "Artificial intelligence for detection of periapical lesions on intraoral radiographs: comparison between convolutional neural networks and human observers." Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, Vol. 131 (No. 5), pp. 610-16, (2021).
11- Gloria Gonella, Marco Paracchini, Elisabetta Binaghi, and Marco Marcon, "Breast Lesion Detection from Mammograms Using Deep Convolutional Neural Networks." in Proceedings of the 2020 European Symposium on Software Engineering, (2020), pp. 120-24.
12- Benedetta Savelli, Alessandro Bria, Mario Molinara, Claudio Marrocco, and Francesco Tortorella, "A multi-context CNN ensemble for small lesion detection." Artificial Intelligence in Medicine, Vol. 103p. 101749, (2020).
13- Joseph Antony, Kevin McGuinness, Noel E O'Connor, and Kieran Moran, "Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks." in 2016 23rd International Conference on Pattern Recognition (ICPR), (2016): IEEE, pp. 1195-200.
14- Lei Bi, Jinman Kim, Ashnil Kumar, and Dagan Feng, "Automatic liver lesion detection using cascaded deep residual networks." arXiv preprint arXiv:1704.02703, (2017).
15- Ruikai Zhang et al., "Automatic detection and classification of colorectal polyps by transferring low-level CNN features from nonmedical domain." IEEE journal of biomedical and health informatics, Vol. 21 (No. 1), pp. 41-47, (2016).
16- Varun Gulshan et al., "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs." Jama, Vol. 316 (No. 22), pp. 2402-10, (2016).
17- Hayit Greenspan, Bram Van Ginneken, and Ronald M Summers, "Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique." IEEE transactions on medical imaging, Vol. 35 (No. 5), pp. 1153-59, (2016).
18- Daniel S Kermany et al., "Identifying medical diagnoses and treatable diseases by image-based deep learning." Cell, Vol. 172 (No. 5), pp. 1122-31. e9, (2018).
19- Chaochao Yan, Jiawen Yao, Ruoyu Li, Zheng Xu, and Junzhou Huang, "Weakly supervised deep learning for thoracic disease classification and localization on chest x-rays." in Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics, (2018), pp. 103-10.
20- Yaniv Bar, Idit Diamant, Lior Wolf, Sivan Lieberman, Eli Konen, and Hayit Greenspan, "Chest pathology detection using deep learning with non-medical training." in 2015 IEEE 12th international symposium on biomedical imaging (ISBI), (2015): IEEE, pp. 294-97.
21- Jakub Olczak et al., "Artificial intelligence for analyzing orthopedic trauma radiographs: deep learning algorithms—are they on par with humans for diagnosing fractures?" Acta orthopaedica, Vol. 88 (No. 6), pp. 581-86, (2017).
22- Robert Lindsey et al., "Deep neural network improves fracture detection by clinicians." Proceedings of the National Academy of Sciences, Vol. 115 (No. 45), pp. 11591-96, (2018).
23- Hieu H. Pham, Tung T. Le, Dat Q. Tran, Dat T. Ngo, and Ha Q. Nguyen, "Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels." Neurocomputing, Vol. 437pp. 186-94, 2021/05/21/ (2021).
24- Stephen Waite et al., "Analysis of perceptual expertise in radiology–Current knowledge and a new perspective." Frontiers in human neuroscience, Vol. 13p. 213, (2019).
25- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems, Vol. 25(2012).
26- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, "Mask r-cnn." in Proceedings of the IEEE international conference on computer vision, (2017), pp. 2961-69.
27- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 770-78.
28- Fatemeh Abdolali, Jeevesh Kapur, Jacob L Jaremko, Michelle Noga, Abhilash R Hareendranathan, and Kumaradevan Punithakumar, "Automated thyroid nodule detection from ultrasound imaging using deep convolutional neural networks." Computers in Biology and Medicine, Vol. 122p. 103871, (2020).
29- Andrew G Howard et al., "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861, (2017).
30- Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation." in International Conference on Medical image computing and computer-assisted intervention, (2015): Springer, pp. 234-41.
31- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, "Rethinking the inception architecture for computer vision." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2818-26.
32- César Ferri, José Hernández-Orallo, and R Modroiu, "An experimental comparison of performance measures for classification." Pattern recognition letters, Vol. 30 (No. 1), pp. 27-38, (2009).
33- Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz, "Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation." in Australasian joint conference on artificial intelligence, (2006): Springer, pp. 1015-21.
34- Jacob Cohen, "A coefficient of agreement for nominal scales." Educational and psychological measurement, Vol. 20 (No. 1), pp. 37-46, (1960).
35- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large-scale hierarchical image database." in 2009 IEEE conference on computer vision and pattern recognition, (2009): Ieee, pp. 248-55.
36- Jianning Chi, Ekta Walia, Paul Babyn, Jimmy Wang, Gary Groot, and Mark Eramian, "Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network." Journal of digital imaging, Vol. 30 (No. 4), pp. 477-86, (2017).
37- Tianjiao Liu, Shuaining Xie, Jing Yu, Lijuan Niu, and Weidong Sun, "Classification of thyroid nodules in ultrasound images using deep model based transfer learning and hybrid features." in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2017): IEEE, pp. 919-23.
38- Sarang Narkhede, "Understanding auc-roc curve." Towards Data Science, Vol. 26 (No. 1), pp. 220-27, (2018).
39- Maya Varma et al., "Automated abnormality detection in lower extremity radiographs using deep learning." Nature Machine Intelligence, Vol. 1 (No. 12), pp. 578-83, (2019).
40- Nitesh Pradhan, Vijaypal Singh Dhaka, and Himanshu Chaudhary, "Classification of human bones using deep convolutional neural network." in IOP conference series: materials science and engineering, (2019), Vol. 594 (No. 1): IOP Publishing, p. 012024.
41- Yunxue Shao and Xin Wang, "A two stage method for abnormality diagnosis of musculoskeletal radiographs." in International Conference on Pattern Recognition and Artificial Intelligence, (2020): Springer, pp. 610-21.
42- Janick Rohrbach, Tobias Reinhard, Beate Sick, and Oliver Dürr, "Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks." Computers & Electrical Engineering, Vol. 78pp. 472-81, (2019).
43- Jesse C Rayan, Nakul Reddy, J Herman Kan, Wei Zhang, and Ananth Annapragada, "Binomial classification of pediatric elbow fractures using a deep learning multiview approach emulating radiologist decision making." Radiology: Artificial Intelligence, Vol. 1 (No. 1), p. e180015, (2019).
44- Gabriel Chartrand et al., "Deep learning: a primer for radiologists." Radiographics, Vol. 37 (No. 7), pp. 2113-31, (2017).
45- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson, "How transferable are features in deep neural networks?" Advances in neural information processing systems, Vol. 27(2014).
46- William Gale, Luke Oakden-Rayner, Gustavo Carneiro, Andrew P Bradley, and Lyle J Palmer, "Detecting hip fractures with radiologist-level performance using deep neural networks." arXiv preprint arXiv:1711.06504, (2017).
47- Elizabeth A Krupinski, Kevin S Berbaum, Robert T Caldwell, Kevin M Schartz, and John Kim, "Long radiology workdays reduce detection and accommodation accuracy." Journal of the American College of Radiology, Vol. 7 (No. 9), pp. 698-704, (2010).
| Files | ||
| Issue | Vol 10 No 2 (2023) | |
| Section | Original Article(s) | |
| DOI | https://doi.org/10.18502/fbt.v10i2.12221 | |
| Keywords | ||
| X-Ray Lower Limb Deep Learning Detection Mask Regional Convolutional Neural Network | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Amiri S, Akbarabadi M, Rimaz S, Abdolali F, Ahadi R, Afshani M, alaei askarabad Z, Kowsarirad T, Sakinehpour S, Eyvazzadeh N, Cheraghi S. Designing an Intelligent Lesion Detection System Using Deep Architecture Neural Networks in the Lower Limb X-Ray Images. Frontiers Biomed Technol. 2023;10(2):169-179.
