Integration of Multimodal Large Language Models in Medical Imaging and Omics Data: A Comprehensive Review
Abstract
This review explores the integration of multimodal large language models (MLLMs) with medical imaging and omics data, highlighting their transformative potential in healthcare, particularly for disease diagnosis and treatment. By combining imaging techniques like MRI, CT, and PET with omics data including genomics, transcriptomics, and proteomics. MLLMs facilitate a holistic approach to understanding disease mechanisms at molecular and structural levels. Advanced AI models, such as deep learning and machine learning algorithms, enhance diagnostic precision by identifying biomarkers, predicting survival outcomes, and enabling targeted cancer therapies. The paper examines key applications, such as multimodal AI in cancer prognosis, single-cell analysis, and radiomics in precision medicine, while discussing challenges like data complexity and feature selection. The comprehensive review underscores the impact of MLLMs on disease management, paving the way for significant improvements in clinical decision-making and patient outcomes.
1- D. Ramazzotti, A. Lal, B. Wang, S. Batzoglou, and A. Sidow, "Multi-omic tumor data reveal diversity of molecular mechanisms that correlate with survival." Nat Commun, Vol. 9 (No. 1), p. 4453, Oct 26 (2018).
2- R. J. Chen et al., "Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis." IEEE Trans Med Imaging, Vol. 41 (No. 4), pp. 757-70, Apr (2022).
3- Jie Hao, Sai Chandra Kosaraju, Nelson Zange Tsaku, Dae Hyun Song, and Mingon Kang, "PAGE-Net: Interpretable and Integrative Deep Learning for Survival Analysis Using Histopathological Images and Genomic Data." Presented at the Biocomputing 2020, (2019).
4- T. Bhattacharya et al., "AI Meets Exascale Computing: Advancing Cancer Research With Large-Scale High Performance Computing." Front Oncol, Vol. 9p. 984, (2019).
5- F. Mohsen, H. Ali, N. El Hajj, and Z. Shah, "Artificial intelligence-based methods for fusion of electronic health records and imaging data." Sci Rep, Vol. 12 (No. 1), p. 17981, Oct 26 (2022).
6- Md Rezaul Karim, Tanhim Islam, Christoph Lange, Dietrich Rebholz-Schuhmann, and Stefan Decker, "Adversary-Aware Multimodal Neural Networks for
Cancer Susceptibility Prediction From Multiomics Data." IEEE Access, Vol. 10pp. 54386-409, (2022).
7- K. Chaudhary, O. B. Poirion, L. Lu, and L. X. Garmire, "Deep Learning-Based Multi-Omics Integration Robustly Predicts Survival in Liver Cancer." Clin Cancer Res, Vol. 24 (No. 6), pp. 1248-59, Mar 15 (2018).
8- K. Sone et al., "Application of artificial intelligence in gynecologic malignancies: A review." J Obstet Gynaecol Res, Vol. 47 (No. 8), pp. 2577-85, Aug (2021).
9- H. Arimura, M. Soufi, H. Kamezawa, K. Ninomiya, and M. Yamada, "Radiomics with artificial intelligence for precision medicine in radiation therapy." J Radiat Res, Vol. 60 (No. 1), pp. 150-57, Jan 1 (2019).
10- M. Efremova and S. A. Teichmann, "Computational methods for single-cell omics across modalities." Nat Methods, Vol. 17 (No. 1), pp. 14-17, Jan (2020).
11- H. Xiao, Zhou, F., Liu, X., Liu, T., Li, Z., Liu, X., & Huang, X, "A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine." DOI: 10.48550/arxiv.2405.08603
12- D. A. Elhag et al., "Inflammatory Bowel Disease Treatments and Predictive Biomarkers of Therapeutic Response." Int J Mol Sci, Vol. 23 (No. 13), Jun 23 (2022).
13- Laura Antonelli, Mario Rosario Guarracino, Lucia Maddalena, and Mara Sangiovanni, "Integrating imaging and omics data: A review." Biomedical Signal Processing and Control, Vol. 52pp. 264-80, (2019).
14- K. M. Boehm, P. Khosravi, R. Vanguri, J. Gao, and S. P. Shah, "Harnessing multimodal data integration to advance precision oncology." Nat Rev Cancer, Vol. 22 (No. 2), pp. 114-26, Feb (2022).
15- University Radiology Associates. PET/CT Scan. [Online]. Available: https://www.upstate.edu/ura/pet-ct-scan.php.
16- Wikipedia contributors. (2023). H&E stain. [Online]. Available: https://en.wikipedia.org/w/index.php?title=H%26E_stain&oldid=1191484566.
17- Using FAST on Whole-Slide Images (WSI). [Online]. Available: https://fast.eriksmistad.no/python-tutorial-wsi.html.
18- Supporting Multi-omics Approaches. (August 27). [Online]. Available: https://www.thermofisher.com/uk/en/home/brands/thermo-scientific/molecular-biology/molecular-biologylearning-center/molecular-biology-resource-library/spotlight-articles/supporting-multi-omics-approaches.html.
19- C. Massard et al., "High-Throughput Genomics and Clinical Outcome in Hard-to-Treat Advanced Cancers: Results of the MOSCATO 01 Trial." Cancer Discov, Vol. 7 (No. 6), pp. 586-95, Jun (2017).
20- W. Silva, & Dos, S. S. W, "Review on multi-modal AI models to integrate imaging and omics data." Utrecht University, (2024).
21- National Cancer Institute. (24 Aug). NCI Genomic Data Commons. [Online]. Available: https://gdc.cancer.gov/.
22- The Cancer Imaging Archive (TCIA). National Cancer Institute. [Online]. Available: https://imaging.cancer.gov/informatics/cancer_imaging_archive.htm.
23- R. Sun et al., "A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study." Lancet Oncol, Vol. 19 (No. 9), pp. 1180-91, Sep (2018).
24- Sina Tabakhi, Mohammod Naimul Islam Suvon, Pegah Ahadian, and Haiping Lu, "Multimodal Learning for Multi-omics: A Survey." World Scientific Annual Review of Artificial Intelligence, Vol. 01(2023).
25- National Genomics Data Center. EWAS Datahub. [Online]. Available: https://ngdc.cncb.ac.cn/ewas/datahub.
26- CPTAC | Office of Cancer Clinical Proteomics Research. National Cancer Institute. [Online]. Available: https://proteomics.cancer.gov/programs/cptac.
27- Z. Wang, M. Gerstein, and M. Snyder, "RNA-Seq: a revolutionary tool for transcriptomics." Nat Rev Genet, Vol. 10 (No. 1), pp. 57-63, Jan (2009).
28- S. C. Sealfon and T. T. Chu, "RNA and DNA microarrays." Methods Mol Biol, Vol. 671pp. 3-34, (2011).
29- ENCODE Project Consortium. (August 23). [Online]. Available: https://www.encodeproject.org/.
30- A. Hosny, C. Parmar, J. Quackenbush, L. H. Schwartz, and Hjwl Aerts, "Artificial intelligence in radiology." Nat Rev Cancer, Vol. 18 (No. 8), pp. 500-10, Aug (2018).
31- UK Biobank. (August 23). [Online]. Available: https://www.ukbiobank.ac.uk/.
32- L. M. Gibson et al., "Factors associated with potentially serious incidental findings and with serious final diagnoses on multi-modal imaging in the UK Biobank Imaging Study: A prospective cohort study." PLoS One, Vol. 14 (No. 6), p. e0218267, (2019).
33- A. Partin, Brettin, T., Zhu, Y., Dolezal, J. M., Kochanny, S., Pearson, A. T., Shukla, M., Evrard, Y. A., Doroshow, J. H., & Stevens, R. L., "Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images." DOI: 10.48550/arXiv.2204.11678
34- Patient-Derived Models Repository (PDMR). [Online]. Available: https://pdmr.cancer.gov/.
35- National Institutes of Health. (August 23). NIH LINCS Program. [Online]. Available: https://lincsproject.org/.
36- P. Bankhead, & others. (August 20). QuPath. [Online]. Available: https://qupath.readthedocs.io/en/stable/
37- J. M. Dolezal, Kochanny, S., Dyer, E., Srisuwananukorn, A., Sacco, M., Howard, F. M., Li, A., Mohan, P., & Pearson, A. and T, "Slideflow: Deep Learning for Digital Histopathology with Real-Time Whole-Slide Visualization." DOI: https://doi.org/10.48550/arXiv.2304.04142
38- E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, "Color transfer between images." IEEE Computer Graphics and Applications, Vol. 21 (No. 4), pp. 34-41, (2001).
39- Robert Tibshirani, "Regression Shrinkage and Selection Via the Lasso." Journal of the Royal Statistical Society Series B: Statistical Methodology, Vol. 58 (No. 1), pp. 267-88, (1996).
40- J. Yan, S. L. Risacher, K. Nho, A. J. Saykin, and L. I. Shen, "Identification of Discriminative Imaging Proteomics Associations in Alzheimer's Disease Via a Novel Sparse Correlation Model." Pac Symp Biocomput, Vol. 22pp. 94-104, (2017).
41- E. M. Airoldi, E. A. Erosheva, S. E. Fienberg, C. Joutard, T. Love, and S. Shringarpure, "Reconceptualizing the classification of PNAS articles." Proc Natl Acad Sci U S A, Vol. 107 (No. 49), pp. 20899-904, Dec 7 (2010).
42- L. Schneider et al., "Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review." Eur J Cancer, Vol. 160pp. 80-91, Jan (2022).
43- Zabir Al Nazi and Wei Peng, "Large Language Models in Healthcare and Medical Domain: A Review." Informatics, Vol. 11 (No. 3), (2024).
44- E. R. Watson, A. Taherian Fard, and J. C. Mar, "Computational Methods for Single-Cell Imaging and Omics Data Integration." Front Mol Biosci, Vol. 8p. 768106, (2021).
45- Raja Vavekanand, Kira Sam, Suresh Kumar, and Teerath Kumar, "CardiacNet: A Neural Networks Based Heartbeat Classifications using ECG Signals." Studies in Medical and Health Sciences, Vol. 1 (No. 2), pp. 1-17, (2024).
46- X. Jiang, Yan, L., Vavekanand, R., & Hu, M, "Large Language Models in Healthcare Current Development and Future Directions." DOI: 10.20944/preprints202407.0923.v1
47- T. Zhong, Zhao, W., Zhang, Y., Pan, Y., Dong, P., Jiang, Z., Kui, X., Shang, Y., Yang, L., Wei, Y., Yang, L., Chen, H., Zhao, H., Liu, Y., Zhu, N., Li, Y., Wang, Y., Yao, J., Wang, J., . . . Zhang, T. , "ChatRadioValuer: a Chat large language model for generalizable radiology report generation based on multi-institution and multi-system data." DOI: 10.48550/arxiv.2310.05242
48- J. Zhou et al., "Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4." Nat Commun, Vol. 15 (No. 1), p. 5649, Jul 5 (2024).
49- S. Zhou et al., "Large language models for disease diagnosis: a scoping review." NPJ Artif Intell, Vol. 1 (No. 1), p. 9, (2025).
50- F. Gaber et al., "Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis." NPJ Digit Med, Vol. 8 (No. 1), p. 263, May 9 (2025).
51- M. Mittermaier, M. M. Raza, and J. C. Kvedar, "Bias in AI-based models for medical applications: challenges and mitigation strategies." NPJ Digit Med, Vol. 6 (No. 1), p. 113, Jun 14 (2023).
52- S. Carey, A. Pang, and M. Kamps, "Fairness in AI for healthcare." Future Healthc J, Vol. 11 (No. 3), p. 100177, Sep (2024).
53- M. A. Ricci Lara, R. Echeveste, and E. Ferrante, "Addressing fairness in artificial intelligence for medical imaging." Nat Commun, Vol. 13 (No. 1), p. 4581, Aug 6 (2022).
54- Emilio Ferrara, "Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies." Sci, Vol. 6 (No. 1), (2023).
55- H. Zeng, L. Chen, Y. Huang, Y. Luo, and X. Ma, "Integrative Models of Histopathological Image Features and Omics Data Predict Survival in Head and Neck Squamous Cell Carcinoma." Front Cell Dev Biol, Vol. 8p. 553099, (2020).
56- G. Puliani et al., "NETest: A Systematic Review Focusing on the Prognostic and Predictive Role." Neuroendocrinology, Vol. 112 (No. 6), pp. 523-36, (2022).
57- Mutlu Cukurova, Carmel Kent, and Rosemary Luckin, "Artificial intelligence and multimodal data in the service of human decision‐making: A case study in debate tutoring." British Journal of Educational Technology, Vol. 50 (No. 6), pp. 3032-46, (2019).
58- Raja Vavekanand, Ganesh Kumar, and Shakhlokhon Kurbanova, "A lightweight physics-conditioned diffusion multi-model for medical image reconstruction." Biomedical Engineering Communications, Vol. 5 (No. 2), (2026).
59- M. R. Farlow, A. Hake, J. Messina, R. Hartman, J. Veach, and R. Anand, "Response of patients with Alzheimer disease to rivastigmine treatment is predicted by the rate of disease progression." Arch Neurol, Vol. 58 (No. 3), pp. 417-22, Mar (2001).
60- Wenjing Gao, Yuanyuan Pei, Huiying Liang, Junjian Lv, Jiale Chen, and Wei Zhong, "Multimodal AI System for the Rapid Diagnosis and Surgical Prediction of Necrotizing Enterocolitis." IEEE Access, Vol. 9pp. 51050-64, (2021).
61- Yexian Zhang et al., "ELMO: An Efficient Logistic Regression-Based Multi-Omic Integrated Analysis Method for Breast Cancer Intrinsic Subtypes." IEEE Access, Vol. 8pp. 5121-30, (2020).
62- Shao W Wang T, Huang Z, Tang H, Zhang J, Ding Z, Huang K, "MORONET: Multi-omics Integration via Graph Convolutional Networks for Biomedical Data Classification." DOI: https://doi.org/10.1101/2020.07.02.184705
63- Weiwei Li, Aniruddha Bhattacharjya, and Xin Feng, "Prediction and analysis of attribute reduction algorithm in rough set in Alzheimer's disease." Presented at the International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2023), (2023).
64- B. ElKarami, A. Alkhateeb, H. Qattous, L. Alshomali, and B. Shahrrava, "Multi-omics Data Integration Model Based on UMAP Embedding and Convolutional Neural Network." Cancer Inform, Vol. 21p. 11769351221124205, (2022).
65- Ayush Garg and Deepika Bansal, "Application of Machine Learning in Disease Prediction." International Journal for Research in Applied Science and Engineering Technology, Vol. 11 (No. 2), pp. 47-51, (2023).
2- R. J. Chen et al., "Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis." IEEE Trans Med Imaging, Vol. 41 (No. 4), pp. 757-70, Apr (2022).
3- Jie Hao, Sai Chandra Kosaraju, Nelson Zange Tsaku, Dae Hyun Song, and Mingon Kang, "PAGE-Net: Interpretable and Integrative Deep Learning for Survival Analysis Using Histopathological Images and Genomic Data." Presented at the Biocomputing 2020, (2019).
4- T. Bhattacharya et al., "AI Meets Exascale Computing: Advancing Cancer Research With Large-Scale High Performance Computing." Front Oncol, Vol. 9p. 984, (2019).
5- F. Mohsen, H. Ali, N. El Hajj, and Z. Shah, "Artificial intelligence-based methods for fusion of electronic health records and imaging data." Sci Rep, Vol. 12 (No. 1), p. 17981, Oct 26 (2022).
6- Md Rezaul Karim, Tanhim Islam, Christoph Lange, Dietrich Rebholz-Schuhmann, and Stefan Decker, "Adversary-Aware Multimodal Neural Networks for
Cancer Susceptibility Prediction From Multiomics Data." IEEE Access, Vol. 10pp. 54386-409, (2022).
7- K. Chaudhary, O. B. Poirion, L. Lu, and L. X. Garmire, "Deep Learning-Based Multi-Omics Integration Robustly Predicts Survival in Liver Cancer." Clin Cancer Res, Vol. 24 (No. 6), pp. 1248-59, Mar 15 (2018).
8- K. Sone et al., "Application of artificial intelligence in gynecologic malignancies: A review." J Obstet Gynaecol Res, Vol. 47 (No. 8), pp. 2577-85, Aug (2021).
9- H. Arimura, M. Soufi, H. Kamezawa, K. Ninomiya, and M. Yamada, "Radiomics with artificial intelligence for precision medicine in radiation therapy." J Radiat Res, Vol. 60 (No. 1), pp. 150-57, Jan 1 (2019).
10- M. Efremova and S. A. Teichmann, "Computational methods for single-cell omics across modalities." Nat Methods, Vol. 17 (No. 1), pp. 14-17, Jan (2020).
11- H. Xiao, Zhou, F., Liu, X., Liu, T., Li, Z., Liu, X., & Huang, X, "A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine." DOI: 10.48550/arxiv.2405.08603
12- D. A. Elhag et al., "Inflammatory Bowel Disease Treatments and Predictive Biomarkers of Therapeutic Response." Int J Mol Sci, Vol. 23 (No. 13), Jun 23 (2022).
13- Laura Antonelli, Mario Rosario Guarracino, Lucia Maddalena, and Mara Sangiovanni, "Integrating imaging and omics data: A review." Biomedical Signal Processing and Control, Vol. 52pp. 264-80, (2019).
14- K. M. Boehm, P. Khosravi, R. Vanguri, J. Gao, and S. P. Shah, "Harnessing multimodal data integration to advance precision oncology." Nat Rev Cancer, Vol. 22 (No. 2), pp. 114-26, Feb (2022).
15- University Radiology Associates. PET/CT Scan. [Online]. Available: https://www.upstate.edu/ura/pet-ct-scan.php.
16- Wikipedia contributors. (2023). H&E stain. [Online]. Available: https://en.wikipedia.org/w/index.php?title=H%26E_stain&oldid=1191484566.
17- Using FAST on Whole-Slide Images (WSI). [Online]. Available: https://fast.eriksmistad.no/python-tutorial-wsi.html.
18- Supporting Multi-omics Approaches. (August 27). [Online]. Available: https://www.thermofisher.com/uk/en/home/brands/thermo-scientific/molecular-biology/molecular-biologylearning-center/molecular-biology-resource-library/spotlight-articles/supporting-multi-omics-approaches.html.
19- C. Massard et al., "High-Throughput Genomics and Clinical Outcome in Hard-to-Treat Advanced Cancers: Results of the MOSCATO 01 Trial." Cancer Discov, Vol. 7 (No. 6), pp. 586-95, Jun (2017).
20- W. Silva, & Dos, S. S. W, "Review on multi-modal AI models to integrate imaging and omics data." Utrecht University, (2024).
21- National Cancer Institute. (24 Aug). NCI Genomic Data Commons. [Online]. Available: https://gdc.cancer.gov/.
22- The Cancer Imaging Archive (TCIA). National Cancer Institute. [Online]. Available: https://imaging.cancer.gov/informatics/cancer_imaging_archive.htm.
23- R. Sun et al., "A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study." Lancet Oncol, Vol. 19 (No. 9), pp. 1180-91, Sep (2018).
24- Sina Tabakhi, Mohammod Naimul Islam Suvon, Pegah Ahadian, and Haiping Lu, "Multimodal Learning for Multi-omics: A Survey." World Scientific Annual Review of Artificial Intelligence, Vol. 01(2023).
25- National Genomics Data Center. EWAS Datahub. [Online]. Available: https://ngdc.cncb.ac.cn/ewas/datahub.
26- CPTAC | Office of Cancer Clinical Proteomics Research. National Cancer Institute. [Online]. Available: https://proteomics.cancer.gov/programs/cptac.
27- Z. Wang, M. Gerstein, and M. Snyder, "RNA-Seq: a revolutionary tool for transcriptomics." Nat Rev Genet, Vol. 10 (No. 1), pp. 57-63, Jan (2009).
28- S. C. Sealfon and T. T. Chu, "RNA and DNA microarrays." Methods Mol Biol, Vol. 671pp. 3-34, (2011).
29- ENCODE Project Consortium. (August 23). [Online]. Available: https://www.encodeproject.org/.
30- A. Hosny, C. Parmar, J. Quackenbush, L. H. Schwartz, and Hjwl Aerts, "Artificial intelligence in radiology." Nat Rev Cancer, Vol. 18 (No. 8), pp. 500-10, Aug (2018).
31- UK Biobank. (August 23). [Online]. Available: https://www.ukbiobank.ac.uk/.
32- L. M. Gibson et al., "Factors associated with potentially serious incidental findings and with serious final diagnoses on multi-modal imaging in the UK Biobank Imaging Study: A prospective cohort study." PLoS One, Vol. 14 (No. 6), p. e0218267, (2019).
33- A. Partin, Brettin, T., Zhu, Y., Dolezal, J. M., Kochanny, S., Pearson, A. T., Shukla, M., Evrard, Y. A., Doroshow, J. H., & Stevens, R. L., "Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images." DOI: 10.48550/arXiv.2204.11678
34- Patient-Derived Models Repository (PDMR). [Online]. Available: https://pdmr.cancer.gov/.
35- National Institutes of Health. (August 23). NIH LINCS Program. [Online]. Available: https://lincsproject.org/.
36- P. Bankhead, & others. (August 20). QuPath. [Online]. Available: https://qupath.readthedocs.io/en/stable/
37- J. M. Dolezal, Kochanny, S., Dyer, E., Srisuwananukorn, A., Sacco, M., Howard, F. M., Li, A., Mohan, P., & Pearson, A. and T, "Slideflow: Deep Learning for Digital Histopathology with Real-Time Whole-Slide Visualization." DOI: https://doi.org/10.48550/arXiv.2304.04142
38- E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, "Color transfer between images." IEEE Computer Graphics and Applications, Vol. 21 (No. 4), pp. 34-41, (2001).
39- Robert Tibshirani, "Regression Shrinkage and Selection Via the Lasso." Journal of the Royal Statistical Society Series B: Statistical Methodology, Vol. 58 (No. 1), pp. 267-88, (1996).
40- J. Yan, S. L. Risacher, K. Nho, A. J. Saykin, and L. I. Shen, "Identification of Discriminative Imaging Proteomics Associations in Alzheimer's Disease Via a Novel Sparse Correlation Model." Pac Symp Biocomput, Vol. 22pp. 94-104, (2017).
41- E. M. Airoldi, E. A. Erosheva, S. E. Fienberg, C. Joutard, T. Love, and S. Shringarpure, "Reconceptualizing the classification of PNAS articles." Proc Natl Acad Sci U S A, Vol. 107 (No. 49), pp. 20899-904, Dec 7 (2010).
42- L. Schneider et al., "Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review." Eur J Cancer, Vol. 160pp. 80-91, Jan (2022).
43- Zabir Al Nazi and Wei Peng, "Large Language Models in Healthcare and Medical Domain: A Review." Informatics, Vol. 11 (No. 3), (2024).
44- E. R. Watson, A. Taherian Fard, and J. C. Mar, "Computational Methods for Single-Cell Imaging and Omics Data Integration." Front Mol Biosci, Vol. 8p. 768106, (2021).
45- Raja Vavekanand, Kira Sam, Suresh Kumar, and Teerath Kumar, "CardiacNet: A Neural Networks Based Heartbeat Classifications using ECG Signals." Studies in Medical and Health Sciences, Vol. 1 (No. 2), pp. 1-17, (2024).
46- X. Jiang, Yan, L., Vavekanand, R., & Hu, M, "Large Language Models in Healthcare Current Development and Future Directions." DOI: 10.20944/preprints202407.0923.v1
47- T. Zhong, Zhao, W., Zhang, Y., Pan, Y., Dong, P., Jiang, Z., Kui, X., Shang, Y., Yang, L., Wei, Y., Yang, L., Chen, H., Zhao, H., Liu, Y., Zhu, N., Li, Y., Wang, Y., Yao, J., Wang, J., . . . Zhang, T. , "ChatRadioValuer: a Chat large language model for generalizable radiology report generation based on multi-institution and multi-system data." DOI: 10.48550/arxiv.2310.05242
48- J. Zhou et al., "Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4." Nat Commun, Vol. 15 (No. 1), p. 5649, Jul 5 (2024).
49- S. Zhou et al., "Large language models for disease diagnosis: a scoping review." NPJ Artif Intell, Vol. 1 (No. 1), p. 9, (2025).
50- F. Gaber et al., "Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis." NPJ Digit Med, Vol. 8 (No. 1), p. 263, May 9 (2025).
51- M. Mittermaier, M. M. Raza, and J. C. Kvedar, "Bias in AI-based models for medical applications: challenges and mitigation strategies." NPJ Digit Med, Vol. 6 (No. 1), p. 113, Jun 14 (2023).
52- S. Carey, A. Pang, and M. Kamps, "Fairness in AI for healthcare." Future Healthc J, Vol. 11 (No. 3), p. 100177, Sep (2024).
53- M. A. Ricci Lara, R. Echeveste, and E. Ferrante, "Addressing fairness in artificial intelligence for medical imaging." Nat Commun, Vol. 13 (No. 1), p. 4581, Aug 6 (2022).
54- Emilio Ferrara, "Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies." Sci, Vol. 6 (No. 1), (2023).
55- H. Zeng, L. Chen, Y. Huang, Y. Luo, and X. Ma, "Integrative Models of Histopathological Image Features and Omics Data Predict Survival in Head and Neck Squamous Cell Carcinoma." Front Cell Dev Biol, Vol. 8p. 553099, (2020).
56- G. Puliani et al., "NETest: A Systematic Review Focusing on the Prognostic and Predictive Role." Neuroendocrinology, Vol. 112 (No. 6), pp. 523-36, (2022).
57- Mutlu Cukurova, Carmel Kent, and Rosemary Luckin, "Artificial intelligence and multimodal data in the service of human decision‐making: A case study in debate tutoring." British Journal of Educational Technology, Vol. 50 (No. 6), pp. 3032-46, (2019).
58- Raja Vavekanand, Ganesh Kumar, and Shakhlokhon Kurbanova, "A lightweight physics-conditioned diffusion multi-model for medical image reconstruction." Biomedical Engineering Communications, Vol. 5 (No. 2), (2026).
59- M. R. Farlow, A. Hake, J. Messina, R. Hartman, J. Veach, and R. Anand, "Response of patients with Alzheimer disease to rivastigmine treatment is predicted by the rate of disease progression." Arch Neurol, Vol. 58 (No. 3), pp. 417-22, Mar (2001).
60- Wenjing Gao, Yuanyuan Pei, Huiying Liang, Junjian Lv, Jiale Chen, and Wei Zhong, "Multimodal AI System for the Rapid Diagnosis and Surgical Prediction of Necrotizing Enterocolitis." IEEE Access, Vol. 9pp. 51050-64, (2021).
61- Yexian Zhang et al., "ELMO: An Efficient Logistic Regression-Based Multi-Omic Integrated Analysis Method for Breast Cancer Intrinsic Subtypes." IEEE Access, Vol. 8pp. 5121-30, (2020).
62- Shao W Wang T, Huang Z, Tang H, Zhang J, Ding Z, Huang K, "MORONET: Multi-omics Integration via Graph Convolutional Networks for Biomedical Data Classification." DOI: https://doi.org/10.1101/2020.07.02.184705
63- Weiwei Li, Aniruddha Bhattacharjya, and Xin Feng, "Prediction and analysis of attribute reduction algorithm in rough set in Alzheimer's disease." Presented at the International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2023), (2023).
64- B. ElKarami, A. Alkhateeb, H. Qattous, L. Alshomali, and B. Shahrrava, "Multi-omics Data Integration Model Based on UMAP Embedding and Convolutional Neural Network." Cancer Inform, Vol. 21p. 11769351221124205, (2022).
65- Ayush Garg and Deepika Bansal, "Application of Machine Learning in Disease Prediction." International Journal for Research in Applied Science and Engineering Technology, Vol. 11 (No. 2), pp. 47-51, (2023).
| Files | ||
| Issue | Vol 13 No 1 (2026) | |
| Section | Literature (Narrative) Review(s) | |
| DOI | https://doi.org/10.18502/fbt.v13i1.20792 | |
| Keywords | ||
| Multimodal Large Language Models Medical Imaging Omics Data Generative Artificial Intelligence | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Vavekanand R. Integration of Multimodal Large Language Models in Medical Imaging and Omics Data: A Comprehensive Review. Frontiers Biomed Technol. 2026;13(1):266-284.
