Multimodal deep learning for predicting COVID-19 patients at high-risk for serious illness

doi:10.19871/j.cnki.xfcrbzz.2021.02.013

Abstract

Abstract: Objective Using deep learning and big data technology to identify potential high-risk patients with severe conversion of COVID-19, we aim to build a deep learning model so as to identify potential high-risk patients and help doctors make timely individualized treatment plan. Methods From 24 January, 2020 to 16 February, 2020, 216 cases of COVID-19 patients' many modalities data from Chongqing Public Health Center were collected. We build a novel deep learning model for assessing patients' current condition and predicting those who are at high risk for developing severe illness. Results Extensive experimental results show that our model can achieve the over 95% and 90% accuracies for assessment and prediction, respectively. Conclusions The multimodal deep learning model outperforms conventional linear regression model. Meanwhile, Multimodal data can accurately predict the risk of severe transformation of COVID-19.

Key words: Coronavirus disease 2019, Condition assessment and prediction, Deep learning, Ultimodal data fusion

Li Shikang, Li Zhuo, Xu Ruiqing, Yan Xiaofeng, Li Jianhua, Lyu Liang, Song Yuyan, Sun Qiangzhong, Li Tongxin, Qian Lanlan, Zhang Ying. Multimodal deep learning for predicting COVID-19 patients at high-risk for serious illness[J]. Electronic Journal of Emerging Infectious Diseases, 2021, 6(2): 133-137.

References

[1] ZHU N, ZHANG D, WANG WL, et al.Gao and Wenjie Tan. A Novel Coronavirus from Patients with Pneumonia in China, 2019[J]. N Engl J Med, 2020, 382(8):727-733.
[2] WU F, ZHAO S, YU B, et al.A New Coronavirus Associated with Human Respiratory Disease in China[J]. Nature, 2020, 579(7798):265-269.
[3] HUANG CL, WANGYM, LI XW, et al. Clinical Features of Patients Infected with2019 Novel Coronavirus in Wuhan, China[J]. The Lancet, 395(10223):497-506.
[4] YANGXB, YU Y, XU JQ, et al.Clinical Course and Outcomes of Critically Ill Patients with SARS-CoV-2 Pneumonia in Wuhan, China: A Single-centered, Retrospective, Observational Study[J]. Lancet Respir Med,2020, 8(5): 475-481.
[5] BHATRAJU PK, GHASSEMIEH BJ, NICHOLS M, et al.Covid-19 in Critically Ill Patients in the Seattle Region - Case Series[J]. N Engl J Med, 2020, 382(21):2012-2022.
[6] 宋璐, 曾莹婷, 龚晓明, 等. 新型冠状病毒肺炎影像表现及鉴别诊断[J/CD]. 新发传染病电子杂志,2020, 5(2):83-86.
[7] 郑秋婷, 卢亦波, 谭理连, 等. 新型冠状病毒肺炎临床及影像学研究进展[J/CD]. 新发传染病电子杂志, 2020, 5(2):140-144.
[8] O'HALLORAN KL.Interdependence, Interaction and Metaphor in Multisemiotic Texts[J]. Social Semiotics, 1999, 9(3):317-354.
[9] BENGIO Y, COURVILLE A, VINCENT P.Representation learning: a review and new perspectives[J]. IEEE Trans Pattern Anal Mach Intell, 2013, 35(8):1798-1828.
[10] BALTRUSAITIS T, AHUJA C, MORENCY LP.Multimodal Machine Learning: A Survey and Taxonomy[J]. IEEE Trans Pattern Anal Mach Intell, 2019, 41(2):423-443.
[11] MONTGOMERY DC, PECK EA, VINING GG.Introduction to Linear Regression Analysis[M]. Wiley, 1982: 18-29.
[12] SCHNEIDER A, HOMMEL G, BLETTNER M.Linear Regression Analysis: Part 14 of A Series on Evaluation of Scientific Publications[J]. Dtsch Arztebl Int, 2010, 107(44): 776-782.
[13] FREEDMAN DA.Statistical Models: Theory and Practice[M]. Cambridge University Press, 2005: 112-118.
[14] BROCKWELL PJ, DAVIS R.A. Introduction to Time Series and Forecasting[M]. Springer(2nd ed.), 2002.
[15] ZHANG PG.Time Series Forecasting Using A Hybrid ARIMA and Neural Network Model[J]. Neuro Computing, 2003, 50:159-175.
[16] SILVER D, HUANG A, MADDISON CJ, et al.Mastering the Game of Go with Deep Neural Networks and Tree Search[J]. Nature, 2016, 529(7587):484-489.
[17] SILVER D, SCHRITTWIESER J, SIMONYAN K, et al.Mastering the Game of Go without Human Knowledge[J]. Nature,2017, 550(7676):354-359.
[18] GRANTER SR, BECK AH, PAPKE DJ.AlphaGo, Deep Learning, and the Future of the Human Microscopist[J]. Arch Pathol Lab Med, 2017, 141(5):619-621.
[19] TOPOL EJ.High-performance Medicine: The Convergence of Human and Artificial Intelligence[J]. Nat Med, 2019, 25(1):44-56.
[20] HE JX, BAXTER SL, XU J, et al.The Practical Implementation of Artificial Intelligence Technologies in Medicine[J]. Nat Med, 2019, 25(1):30-36.
[21] GUROVICH Y, HANANI Y, BAR O, at al. Identifying Facial Phenotypes of Genetic Disorders using Deep Learning[J]. Nat Med, 2019, 25(1):60-64.
[22] ZACHI IA, SURAJ K, FRANCISCO LJ, et al.Screening for Cardiac Contractile Dysfunction Sing An Artificial Intelligence-enabled Electrocardiogram[J]. Nat Med, 2019, 25(1):70-74.
[23] HANNUN AY, RAJPURKAR P, HAGHPANAHI M, et al.Cardiologist-level Arrhythmia Detection and Classification in Ambulatory Electrocardiograms Using A Deep Neural Network[J]. Nat Med, 2019, 25(1):65-69.
[24] TAGHANAKI SA, ABHISHEK K, COHEN JP, et al.Deep Semantic Segmentation of Natural and Medical Images: A Review[J]. Artif Intell Rev, 2020,54:137-178.
[25] ESTEVA A, ROBICQUET A, RAMSUNDAR B, et al.A Guide to Deep Learning in Healthcare[J]. Nat Med, 2019, 25(1):24-29.
[26] IOFFE S, SZEGEDY C.Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning[C]. JMLR.org, 2015, 37:448-456.
[27] LI Z, WANG KJ, LI CC, et al.Multimodal Deep Learning for Solar Irradiance Prediction. In Proceedings of the 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)[C]. IEEE, 2019:784-792.
[28] GOODFELLOW I, BENGIO Y, COURVILLE A.Softmax Units for Multinoulli Output Distributions. Deep Learning[M]. MIT Press, 2016: 180-184.