自监督脓毒症治疗推荐算法

朱思涵; 浦剑

doi:10.1631/FITEE.2000127

Your Location：

Home >

Browse articles >

自监督脓毒症治疗推荐算法

常规文章 | Updated：2022-06-06

- 自监督脓毒症治疗推荐算法
  Enhanced Publication
- A self-supervised method for treatment recommendation in sepsis
- 信息与电子工程前沿（英文） 2021年22卷第7期页码：926-939
- Affiliations：
  
  School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
  Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai 200433, China
- Author bio：
  
  Jian PU, E-mail: jianpu@fudan.edu.cn
- Funds：
  
  Project supported by the National Natural Science Foundation of China (No. 61702186)
- DOI：10.1631/FITEE.2000127
  中图分类号：
- 纸质出版日期：2021-07，
  
  收稿日期：2020-03-27，
  
  修回日期：2021-07-05，
  
  录用日期：2020-6-7
- Accepted：
Scan QR Code
朱思涵, 浦剑. 自监督脓毒症治疗推荐算法[J]. 信息与电子工程前沿（英文）, 2021,22(7):926-939.

SIHAN ZHU, JIAN PU. A self-supervised method for treatment recommendation in sepsis. [J]. Frontiers of information technology & electronic engineering, 2021, 22(7): 926-939.
朱思涵, 浦剑. 自监督脓毒症治疗推荐算法[J]. 信息与电子工程前沿（英文）, 2021,22(7):926-939. DOI： 10.1631/FITEE.2000127.

SIHAN ZHU, JIAN PU. A self-supervised method for treatment recommendation in sepsis. [J]. Frontiers of information technology & electronic engineering, 2021, 22(7): 926-939. DOI： 10.1631/FITEE.2000127.

摘要

由于每个脓毒症患者治疗反应可能不同，为病人提供量身定制的治疗建议来帮助医生有效、准确地做出决定，并采取有效治疗方案，是降低医院重症监护病房死亡率的一项极具挑战性的工作。本文将强化学习应用于个人治疗推荐，采用对样本不确定性进行建模并评估的方法，根据患者对治疗的反应和状态，将患者样本分为两个域，然后使用辅助迁移学习任务重建两个域的样本，使用特权学习的蒸馏方法与用于迁移学习的变分自动编码器框架关联低质量域和高质量域间的任务。通过结合自监督方式获得更好的状态和动作表示，本文提出一种针对引起较高风险的不确定性进行控制的深度强化学习方法；模型提供一定的灵活性使之可以在不同场景对模糊样本做出保守预测或明确判断，并降低预期死亡率。在大规模公开可用的真实医疗数据集MIMIC-III上的实验表明，所提模型将总体估计死亡率降低了2.3%，并将主要估计死亡率降低到9.5%。

Abstract

Sepsis treatment is a highly challenging effort to reduce mortality in hospital intensive care units since the treatment response may vary for each patient. Tailored treatment recommendations are desired to assist doctors in making decisions efficiently and accurately. In this work

we apply a self-supervised method based on reinforcement learning (RL) for treatment recommendation on individuals. An uncertainty evaluation method is proposed to separate patient samples into two domains according to their responses to treatments and the state value of the chosen policy. Examples of two domains are then reconstructed with an auxiliary transfer learning task. A distillation method of privilege learning is tied to a variational auto-encoder framework for the transfer learning task between the low- and high-quality domains. Combined with the self-supervised way for better state and action representations

we propose a deep RL method called high-risk uncertainty (HRU) control to provide flexibility on the trade-off between the effectiveness and accuracy of ambiguous samples and to reduce the expected mortality. Experiments on the large-scale publicly available real-world dataset MIMIC-Ⅲ demonstrate that our model reduces the estimated mortality rate by up to 2.3% in total

and that the estimated mortality rate in the majority of cases is reduced to 9.5%.

关键词

治疗推荐脓毒症自监督学习强化学习电子病历

Keywords

Treatment recommendationSepsisSelf-supervised learningReinforcement learningElectronic health records

references

D Almirall, , , SN Compton, , , M Gunlicks-Stoessel, , , 等. . Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. . Stat Med, , 2012. . 31((17):):1887--1902. . DOI:10.1002/sim.4512http://doi.org/10.1002/sim.4512..

E Asiain, , , JB Clempner, , , AS Poznyak. . A reinforcement learning approach for solving the mean variance customer portfolio in partially observable models. . Int J Artif Intell Tools, , 2018. . 27((8):):1850034DOI:10.1142/S0218213018500343http://doi.org/10.1142/S0218213018500343..

JM Bajor, , , TA Lasko. . Predicting medications from diagnostic codes with recurrent neural networks. . Int Conf on Learning Representations, , 2017. . p. 1--19. . ..

JG Chen, , , KL Li, , , HG Rong, , , 等. . A disease diagnosis and treatment recommendation system based on big data mining and cloud computing. . Inform Sci, , 2018. . 435124--149. . DOI:10.1016/j.ins.2018.01.001http://doi.org/10.1016/j.ins.2018.01.001..

Z Chen, , , K Marple, , , E Salazar, , , 等. . A physician advisory system for chronic heart failure management based on knowledge patterns. . Theory Pract Log Progr, , 2016. . 16((5-6):):604--618. . DOI:10.1017/S1471068416000429http://doi.org/10.1017/S1471068416000429..

J Futoma, , , S Hariharan, , , KA Heller, , , 等. . An improved multi-output Gaussian process RNN with real-time validation for early sepsis detection. . Proc 2nd Machine Learning for Healthcare Conf, , 2017. . p. 243--254. . ..

S Gidaris, , , P Singh, , , N Komodakis. . Unsupervised representation learning by predicting image rotations. . Int Conf on Learning Representations, , 2018. . p. 1--16. . ..

M Gunlicks-Stoessel, , , L Mufson, , , A Westervelt, , , 等. . A pilot smart for developing an adaptive treatment strategy for adolescent depression. . J Clin Child Adolesc Psychol, , 2016. . 45((4):):480--494. . DOI:10.1080/15374416.2015.1015133http://doi.org/10.1080/15374416.2015.1015133..

D Hendrycks, , , M Mazeika, , , S Kadavath, , , 等. . Using self-supervised learning can improve model robustness and uncertainty. . Proc 33rd Conf on Neural Information Processing Systems, , 2019. . p. 1--13. . ..

G Hinton, , , O Vinyals, , , J Dean. . Distilling the knowledge in a neural network. . 2015. . https://arxiv.org/abs/1503.02531https://arxiv.org/abs/1503.02531, , ..

N Jiang, , , LH Li. . Doubly robust off-policy value evaluation for reinforcement learning. . Proc 33rd Int Conf on Machine Learning, , 2016. . p. 652--661. . ..

AEW Johnson, , , TJ Pollard, , , L Shen, , , 等. . MIMIC-Ⅲ, a freely accessible critical care database. . Sci Data, , 2016. . 3160035DOI:10.1038/sdata.2016.35http://doi.org/10.1038/sdata.2016.35..

LP Kaelbling, , , ML Littman, , , AW Moore. . An introduction to reinforcement learning. . In: Steels L (Ed. ), The Biology and Technology of Intelligent Autonomous Agents. Springer, Berlin, , 1995. . p. 90--127. . DOI:10.1007/978-3-642-79629-6_5http://doi.org/10.1007/978-3-642-79629-6_5..

JL Katzman, , , U Shaham, , , A Cloninger, , , 等. . DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. . BMC Med Res Method, , 2018. . 18((1):):24DOI:10.1186/s12874-018-0482-1http://doi.org/10.1186/s12874-018-0482-1..

DP Kingma, , , M Welling. . Auto-encoding variational Bayes. . Int Conf on Learning Representations Ithacap, , 2014. . p. 1--14. . ..

DP Kingma, , , T Salimans, , , R Jozefowicz, , , 等. . Improved variational inference with inverse autoregressive flow. . Proc 30th Int Conf on Neural Information Processing Systems, , 2016. . p. 4743--4751. . ..

M Komorowski, , , LA Celi, , , O Badawi, , , 等. . The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. . Nat Med, , 2018. . 24((11):):1716--1720. . DOI:10.1038/s41591-018-0213-5http://doi.org/10.1038/s41591-018-0213-5..

Y Li, , , JB Zeng, , , SG Shan, , , 等. . Self-supervised representation learning from videos for facial action unit detection. . IEEE/CVF Conf on Computer Vision and Pattern Recognition, , 2019. . p. 10924--10933. . DOI:10.1109/CVPR.2019.01118http://doi.org/10.1109/CVPR.2019.01118..

M Long, , , Y Cao, , , J Wang, , , 等. . Learning transferable features with deep adaptation networks. . Int Conf on Machine Learning, , 2015. . p. 97--105. . ..

D Lopez-Paz, , , L Bottou, , , B Schlkopf, , , 等. . Unifying distillation and privileged information. . 2016. . https://arxiv.org/abs/1511.03643https://arxiv.org/abs/1511.03643, , ..

V Mnih, , , K Kavukcuoglu, , , D Silver, , , 等. . Playing Atari with deep reinforcement learning. . 2015. . https://arxiv.org/abs/1312.5602https://arxiv.org/abs/1312.5602, , ..

S Nemati, , , MM Ghassemi, , , GD Clifford. . Optimal medication dosing from suboptimal clinical examples: a deep reinforcement learning approach. . Proc 38th Annual Int Conf of the IEEE Engineering in Medicine and Biology Society, , 2016. . p. 2978--2981. . DOI:10.1109/EMBC.2016.7591355http://doi.org/10.1109/EMBC.2016.7591355..

XF Peng, , , Y Ding, , , D Wihl, , , 等. . Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. . American Medical Informatics Association$^ \circledR$ Annual Symp, , 2018. . p. 887--896. . ..

A Raghu, , , M Komorowski, , , I Ahmed, , , 等. . Deep reinforcement learning for sepsis treatment. . Proc 31st Conf on Neural Information Processing Systems, , 2017. . p. 1--9. . ..

A Raghu, , , M Komorowski, , , S Singh. . Model-based reinforcement learning for sepsis treatment. . 2018. . https://arxiv.org/abs/1811.09602https://arxiv.org/abs/1811.09602, , ..

S Saria. . Individualized sepsis treatment using reinforcement learning. . Nat Med, , 2018. . 24((11):):1641--1642. . DOI:10.1038/s41591-018-0253-xhttp://doi.org/10.1038/s41591-018-0253-x..

SM Shortreed, , , E Laber, , , DJ Lizotte, , , 等. . Informing sequential clinical decision-making through reinforcement learning: an empirical study. . Mach Learn, , 2011. . 84((1-2):):109--136. . DOI:10.1007/s10994-010-5229-0http://doi.org/10.1007/s10994-010-5229-0..

M Singer, , , CS Deutschman, , , CW Seymour, , , 等. . The third international consensus definitions for sepsis and septic shock (sepsis-3). . JAMA, , 2016. . 315((8):):801--810. . DOI:10.1001/jama.2016.0287http://doi.org/10.1001/jama.2016.0287..

V Vapnik, , , R Izmailov. . Learning using privileged information: similarity control and knowledge transfer. . J Mach Learn Res, , 2015. . 16((1):):2023--2049. . ..

C Vondrick, , , H Pirsiavash, , , A Torralba. . Anticipating visual representations from unlabeled video. . IEEE Conf on Computer Vision and Pattern Recognition, , 2016. . p. 98--106. . DOI:10.1109/CVPR.2016.18http://doi.org/10.1109/CVPR.2016.18..

L Wang, , , W Zhang, , , XF He, , , 等. . Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. . Proc 24th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining, , 2018. . p. 2447--2456. . DOI:10.1145/3219819.3219961http://doi.org/10.1145/3219819.3219961..

ZY Wang, , , T Schaul, , , M Hessel, , , 等. . Dueling network architectures for deep reinforcement learning. . Proc 33rd Int Conf on Machine Learning, , 2016. . p. 1995--2003. . ..

WH Weng, , , MW Gao, , , Z He, , , 等. . Representation and reinforcement learning for personalized glycemic control in septic patients. . Proc 31st Conf on Neural Information Processing Systems, , 2017. . p. 1--5. . ..

C Yu, , , JM Liu, , , S Nemati. . Reinforcement learning in healthcare: a survey. . 2019. . https://arxiv.org/abs/1908.08796https://arxiv.org/abs/1908.08796, , ..

XH Zhai, , , A Oliver, , , A Kolesnikov, , , 等. . S4L: self-supervised semi-supervised learning. . IEEE/CVF Int Conf on Computer Vision, , 2019. . 1476--p. 1485. . DOI:10.1109/ICCV.2019.00156http://doi.org/10.1109/ICCV.2019.00156..

YT Zhang, , , R Chen, , , J Tang, , , 等. . LEAP: learning to prescribe effective and safe treatment combinations for multimorbidity. . Proc 23rd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, , 2017. . p. 1315--1324. . DOI:10.1145/3097983.3098109http://doi.org/10.1145/3097983.3098109..

SJ Zhao, , , JM Song, , , S Ermon. . InfoVAE: information maximizing variational autoencoders. . 2017. . https://arxiv.org/abs/1706.02262https://arxiv.org/abs/1706.02262, , ..

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Multi-agent reinforcement learning behavioral control for nonlinear second-order systems

An anti-collision algorithm for robotic search-and-rescue tasks in unknown dynamic environments

Embedding expert demonstrations into clustering buffer for effective deep reinforcement learning

A home energy management approach using decoupling value and policy in reinforcement learning