Interactive medical image segmentation with self-adaptive confidence calibration

Chuyun SHEN; Wenhao LI; Qisen XU; Bin HU; Bo JIN; Haibin CAI; Fengping ZHU; Yuxin LI; Xiangfeng WANG

doi:10.1631/FITEE.2200299

Your Location：

Home >

Browse articles >

Interactive medical image segmentation with self-adaptive confidence calibration

Regular Papers | Updated：2023-09-21

- Interactive medical image segmentation with self-adaptive confidence calibration
- 基于自适应置信度校准的交互式医疗图像分割框架
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 9, Pages: 1332-1348(2023)
- Affiliations：
  
  1.School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
  2.Huashan Hospital, Fudan University, Shanghai 200040, China
  3.Software Engineering Institute, East China Normal University, Shanghai 200062, China
- Author bio：
  
  †E-mail: cyshen@stu.ecnu.edu.cn
  52194501026@stu.ecnu.edu.cn
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2200299
  CLC： TP391.4
- Received：13 July 2022，
  
  Accepted：20 February 2023，
  
  Published：0 September 2023
- Accepted：
Scan QR Code
Chuyun SHEN, Wenhao LI, Qisen XU, et al. Interactive medical image segmentation with self-adaptive confidence calibration[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(9): 1332-1348.
DOI：

Chuyun SHEN, Wenhao LI, Qisen XU, et al. Interactive medical image segmentation with self-adaptive confidence calibration[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(9): 1332-1348. DOI： 10.1631/FITEE.2200299.

摘要

基于人机交互的医疗图像分割方法是一种新的范式，其通过引入专家交互信息来指导算法完成图像分割任务。然而，现有医疗图像分割模型往往容易产生“交互误解”，即无法合理权衡短期和长期交互信息的重要性。为更好地利用不同时间尺度上的交互信息，本文提出一种基于自适应置信度校准的交互式医疗图像分割框架MECCA，其结合了基于分割决策的置信度学习技术和多智能体强化学习技术，并通过预测分割决策与短期交互信息的对齐水平来学习一个新颖的置信度网络。随后，提出一种基于置信度的奖励塑造机制，在策略梯度计算中引入置信度，从而直接纠正模型产生的交互误解。MECCA还通过标签生成和交互指导来降低交互强度和难度，从而实现用户友好交互。实验结果表明，MECCA在不同分割任务中可以显著提高短期和长期交互信息的利用效率，且仅需较少的标注样本。演示视频可通过https://bit.ly/mecca-demo-video访问。

Abstract

Interactive medical image segmentation based on human-in-the-loop machine learning is a novel paradigm that draws on human expert knowledge to assist medical image segmentation. However

existing methods often fall into what we call interactive misunderstanding

the essence of which is the dilemma in trading off short- and long-term interaction information. To better use the interaction information at various timescales

we propose an interactive segmentation framework

called interactive MEdical image segmentation with self-adaptive Confidence CAlibration (MECCA)

which combines action-based confidence learning and multi-agent reinforcement learning. A novel confidence network is learned by predicting the alignment level of the action with short-term interaction information. A confidence-based reward-shaping mechanism is then proposed to explicitly incorporate confidence in the policy gradient calculation

thus directly correcting the model’s interactive misunderstanding. MECCA also enables user-friendly interactions by reducing the interaction intensity and difficulty via label generation and interaction guidance

respectively. Numerical experiments on different segmentation tasks show that MECCA can significantly improve short- and long-term interaction information utilization efficiency with remarkably fewer labeled samples. The demo video is available at https://bit.ly/mecca-demo-video.

关键词

Keywords

references

Abel D , Jinnai Y , Guo SY , et al. , 2018 . Policy and value transfer in lifelong reinforcement learning . Proc 35 th Int Conf on Machine Learning , p. 20 - 29 .

Achanta R , Shaji A , Smith K , et al. , 2012 . SLIC superpixels compared to state-of-the-art superpixel methods . IEEE Trans Patt Anal Mach Intell , 34 （ 11 ）: 2274 - 2282 . doi: 10.1109/TPAMI.2012.120 http://doi.org/10.1109/TPAMI.2012.120

Acuna D , Ling H , Kar A , et al. , 2018 . Efficient interactive annotation of segmentation datasets with Polygon-RNN++ . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 859 - 868 . doi: 10.1109/CVPR.2018.00096 http://doi.org/10.1109/CVPR.2018.00096

Aljabri M , AlAmir M , AlGhamdi M , et al. , 2022 . Towards a better understanding of annotation tools for medical imaging: a survey . Multim Tools Appl , 81 （ 18 ）: 25877 - 25911 . doi: 10.1007/s11042-022-12100-1 http://doi.org/10.1007/s11042-022-12100-1

Bredell G , Tanner C , Konukoglu E , 2018 . Iterative interaction training for segmentation editing networks . Proc 9 th Int Workshop on Machine Learning in Medical Imaging , p. 363 - 370 . doi: 10.1007/978-3-030-00919-9_42 http://doi.org/10.1007/978-3-030-00919-9_42

Castrejón L , Kundu K , Urtasun R , et al. , 2017 . Annotating object instances with a polygon-RNN . IEEE Conf on Computer Vision and Pattern Recognition , p. 4485 - 4493 . doi: 10.1109/CVPR.2017.477 http://doi.org/10.1109/CVPR.2017.477

DeVries T , Taylor GW , 2018a . Learning confidence for out-of-distribution detection in neural networks . doi: 10.48550/arXiv.1802.04865 http://doi.org/10.48550/arXiv.1802.04865

DeVries T , Taylor GW , 2018b . Leveraging uncertainty estimates for predicting segmentation quality . doi: 10.48550/arXiv.1807.00502 http://doi.org/10.48550/arXiv.1807.00502

Feng RW , Zheng XS , Gao TX , et al. , 2021 . Interactive few-shot learning: limited supervision, better medical image segmentation . IEEE Trans Med Imag , 40 （ 10 ）: 2575 - 2588 . doi: 10.1109/TMI.2021.3060551 http://doi.org/10.1109/TMI.2021.3060551

Furuta R , Inoue N , Yamasaki T , 2020 . PixelRL: fully convolutional network with reinforcement learning for image processing . IEEE Trans Multim , 22 （ 7 ）: 1704 - 1719 . doi: 10.1109/TMM.2019.2960636 http://doi.org/10.1109/TMM.2019.2960636

Glorot X , Bengio Y , 2010 . Understanding the difficulty of training deep feedforward neural networks . Proc 13 th Int Conf on Artificial Intelligence and Statistics , p. 249 - 256 .

Hung W , Tsai Y , Liou Y , et al. , 2018 . Adversarial learning for semi-supervised semantic segmentation . Proc British Machine Vision Conf , p. 65 .

Jungo A , Reyes M , 2019 . Assessing reliability and challenges of uncertainty estimations for medical image segmentation . Proc 22 nd Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 48 - 56 . doi: 10.1007/978-3-030-32245-8_6 http://doi.org/10.1007/978-3-030-32245-8_6

Kendall A , Gal Y , 2017 . What uncertainties do we need in bayesian deep learning for computer vision ? Proc 3 rd Int Conf on Neural Information Processing System , p. 5580 - 5590 .

Kingma DP , Ba J , 2015 . Adam: a method for stochastic optimization . Proc 3 rd Int Conf on Learning Representations . doi: 10.48550/arXiv.1412.6980 http://doi.org/10.48550/arXiv.1412.6980

Lee KM , Song G , 2018 . SeedNet: automatic seed generation with deep reinforcement learning for robust interactive segmentation . IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 1760 - 1768 . doi: 10.1109/CVPR.2018.00189 http://doi.org/10.1109/CVPR.2018.00189

Li L , Zimmer VA , Schnabel JA , et al. , 2021 . AtrialGeneral: domain generalization for left atrial segmentation of multi-center LGE MRIs . Proc 24 th Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 557 - 566 . doi: 10.1007/978-3-030-87231-1_54 http://doi.org/10.1007/978-3-030-87231-1_54

Liao X , Li WH , Xu QS , et al. , 2020 . Iteratively-refined interactive 3D medical image segmentation with multi-agent reinforcement learning . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 9394 - 9402 . doi: 10.1109/CVPR42600.2020.00941 http://doi.org/10.1109/CVPR42600.2020.00941

Lin D , Dai JF , Jia JY , et al. , 2016 . ScribbleSup: scribble-supervised convolutional networks for semantic segmentation . IEEE Conf on Computer Vision and Pattern Recognition , p. 3159 - 3167 . doi: 10.1109/CVPR.2016.344 http://doi.org/10.1109/CVPR.2016.344

Lin TY , Goyal P , Girshick R , et al. , 2017 . Focal loss for dense object detection . Proc IEEE Int Conf on Computer Vision , p. 2999 - 3007 . doi: 10.1109/ICCV.2017.324 http://doi.org/10.1109/ICCV.2017.324

Ma CF , Xu QS , Wang XF , et al. , 2021 . Boundary-aware supervoxel-level iteratively refined interactive 3D image segmentation with multi-agent reinforcement learning . IEEE Trans Med Imag , 40 （ 10 ）: 2563 - 2574 . doi: 10.1109/TMI.2020.3048477 http://doi.org/10.1109/TMI.2020.3048477

Menze BH , Jakab A , Bauer S , et al. , 2015 . The multimodal brain tumor image segmentation benchmark （BRATS） . IEEE Trans Med Imag , 34 （ 10 ）: 1993 - 2024 . doi: 10.1109/TMI.2014.2377694 http://doi.org/10.1109/TMI.2014.2377694

Mnih V , Badia AP , Mirza M , et al. , 2016 . Asynchronous methods for deep reinforcement learning . Proc 33 rd Int Conf on Machine Learning , p. 1928 - 1937 .

Moeskops P , Veta M , Lafarge MW , et al. , 2017 . Adversarial training and dilated convolutions for brain MRI segmentation . Proc 3 rd Int Workshop on Deep Learning in Medical Image Analysis and 7 th Int Workshop on Multimodal Learning for Clinical Decision Support , p. 56 - 64 . doi: 10.1007/978-3-319-67558-9_7 http://doi.org/10.1007/978-3-319-67558-9_7

Nie D , Wang L , Xiang L , et al. , 2019 . Difficulty-aware attention network with confidence learning for medical image segmentation . Proc 33 rd AAAI Conf on Artificial Intelligence, 31 st Innovative Applications of Artificial Intelligence Conf, and 9 th AAAI Symp on Educational Advances in Artificial Intelligence , p. 1085 - 1092 . doi: 10.1609/aaai.v33i01.33011085 http://doi.org/10.1609/aaai.v33i01.33011085

Open AI , 2022 . ChatGPT: Optimizing Language Models for Dialogue . https://openai.casa/blog/chatgpt/ https://openai.casa/blog/chatgpt/ ［Accessed on July 10, 2022 ］.

Paszke A , Gross S , Massa F , et al. , 2019 . PyTorch: an imperative style, high-performance deep learning library . Proc 33 rd Int Conf on Neural Information Processing Systems , p. 8026 - 8037 .

Prabhu A , Torr PHS , Dokania PK , 2020 . GDumb: a simple approach that questions our progress in continual learning . Proc 16 th European Conf on Computer Vision , p. 524 - 540 . doi: 10.1007/978-3-030-58536-5_31 http://doi.org/10.1007/978-3-030-58536-5_31

Rajchl M , Lee MCH , Oktay O , et al. , 2017 . DeepCut: object segmentation from bounding box annotations using convolutional neural networks . IEEE Trans Med Imag , 36 （ 2 ）: 674 - 683 . doi: 10.1109/TMI.2016.2621185 http://doi.org/10.1109/TMI.2016.2621185

Rebuffi SA , Kolesnikov A , Sperl G , 2017 . iCaRL: incremental classifier and representation learning . IEEE Conf on Computer Vision and Pattern Recognition , p. 5533 - 5542 . doi: 10.1109/CVPR.2017.587 http://doi.org/10.1109/CVPR.2017.587

Robinson R , Oktay O , Bai WJ , et al. , 2018 . Real-time prediction of segmentation quality . Proc 21 st Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 578 - 585 . doi: 10.1007/978-3-030-00937-3_66 http://doi.org/10.1007/978-3-030-00937-3_66

Ronneberger O , Fischer P , Brox T , 2015 . U-Net: convolutional networks for biomedical image segmentation . Proc 18 th Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 234 - 241 . doi: 10.1007/978-3-319-24574-4_28 http://doi.org/10.1007/978-3-319-24574-4_28

Shrivastava A , Gupta A , Girshick R , 2016 . Training region-based object detectors with online hard example mining . IEEE Conf on Computer Vision and Pattern Recognition , p. 761 - 769 . doi: 10.1109/CVPR.2016.89 http://doi.org/10.1109/CVPR.2016.89

Simpson AL , Antonelli M , Bakas S , et al. , 2019 . A large annotated medical image dataset for the development and evaluation of segmentation algorithms . doi: 10.48550/arXiv.1902.09063 http://doi.org/10.48550/arXiv.1902.09063

Wang GT , Li WQ , Zuluaga MA , et al. , 2018 . Interactive medical image segmentation using deep learning with image-specific fine tuning . IEEE Trans Med Imag , 37 （ 7 ）: 1562 - 1573 . doi: 10.1109/TMI.2018.2791721 http://doi.org/10.1109/TMI.2018.2791721

Wang GT , Zuluaga MA , Li WQ , et al. , 2019 . DeepIGeoS: a deep interactive geodesic framework for medical image segmentation . IEEE Trans Patt Anal Mach Intell , 41 （ 7 ）: 1559 - 1572 . doi: 10.1109/TPAMI.2018.2840695 http://doi.org/10.1109/TPAMI.2018.2840695

Xie AN , Harrison J , Finn C , 2020 . Deep reinforcement learning amidst lifelong non-stationarity . doi: 10.48550/arXiv.2006.10701 http://doi.org/10.48550/arXiv.2006.10701

Xu N , Price B , Cohen S , et al. , 2016 . Deep interactive object selection . IEEE Conf on Computer Vision and Pattern Recognition , p. 373 - 381 . doi: 10.1109/CVPR.2016.47 http://doi.org/10.1109/CVPR.2016.47

Ye QH , Gao Y , Ding WP , et al. , 2022 . Robust weakly supervised learning for COVID-19 recognition using multi-center CT images . Appl Soft Comput , 116 : 108291 . doi: 10.1016/j.asoc.2021.108291 http://doi.org/10.1016/j.asoc.2021.108291

Yu LQ , Wang SJ , Li XM , et al. , 2019 . Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation . Proc 22 nd Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 605 - 613 . doi: 10.1007/978-3-030-32245-8_67 http://doi.org/10.1007/978-3-030-32245-8_67

Zhang KQ , Yang ZR , Basar T , 2021 . Decentralized multi-agent reinforcement learning with networked agents: recent advances . Front Inform Technol Electron Eng , 22 （ 6 ）: 802 - 814 . doi: 10.1631/FITEE.1900661 http://doi.org/10.1631/FITEE.1900661

Zhang SY , Liew JH , Wei YC , et al. , 2020 . Interactive object segmentation with inside-outside guidance . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 12231 - 12241 . doi: 10.1109/CVPR42600.2020.01225 http://doi.org/10.1109/CVPR42600.2020.01225

Zhuang XH , Shen J , 2016 . Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI . Med Image Anal , 31 : 77 - 87 . doi: 10.1016/j.media.2016.02.006 http://doi.org/10.1016/j.media.2016.02.006

Views

255

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Dynamic prompting class distribution optimization for semi-supervised sound event detection

Adaptive multi-layer deployment for a digital-twin-empowered satellite-terrestrial integrated network

A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space

NGAT: attention in breadth and depth exploration for semi-supervised graph representation learning

A local density optimization method based on a graph convolutional network

Related Author

Lijian GAO

Qing ZHU

Yaxin SHEN

Qirong MAO

Yongzhao ZHAN

Xing ZHANG

Jingkai CHEN

Haoyang SHI

Related Institution

Jiangsu Engineering Research Center of Big Data Ubiquitous Perception and Intelligent Agricultural Applications

School of Computer Science and Communication Engineering, Jiangsu University

Research Institute of China Telecom Co., Ltd.

Wireless Signal Processing and Network Laboratory, Beijing University of Posts and Telecommunications

College of Computer, National University of Defense Technology

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰