Multi-focus image fusion based on fully convolutional networks

Rui GUO; Xuan-jing SHEN; Xiao-yu DONG; Xiao-li ZHANG

doi:10.1631/FITEE.1900336

Your Location：

Home >

Browse articles >

Multi-focus image fusion based on fully convolutional networks

Regular Papers | Updated：2022-05-19

- Multi-focus image fusion based on fully convolutional networks
- 基于全卷积网络的多焦距图像融合算法
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 7, Pages: 1019-1033(2020)
- Affiliations：
  
  Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China
  College of Computer Science and Technology, Jilin University, Changchun 130012, China
- Author bio：
  
  Xiao-li ZHANG, E-mail: zhangxiaoli@jlu.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900336
  CLC： TP37
- Received：07 July 2019，
  
  Revised：2020-;6-10，
  
  Published：2020-07
- Accepted：
Scan QR Code
Rui GUO, Xuan-jing SHEN, Xiao-yu DONG, et al. Multi-focus image fusion based on fully convolutional networks[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(7): 1019-1033.
DOI：

Rui GUO, Xuan-jing SHEN, Xiao-yu DONG, et al. Multi-focus image fusion based on fully convolutional networks[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(7): 1019-1033. DOI： 10.1631/FITEE.1900336.

摘要

提出一种多焦距图像融合方法，在该算法中构造用于焦点检测的全卷积网络（fully convolutional network for focus detection，FD-FCN）。为获得更精确的焦点检测图谱，在该网络中添加跳层，从而在生成图谱过程中同时提供详细和抽象的视觉信息。基于数据集CIFAR-10，为该网络构建一个新的训练数据集。运用FD-FCN的图像融合算法包含3个步骤：使用FD-FCN获得焦点图谱，通过对焦点图谱进行形态学处理生成决策图，使用决策图进行图像融合。开展了多组实验，主客观评估结果均表明该融合方法优于同类先进算法。

Abstract

We propose a multi-focus image fusion method

in which a fully convolutional network for focus detection (FD-FCN) is constructed. To obtain more precise focus detection maps

we propose to add skip layers in the network to make both detailed and abstract visual information available when using FD-FCN to generate maps. A new training dataset for the proposed network is constructed based on dataset CIFAR-10. The image fusion algorithm using FD-FCN contains three steps: focus maps are obtained using FD-FCN

decision map generation occurs by applying a morphological process on the focus maps

and image fusion occurs using a decision map. We carry out several sets of experiments

and both subjective and objective assessments demonstrate the superiority of the proposed fusion method to state-of-the-art algorithms.

关键词

Keywords

references

M Amin-Naji , , , A Aghagolzadeh , , , M Ezoji . . Ensemble of CNN for multi-focus image fusion . . Inform Fus , , 2019 . . 51 201 - - 214 . . DOI: 10.1016/j.inffus.2019.02.003 http://doi.org/10.1016/j.inffus.2019.02.003 . .

V Aslantas , , , R Kurban . . Fusion of multi-focus images using differential evolution algorithm . . Expert Syst Appl , , 2010 . . 37 ( ( 12 ): ): 8861 - - 8870 . . DOI: 10.1016/j.eswa.2010.06.011 http://doi.org/10.1016/j.eswa.2010.06.011 . .

KR Ayyalasomayajula , , , F Malmberg , , , A Brun . . PDNet: semantic segmentation integrated with a primal-dual network for document binarization . . Patt Recogn Lett , , 2019 . . 121 52 - - 60 . . DOI: 10.1016/j.patrec.2018.05.011 http://doi.org/10.1016/j.patrec.2018.05.011 . .

XZ Bai , , , Y Zhang , , , FG Zhou , , , 等 . . Quadtree-based multi-focus image fusion using a weighted focus-measure . . Inform Fus , , 2015 . . 22 105 - - 118 . . DOI: 10.1016/j.inffus.2014.05.003 http://doi.org/10.1016/j.inffus.2014.05.003 . .

DP Bavirisetti , , , R Dhuli . . Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform . . IEEE Sens J , , 2016 . . 16 ( ( 1 ): ): 203 - - 209 . . DOI: 10.1109/JSEN.2015.2478655 http://doi.org/10.1109/JSEN.2015.2478655 . .

Y Chen , , , Z Qin . . Gradient-based compressive image fusion . . Front Inform Technol Electron Eng , , 2015 . . 16 ( ( 3 ): ): 227 - - 237 . . DOI: 10.1631/FITEE.1400217 http://doi.org/10.1631/FITEE.1400217 . .

Y Han , , , YZ Cai , , , Y Cao , , , 等 . . A new image fusion performance metric based on visual information fidelity . . Inform Fus , , 2013 . . 14 ( ( 2 ): ): 127 - - 135 . . DOI: 10.1016/j.inffus.2011.08.002 http://doi.org/10.1016/j.inffus.2011.08.002 . .

KM He , , , J Sun , , , XO Tang . . Guided image filtering . . IEEE Trans Patt Anal Mach Intell , , 2013 . . 35 ( ( 6 ): ): 1397 - - 1409 . . DOI: 10.1109/TPAMI.2012.213 http://doi.org/10.1109/TPAMI.2012.213 . .

W Huang , , , ZL Jing . . Multi-focus image fusion using pulse coupled neural network . . Patt Recogn Lett , , 2007 . . 28 ( ( 9 ): ): 1123 - - 1132 . . DOI: 10.1016/j.patrec.2007.01.013 http://doi.org/10.1016/j.patrec.2007.01.013 . .

B Huhle , , , T Schairer , , , P Jenke , , , 等 . . Fusion of range and color images for denoising and resolution enhancement with a non-local filter . . Comput Vis Image Understand , , 2010 . . 114 ( ( 12 ): ): 1336 - - 1345 . . DOI: 10.1016/j.cviu.2009.11.004 http://doi.org/10.1016/j.cviu.2009.11.004 . .

L Juočas , , , V Raudonis , , , R Maskeliūnas , , , 等 . . Multi- focusing algorithm for microscopy imagery in assembly line using low-cost camera . . Int J Adv Manuf Technol , , 2019 . . 102 ( ( 9-12 ): ): 3217 - - 3227 . . DOI: 10.1007/s00170-019-03407-9 http://doi.org/10.1007/s00170-019-03407-9 . .

KI Kim , , , Y Kwon . . Single-image super-resolution using sparse regression and natural image prior . . IEEE Trans Patt Anal Mach Intell , , 2010 . . 32 ( ( 6 ): ): 1127 - - 1133 . . DOI: 10.1109/TPAMI.2010.25 http://doi.org/10.1109/TPAMI.2010.25 . .

Y LeCun , , , Y Bengio . . Convolutional networks for images, speech, and time-series . . In: Arbib MA (Ed.), The Handbook of Brain Theory and Neural Networks. MIT Press, Cambridge , , 1995 . . p.255 - - 258 . . . .

ST Li , , , JT Kwok , , , YN Wang . . Multifocus image fusion using artificial neural networks . . Patt Recogn Lett , , 2002 . . 23 ( ( 8 ): ): 985 - - 997 . . DOI: 10.1016/s0167-8655(02)00029-6 http://doi.org/10.1016/s0167-8655(02)00029-6 . .

ST Li , , , XD Kang , , , JW Hu . . Image fusion with guided filtering . . IEEE Trans Image Process , , 2013a . . 22 ( ( 7 ): ): 2864 - - 2875 . . DOI: 10.1109/TIP.2013.2244222 http://doi.org/10.1109/TIP.2013.2244222 . .

ST Li , , , XD Kang , , , JW Hu , , , 等 . . Image matting for fusion of multi-focus images in dynamic scenes . . Inform Fus , , 2013b . . 14 ( ( 2 ): ): 147 - - 162 . . DOI: 10.1016/j.inffus.2011.07.001 http://doi.org/10.1016/j.inffus.2011.07.001 . .

ST Li , , , XD Kang , , , LY Fang , , , 等 . . Pixel-level image fusion: a survey of the state of the art . . Inform Fus , , 2017 . . 33 100 - - 112 . . DOI: 10.1016/j.inffus.2016.05.004 http://doi.org/10.1016/j.inffus.2016.05.004 . .

Y Liu , , , X Chen , , , H Peng , , , 等 . . Multi-focus image fusion with a deep convolutional neural network . . Inform Fus , , 2017 . . 36 191 - - 207 . . DOI: 10.1016/j.inffus.2016.12.001 http://doi.org/10.1016/j.inffus.2016.12.001 . .

HT Mustafa , , , J Yang , , , M Zareapoor . . Multi-scale convolutional neural network for multi-focus image fusion . . Image Vis Comput , , 2019 . . 85 26 - - 35 . . DOI: 10.1016/j.imavis.2019.03.001 http://doi.org/10.1016/j.imavis.2019.03.001 . .

G Piella , , , H Heijmans . . A new quality metric for image fusion . . Proc Int Conf on Image Processing , , 2003 . . p.173 - - 176 . . DOI: 10.1109/ICIP.2003.1247209 http://doi.org/10.1109/ICIP.2003.1247209 . .

GH Qu , , , DL Zhang , , , PF Yan . . Information measure for performance of image fusion . . Electron Lett , , 2002 . . 38 ( ( 7 ): ): 313 - - 315 . . DOI: 10.1049/el:20020212 http://doi.org/10.1049/el:20020212 . .

WQ Ren , , , JG Zhang , , , XY Xu , , , 等 . . Deep video dehazing with semantic segmentation . . IEEE Trans Image Process , , 2019 . . 28 ( ( 4 ): ): 1895 - - 1908 . . DOI: 10.1109/TIP.2018.2876178 http://doi.org/10.1109/TIP.2018.2876178 . .

A Saha , , , G Bhatnagar , , , QMJ Wu . . Mutual spectral residual approach for multifocus image fusion . . Dig Signal Process , , 2013 . . 23 ( ( 4 ): ): 1121 - - 1135 . . DOI: 10.1016/j.dsp.2013.03.001 http://doi.org/10.1016/j.dsp.2013.03.001 . .

A Saleem , , , A Beghdadi , , , B Boashash . . Image quality metrics based multifocus image fusion . . 3 rd European Workshop on Visual Information Processing , , 2011 . . p.77 - - 82 . . DOI: 10.1109/EuVIP.2011.6045547 http://doi.org/10.1109/EuVIP.2011.6045547 . .

E Shelhamer , , , J Long , , , T Darrell . . Fully convolutional networks for semantic segmentation . . IEEE Trans Patt Anal Mach Intell , , 2017 . . 39 ( ( 4 ): ): 640 - - 651 . . DOI: 10.1109/TPAMI.2016.2572683 http://doi.org/10.1109/TPAMI.2016.2572683 . .

B Wang , , , XY Yuan , , , XB Gao , , , 等 . . A hybrid level set with semantic shape constraint for object segmentation . . IEEE Trans Cybern , , 2019 . . 49 ( ( 5 ): ): 1558 - - 1569 . . DOI: 10.1109/TCYB.2018.2799999 http://doi.org/10.1109/TCYB.2018.2799999 . .

CS Xydeas , , , V Petrovic . . Objective image fusion performance measure . . Electron Lett , , 2000 . . 36 ( ( 4 ): ): 308 - - 309 . . DOI: 10.1049/el:20000267 http://doi.org/10.1049/el:20000267 . .

Y Yang , , , SY Huang , , , JF Gao , , , 等 . . Multi-focus image fusion using an effective discrete wavelet transform based algorithm . . Meas Sci Rev , , 2014 . . 14 ( ( 2 ): ): 102 - - 108 . . DOI: 10.2478/msr-2014-0014 http://doi.org/10.2478/msr-2014-0014 . .

Y Yang , , , S Tong , , , SY Huang , , , 等 . . Multifocus image fusion based on NSCT and focused area detection . . IEEE Sens J , , 2015 . . 15 ( ( 5 ): ): 2824 - - 2838 . . DOI: 10.1109/JSEN.2014.2380153 http://doi.org/10.1109/JSEN.2014.2380153 . .

BH Zhang , , , XQ Lu , , , WT Jia . . A multi-focus image fusion algorithm based on an improved dual-channel PCNN in NSCT domain . . Optik , , 2013 . . 124 ( ( 20 ): ): 4104 - - 4109 . . DOI: 10.1016/j.ijleo.2012.12.032 http://doi.org/10.1016/j.ijleo.2012.12.032 . .

Q Zhang , , , MD Levine . . Robust multi-focus image fusion using multi-task sparse representation and spatial context . . IEEE Trans Image Process , , 2016 . . 25 ( ( 5 ): ): 2045 - - 2058 . . DOI: 10.1109/TIP.2016.2524212 http://doi.org/10.1109/TIP.2016.2524212 . .

WD Zhao , , , D Wang , , , HC Lu . . Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network . . IEEE Trans Circ Syst Video Technol , , 2019 . . 29 ( ( 4 ): ): 1102 - - 1115 . . DOI: 10.1109/TCSVT.2018.2821177 http://doi.org/10.1109/TCSVT.2018.2821177 . .

ZQ Zhou , , , S Li , , , B Wang . . Multi-scale weighted gradient- based fusion for multi-focus images . . Inform Fus , , 2014 . . 20 60 - - 72 . . DOI: 10.1016/j.inffus.2013.11.005 http://doi.org/10.1016/j.inffus.2013.11.005 . .

XB Zhu , , , XM Zhang , , , XY Zhang , , , 等 . . A novel framework for semantic segmentation with generative adversarial network . . J Vis Commun Image Represent , , 2019 . . 58 532 - - 543 . . DOI: 10.1016/j.jvcir.2018.11.020 http://doi.org/10.1016/j.jvcir.2018.11.020 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰