MPIN: a macro-pixel integration network for light field super-resolution

WANG Xinya; MA Jiayi; GAO Wenjing; JIANG Junjun

doi:10.1631/FITEE.2000566

Your Location：

Home >

Browse articles >

MPIN: a macro-pixel integration network for light field super-resolution

Regular Papers | Updated：2022-01-05

- MPIN: a macro-pixel integration network for light field super-resolution
- MPIN：基于宏像素聚合的光场图像超分辨率网络
- Frontiers of Information Technology & Electronic Engineering Vol. 22, Issue 10, Pages: 1299-1310(2021)
- Affiliations：
  
  Electronic Information School, Wuhan University, Wuhan 430072, China
  School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China
- Author bio：
  
  [ "", "Xinya WANG, first author of this invited paper, received her BS degree from the Electronic Information School, Wuhan University, Wuhan, China, in 2018. She is currently pursuing her PhD degree with the Electronic Information School, Wuhan University. Her research interests include neural networks, machine learning, and image processing. E-mail: wangxinya@whu.edu.cn" ]
  [ "", "Jiayi MA, corresponding author of this invited paper, received his BS degree in information and computing science and PhD degree in control science and engineering from Huazhong University of Science and Technology, Wuhan, China, in 2008 and 2014, respectively. He is currently a professor with the Electronic Information School, Wuhan University. He has authored or co-authored more than 200 refereed journal and conference papers. His research interests include computer vision, machine learning, and robotics. Dr. Ma has been identified in the 2020 and 2019 Highly Cited Researcher lists from Web of Science. He is an area editor of Information Fusion, an associate editor of Neurocomputing, Sensors, and Entropy, and a guest editor of Remote Sensing" ]
  [ "Wenjing GAO, E-mail: wenjinggao@whu.edu.cn" ]
  [ "Junjun JIANG, E-mail: junjun0595@163.com" ]
- Funds：
- DOI：10.1631/FITEE.2000566
  CLC： TP312
- Received：20 October 2020，
  
  Revised：2021-;8-;3，
  
  Accepted：26 March 2021，
  
  Published：2021-10
- Accepted：
Scan QR Code
WANG Xinya, MA Jiayi, GAO Wenjing, et al. MPIN: a macro-pixel integration network for light field super-resolution[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(10): 1299-1310.
DOI：

WANG Xinya, MA Jiayi, GAO Wenjing, et al. MPIN: a macro-pixel integration network for light field super-resolution[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(10): 1299-1310. DOI： 10.1631/FITEE.2000566.

摘要

现有的大多数光场超分辨率方法不能充分利用角度信息，或者由于利用部分视图而产生不均衡的性能。为解决这些问题，本文提出一种基于宏像素表示的光场图像超分辨率聚合网络模型（称为MPIN）。该网络通过将四维光场图像重新排列成二维宏像素图像，将空间和角度信息进行耦合，从而同时恢复整张光场图像。网络利用两种特殊的卷积分别提取空间和角度信息。为充分利用空间-角度相关性，所设计的聚合残差模块融合两种信息使其相互引导，以实现角度相干性。在宏像素表示下，该网络通过扩展角度混洗层来提高宏像素图像的空间分辨率，有效避免了混叠。在合成和真实光场数据集上的大量实验表明，本文提出的方法在定性和定量上均实现了比现有方法更好的性能。此外，该方法在保持光场图像固有极线结构的同时，具有均衡性能分布的优点。

Abstract

Most existing light field (LF) super-resolution (SR) methods either fail to fully use angular information or have an unbalanced performance distribution because they use parts of views. To address these issues

we propose a novel integration network based on macro-pixel representation for the LF SR task

named MPIN. Restoring the entire LF image simultaneously

we couple the spatial and angular information by rearranging the four-dimensional LF image into a two-dimensional macro-pixel image. Then

two special convolutions are deployed to extract spatial and angular information

separately. To fully exploit spatial-angular correlations

the integration resblock is designed to merge the two kinds of information for mutual guidance

allowing our method to be angular-coherent. Under the macro-pixel representation

an angular shuffle layer is tailored to improve the spatial resolution of the macro-pixel image

which can effectively avoid aliasing. Extensive experiments on both synthetic and real-world LF datasets demonstrate that our method can achieve better performance than the state-of-the-art methods qualitatively and quantitatively. Moreover

the proposed method has an advantage in preserving the inherent epipolar structures of LF images with a balanced distribution of performance.

关键词

Keywords

references

M Alain , , , A Smolic . . Light field super-resolution via LFBM5D sparse coding . . Proc 25 th IEEE Int Conf on Image Processing , , 2018 . . p.2501 - - 2505 . . DOI: 10.1109/ICIP.2018.8451162 http://doi.org/10.1109/ICIP.2018.8451162 . .

TE Bishop , , , P Favaro . . The light field camera: extended depth of field, aliasing, and superresolution . . IEEE Trans Patt Anal Mach Intell , , 2012 . . 34 ( ( 5 ): ): 972 - - 986 . . DOI: 10.1109/TPAMI.2011.168 http://doi.org/10.1109/TPAMI.2011.168 . .

TE Bishop , , , S Zanetti , , , P Favaro . . Light field super-resolution . . Proc IEEE Int Conf on Computational Photography , , 2009 . . p. 1 - - 9 . . DOI: 10.1109/ICCPHOT.2009.5559010 http://doi.org/10.1109/ICCPHOT.2009.5559010 . .

C Dong , , , CC Loy , , , KM He , , , 等 . . Learning a deep convolutional network for image super-resolution . . Proc 13 th European Conf on Computer Vision , , 2014 . . p. 184 - - 199 . . DOI: 10.1007/978-3-319-10593-2_13 http://doi.org/10.1007/978-3-319-10593-2_13 . .

HZ Fan , , , D Liu , , , ZW Xiong , , , 等 . . Two-stage convolutional neural network for light field super-resolution . . Proc IEEE Int Conf on Image Processing , , 2017 . . p. 1167 - - 1171 . . DOI: 10.1109/ICIP.2017.8296465 http://doi.org/10.1109/ICIP.2017.8296465 . .

RA Farrugia , , , C Galea , , , C Guillemot . . Super resolution of light field images using linear subspace projection of patch-volumes . . IEEE J Sel Top Signal Process , , 2017 . . 11 ( ( 7 ): ): 1058 - - 1071 . . DOI: 10.1109/JSTSP.2017.2747127 http://doi.org/10.1109/JSTSP.2017.2747127 . .

X Glorot , , , Y Bengio . . Understanding the difficulty of training deep feedforward neural networks . . Proc 13 th Int Conf on Artificial Intelligence and Statistics , , 2010 . . p. 249 - - 256 . . . .

MSK Gul , , , BK Gunturk . . Spatial and angular resolution enhancement of light fields using convolutional neural networks . . IEEE Trans Image Process , , 2018 . . 27 ( ( 5 ): ): 2146 - - 2159 . . DOI: 10.1109/TIP.2018.2794181 http://doi.org/10.1109/TIP.2018.2794181 . .

K Honauer , , , O Johannsen , , , D Kondermann , , , 等 . . A dataset and evaluation methodology for depth estimation on 4D light fields . . Proc 13 th Asian Conf on Computer Vision , , 2016 . . p. 19 - - 34 . . DOI: 10.1007/978-3-319-54187-7_2 http://doi.org/10.1007/978-3-319-54187-7_2 . .

J Jin , , , JH Hou , , , J Chen , , , 等 . . Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2020 . . p. 2257 - - 2266 . . DOI: 10.1109/CVPR42600.2020.00233 http://doi.org/10.1109/CVPR42600.2020.00233 . .

NK Kalantari , , , TC Wang , , , R Ramamoorthi . . Learning-based view synthesis for light field cameras . . ACM Trans Graph , , 2016 . . 35 ( ( 6 ): ): 193 DOI: 10.1145/2980179.2980251 http://doi.org/10.1145/2980179.2980251 . .

M Levoy , , , P Hanrahan . . Light field rendering . . Proc 23 rd Annual Conf on Computer Graphics and Interactive Techniques , , 1996 . . p. 31 - - 42 . . DOI: 10.1145/237170.237199 http://doi.org/10.1145/237170.237199 . .

M Li , , , CY Diao , , , DQ Xu , , , 等 . . A non-Lambertian photometric stereo under perspective projection . . Front Inform Technol Electron Eng , , 2020 . . 21 ( ( 8 ): ): 1191 - - 1205 . . DOI: 10.1631/FITEE.1900156 http://doi.org/10.1631/FITEE.1900156 . .

CK Liang , , , R Ramamoorthi . . A light transport framework for lenslet light field cameras . . ACM Trans Graph , , 2015 . . 34 ( ( 2 ): ): 16 DOI: 10.1145/2665075 http://doi.org/10.1145/2665075 . .

B Lim , , , S Son , , , H Kim , , , 等 . . Enhanced deep residual networks for single image super-resolution . . Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops , , 2017 . . p. 1132 - - 1140 . . DOI: 10.1109/CVPRW.2017.151 http://doi.org/10.1109/CVPRW.2017.151 . .

J Lim , , , H Ok , , , B Park , , , 等 . . Improving the spatail resolution based on 4D light field data . . Proc 16 th IEEE Int Conf on Image Processing , , 2009 . . p. 1173 - - 1176 . . DOI: 10.1109/ICIP.2009.5413719 http://doi.org/10.1109/ICIP.2009.5413719 . .

K Mitra , , , A Veeraraghavan . . Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior . . Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition Workshops , , 2012 . . p. 22 - - 28 . . DOI: 10.1109/CVPRW.2012.6239346 http://doi.org/10.1109/CVPRW.2012.6239346 . .

FP Nava , , , JP Luke . . Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera . . Proc 3DTV Conf: the True Vision——Capture, Transmission and Display of 3D Video , , 2009 . . p. 1 - - 4 . . DOI: 10.1109/3DTV.2009.5069675 http://doi.org/10.1109/3DTV.2009.5069675 . .

R Ng , , , M Levoy , , , M Brdif , , , 等 . . Light Field Photography with a Hand-Held Plenoptic Camera . . Technical Report No. CTSR 2005-02, Stanford University, USA , , 2005 . . .

JY Peng , , , ZW Xiong , , , D Liu , , , 等 . . Unsupervised depth estimation from light field using a convolutional neural network . . Proc Int Conf on 3D Vision , , 2018 . . p.295 - - 303 . . DOI: 10.1109/3DV.2018.00042 http://doi.org/10.1109/3DV.2018.00042 . .

F Prez , , , A Prez , , , M Rodrguez , , , 等 . . Fourier slice super-resolution in plenoptic cameras . . Proc IEEE Int Conf on Computational Photography , , 2012 . . p.1 - - 11 . . DOI: 10.1109/ICCPhot.2012.6215210 http://doi.org/10.1109/ICCPhot.2012.6215210 . .

F Prez , , , A Prez , , , M Rodrguez , , , 等 . . Super-resolved Fourier-slice refocusing in plenoptic cameras . . J Math Imag Vis , , 2015 . . 52 ( ( 2 ): ): 200 - - 217 . . DOI: 10.1007/s10851-014-0540-1 http://doi.org/10.1007/s10851-014-0540-1 . .

M Rerabek , , , T Ebrahimi . . New light field image dataset . . Proc 8 th Int Conf on Quality of Multimedia Experience , , 2016 . . p.1 - - 2 . . DOI: 10.5281/zenodo.209499 http://doi.org/10.5281/zenodo.209499 . .

M Rossi , , , P Frossard . . Graph-based light field super-resolution . . Proc IEEE 19 th Int Workshop on Multimedia Signal Processing , , 2017 . . p.1 - - 6 . . DOI: 10.1109/MMSP.2017.8122224 http://doi.org/10.1109/MMSP.2017.8122224 . .

M Rossi , , , P Frossard . . Geometry-consistent light field super-resolution via graph-based regularization . . IEEE Trans Image Process , , 2018 . . 27 ( ( 9 ): ): 4207 - - 4218 . . DOI: 10.1109/TIP.2018.2828983 http://doi.org/10.1109/TIP.2018.2828983 . .

WZ Shi , , , J Caballero , , , F Huszr , , , 等 . . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.1874 - - 1883 . . DOI: 10.1109/CVPR.2016.207 http://doi.org/10.1109/CVPR.2016.207 . .

TC Wang , , , JY Zhu , , , E Hiroaki , , , 等 . . A 4D light-field dataset and CNN architectures for material recognition . . Proc 14 th European Conf on Computer Vision , , 2016 . . p.121 - - 138 . . DOI: 10.1007/978-3-319-46487-9_8 http://doi.org/10.1007/978-3-319-46487-9_8 . .

YL Wang , , , GQ Hou , , , ZN Sun , , , 等 . . A simple and robust super resolution method for light field images . . Proc IEEE Int Conf on Image Processing , , 2016 . . p.1459 - - 1463 . . DOI: 10.1109/ICIP.2016.7532600 http://doi.org/10.1109/ICIP.2016.7532600 . .

YL Wang , , , F Liu , , , KB Zhang , , , 等 . . LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution . . IEEE Trans Image Process , , 2018 . . 27 ( ( 9 ): ): 4274 - - 4286 . . DOI: 10.1109/TIP.2018.2834819 http://doi.org/10.1109/TIP.2018.2834819 . .

YQ Wang , , , LG Wang , , , JG Yang , , , 等 . . Spatial-angular interaction for light field image super-resolution . . Proc 16 th European Conf on Computer Vision , , 2020 . . p.290 - - 308 . . DOI: 10.1007/978-3-030-58592-1_18 http://doi.org/10.1007/978-3-030-58592-1_18 . .

S Wanner , , , B Goldluecke . . Variational light field analysis for disparity estimation and super-resolution . . IEEE Trans Patt Anal Mach Intell , , 2014 . . 36 ( ( 3 ): ): 606 - - 619 . . DOI: 10.1109/TPAMI.2013.147 http://doi.org/10.1109/TPAMI.2013.147 . .

S Wanner , , , S Meister , , , B Goldluecke . . Datasets and benchmarks for densely sampled 4D light fields . . Proc 18 th Int Workshop on Vision, Modeling, and Visualization , , 2013 . . p.225 - - 226 . . DOI: 10.2312/PE.VMV.VMV13.225-226 http://doi.org/10.2312/PE.VMV.VMV13.225-226 . .

HWF Yeung , , , JH Hou , , , XM Chen , , , 等 . . Light field spatial super-resolution using deep efficient spatial-angular separable convolution . . IEEE Trans Image Process , , 2019 . . 28 ( ( 5 ): ): 2319 - - 2330 . . DOI: 10.1109/TIP.2018.2885236 http://doi.org/10.1109/TIP.2018.2885236 . .

P Yi , , , ZY Wang , , , K Jiang , , , 等 . . Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations . . Proc IEEE/CVF Int Conf on Computer Vision , , 2019 . . p.3106 - - 3115 . . DOI: 10.1109/ICCV.2019.00320 http://doi.org/10.1109/ICCV.2019.00320 . .

Y Yoon , , , HG Jeon , , , D Yoo , , , 等 . . Learning a deep convolutional network for light-field image super-resolution . . Proc IEEE Int Conf on Computer Vision Workshop , , 2015 . . p.57 - - 65 . . DOI: 10.1109/ICCVW.2015.17 http://doi.org/10.1109/ICCVW.2015.17 . .

Y Yuan , , , ZQ Cao , , , LJ Su . . Light-field image superresolution using a combined deep CNN based on EPI . . IEEE Signal Process Lett , , 2018 . . 25 ( ( 9 ): ): 1359 - - 1363 . . DOI: 10.1109/LSP.2018.2856619 http://doi.org/10.1109/LSP.2018.2856619 . .

K Ycer , , , A Sorkine-Hornung , , , O Wang , , , 等 . . Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction . . ACM Trans Graph , , 2016 . . 35 ( ( 3 ): ): 22 DOI: 10.1145/2876504 http://doi.org/10.1145/2876504 . .

S Zhang , , , YF Lin , , , H Sheng . . Residual networks for light field image super-resolution . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2019 . . p.11038 - - 11047 . . DOI: 10.1109/CVPR.2019.01130 http://doi.org/10.1109/CVPR.2019.01130 . .

H Zhu , , , Q Wang , , , JY Yu . . Light field imaging: models, calibrations, reconstructions, and applications . . Front Inform Technol Electron Eng , , 2017 . . 18 ( ( 9 ): ): 1236 - - 1249 . . DOI: 10.1631/FITEE.1601727 http://doi.org/10.1631/FITEE.1601727 . .

Views

190

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰