Deep learning based wavefront sensor for complex wavefront detection in adaptive optical microscopes

Shuwen HU; Lejia HU; Wei GONG; Zhenghan LI; Ke SI

doi:10.1631/FITEE.2000422

Your Location：

Home >

Browse articles >

Deep learning based wavefront sensor for complex wavefront detection in adaptive optical microscopes

Regular Papers | Updated：2022-06-06

- Deep learning based wavefront sensor for complex wavefront detection in adaptive optical microscopes
- 自适应光学显微中基于深度学习的复杂波前探测方法
- Frontiers of Information Technology & Electronic Engineering Vol. 22, Issue 10, Pages: 1277-1288(2021)
- Affiliations：
  
  Department of Neurology of the First Affiliated Hospital, State Key Laboratory of Modern Optical Instrumentation, Zhejiang University School of Medicine, Hangzhou 310009, China
  College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China
  Research Units for Emotion and Emotion Disorders, Chinese Academy of Medical Sciences, MOE Frontier Science Center for Brain Research and Brain-Machine Integration, School of Brain Science and Brain Medicine, Zhejiang University, Hangzhou 310058, China
- Author bio：
  
  [ "", "Shuwen HU, first author of this invited paper, received her BS degree from Tianjin University, China. She is currently a master degree candidate at the College of Optical Science and Engineering, Zhejiang University, China. Her research interests include adaptive optics, machine learning, and deep tissue imaging" ]
  [ "", "Wei GONG, corresponding author of this invited paper, is a PI at Zhejiang University School of Medicine, China. She received her BS and MS degrees from Zhejiang University, China, and her PhD degree from the National University of Singapore, Singapore. She is a special expert of the Ministry of Education (MOE) and the Outstanding Youth of Zhejiang Province. Her research interests include biomedical imaging, optical clearing, and artificial intelligence in biomedicine. E-mail: weigong@zju.edu.cn" ]
  [ "", "Ke SI, corresponding author of this invited paper, is a professor at the College of Optical Science and Engineering, Zhejiang University, China, and a joint professor at Zhejiang University School of Medicine. He is the Vice Director of MOE Frontier Science Center for Brain Science and Brain-Machine Integration, and the Vice Dean of the School of Brain Science and Brain Medicine. He is now a corresponding expert of Front Inform Technol Electron Eng. His research focuses on biophotonics, deep tissue imaging, adaptive optics, and optogenetics" ]
- Funds：
- DOI：10.1631/FITEE.2000422
  CLC： O439
- Received：21 August 2020，
  
  Revised：2021-;1-25，
  
  Published Online：29 March 2021，
  
  Published：2021-10
- Accepted：
Scan QR Code
Shuwen HU, Lejia HU, Wei GONG, et al. Deep learning based wavefront sensor for complex wavefront detection in adaptive optical microscopes[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(10): 1277-1288.
DOI：

Shuwen HU, Lejia HU, Wei GONG, et al. Deep learning based wavefront sensor for complex wavefront detection in adaptive optical microscopes[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(10): 1277-1288. DOI： 10.1631/FITEE.2000422.

摘要

Shack-Hartmann波前传感器（SHWS）是自适应光学显微镜中用于波前传感的重要工具。然而，由复杂波前相位分布引起的畸变点阵限制了其探测性能。本文提出一种基于深度学习的波前探测方法，该方法结合了基于点扩散函数图像的泽尼克（Zernike）系数估计和波前相位分布拼接。该方法不仅仅使用每个子孔径的质心位移，而是通过子孔径的点扩散函数分布估计局部波前对应的Zernike系数，然后拼接局部波前进行重建。本文所提方法可实现高精度的复杂波前检测，获得的波前残差均方根误差值显著降低，在自适应光学显微中具有极大应用潜力。

Abstract

The Shack-Hartmann wavefront sensor (SHWS) is an essential tool for wavefront sensing in adaptive optical microscopes. However

the distorted spots induced by the complex wavefront challenge its detection performance. Here

we propose a deep learning based wavefront detection method which combines point spread function image based Zernike coefficient estimation and wavefront stitching. Rather than using the centroid displacements of each micro-lens

this method first estimates the Zernike coefficients of local wavefront distribution over each micro-lens and then stitches the local wavefronts for reconstruction. The proposed method can offer low root mean square wavefront errors and high accuracy for complex wavefront detection

and has potential to be applied in adaptive optical microscopes.

关键词

Keywords

references

MJ Booth . . Adaptive optical microscopy: the ongoing quest for a perfect image . . Light Sci Appl , , 2014 . . 3 ( ( 4 ): ): e165 DOI: 10.1038/lsa.2014.46 http://doi.org/10.1038/lsa.2014.46 . .

MJ Booth , , , MAA Neil , , , R Jukaitis , , , 等 . . Adaptive aberration correction in a confocal microscope . . Proc Nat Acad Sci , , 2002 . . 99 ( ( 9 ): ): 5788 - - 5792 . . DOI: 10.1073/pnas.082544799 http://doi.org/10.1073/pnas.082544799 . .

SF Cheng , , , HH Li , , , YQ Luo , , , 等 . . Artificial intelligence- assisted light control and computational imaging through scattering media . . J Innov Opt Health Sci , , 2019 . . 12 ( ( 4 ): ): 193006 DOI: 10.1142/s1793545819300064 http://doi.org/10.1142/s1793545819300064 . .

A Cornea , , , PM Conn . . Fluorescence Microscopy: Super Resolution and Other Novel Techniques . . Elsevier, London, UK , , 2014 . . p. 249 . .

M Cui . . Parallel wavefront optimization method for focusing light through random scattering media . . Opt Lett , , 2011 . . 36 ( ( 6 ): ): 870 - - 872 . . DOI: 10.1364/ol.36.000870 http://doi.org/10.1364/ol.36.000870 . .

BP Cumming , , , M Gu . . Direct determination of aberration functions in microscopy by an artificial neural network . . Opt Expr , , 2020 . . 28 ( ( 10 ): ): 14511 - - 14521 . . DOI: 10.1364/OE.390856 http://doi.org/10.1364/OE.390856 . .

GM Dai . . Wavefront Optics for Vision Correction , , : : Bellingham, USA SPIE Press , , 2008 . . DOI: 10.1117/3.769212 http://doi.org/10.1117/3.769212 . .

M Drozdzal , , , E Vorontsov , , , G Chartrand , , , 等 . . The importance of skip connections in biomedical image segmentation . . Int Workshop on Deep Learning in Medical Image Analysis and Int Workshop on Large- Scale Annotation of Biomedical Data and Expert Label Synthesis , , 2016 . . p. 179 - - 187 . . DOI: 10.1007/978-3-319-46976-8_19 http://doi.org/10.1007/978-3-319-46976-8_19 . .

TB Dubose , , , DF Gardner , , , AT Watnik . . Intensity-enhanced deep network wavefront reconstruction in ShackɃ Hartmann sensors . . Opt Lett , , 2020 . . 45 ( ( 7 ): ): 1699 - - 1702 . . DOI: 10.1364/OL.389895 http://doi.org/10.1364/OL.389895 . .

SLS Gómez , , , C González-Gutiérrez , , , ED Alonso , , , 等 . . Improving adaptive optics reconstructions with a deep learning approach . . Int Conf on Hybrid Artificial Intelligence Systems , , 2018 . . p. 74 - - 83 . . DOI: 10.1007/978-3-319-92639-1_7 http://doi.org/10.1007/978-3-319-92639-1_7 . .

LJ Hu , , , SW Hu , , , W Gong , , , 等 . . Learning-based Shack- Hartmann wavefront sensor for high-order aberration detection . . Opt Expr , , 2019 . . 27 ( ( 23 ): ): 33504 - - 33517 . . DOI: 10.1364/OE.27.033504 http://doi.org/10.1364/OE.27.033504 . .

LJ Hu , , , SW Hu , , , YN Li , , , 等 . . Reliability of wavefront shaping based on coherent optical adaptive technique in deep tissue focusing . . J Biophoton , , 2020 . . 13 ( ( 1 ): ): e201900245 DOI: 10.1002/jbio.201900245 http://doi.org/10.1002/jbio.201900245 . .

SW Hu , , , LJ Hu , , , BW Zhang , , , 等 . . Simplifying the detection of optical distortions by machine learning . . J Innov Opt Health Sci , , 2020 . . 13 ( ( 3 ): ): 2040001 DOI: 10.1142/s1793545820400015 http://doi.org/10.1142/s1793545820400015 . .

N Ji . . Adaptive optical fluorescence microscopy . . Nat Methods , , 2017 . . 14 ( ( 4 ): ): 374 - - 280 . . DOI: 10.1038/nmeth.4218 http://doi.org/10.1038/nmeth.4218 . .

YC Jin , , , YY Zhang , , , LJ Hu , , , 等 . . Machine learning guided rapid focusing with sensor-less aberration corrections . . Opt Expr , , 2018 . . 26 ( ( 23 ): ): 30162 - - 30171 . . DOI: 10.1364/OE.26.030162 http://doi.org/10.1364/OE.26.030162 . .

ZH Li , , , ZP Yu , , , H Hui , , , 等 . . Edge enhancement through scattering media enabled by optical wavefront shaping . . Photon Res , , 2020 . . 8 ( ( 6 ): ): 954 - - 962 . . DOI: 10.1364/PRJ.388062 http://doi.org/10.1364/PRJ.388062 . .

R Liu , , , ZY Li , , , JS Marvin , , , 等 . . Direct wavefront sensing enables functional imaging of infragranular axons and spines . . Nat Methods , , 2019 . . 16 ( ( 7 ): ): 615 - - 618 . . DOI: 10.1038/s41592-019-0434-7 http://doi.org/10.1038/s41592-019-0434-7 . .

TL Liu , , , S Upadhyayula , , , DE Milkie , , , 等 . . Observing the cell in its native state: imaging subcellular dynamics in multicellular organisms . . Science , , 2018 . . 360 ( ( 6386 ): ): eaaq1392 DOI: 10.1126/science.aaq1392 http://doi.org/10.1126/science.aaq1392 . .

VN Mahajan , , , GM Dai . . Orthonormal polynomials in wavefront analysis: analytical solution . . J Opt Soc Am A , , 2007 . . 24 ( ( 9 ): ): 2994 - - 3016 . . DOI: 10.1364/JOSAA.24.002994 http://doi.org/10.1364/JOSAA.24.002994 . .

Y Nishizaki , , , M Valdivia , , , R Horisaki , , , 等 . . Deep learning wavefront sensing . . Opt Expr , , 2019 . . 27 ( ( 1 ): ): 240 - - 251 . . DOI: 10.1364/OE.27.000240 http://doi.org/10.1364/OE.27.000240 . .

SW Paine , , , JR Fienup . . Machine learning for improved image-based wavefront sensing . . Opt Lett , , 2018 . . 43 ( ( 6 ): ): 1235 - - 1238 . . DOI: 10.1364/OL.43.001235 http://doi.org/10.1364/OL.43.001235 . .

JH Park , , , LJ Kong , , , YF Zhou , , , 等 . . Large-field-of-view imaging by multi-pupil adaptive optics . . Nat Methods , , 2017 . . 14 ( ( 6 ): ): 581 - - 583 . . DOI: 10.1038/nmeth.4290 http://doi.org/10.1038/nmeth.4290 . .

C Rodríguez , , , N Ji . . Adaptive optical microscopy for neurobiology . . Curr Opin Neurobiol , , 2018 . . 50 83 - - 91 . . DOI: 10.1016/j.conb.2018.01.011 http://doi.org/10.1016/j.conb.2018.01.011 . .

S Schott , , , J Bertolotti , , , JF Léger , , , 等 . . Characterization of the angular memory effect of scattered light in biological tissues . . Opt Expr , , 2015 . . 23 ( ( 10 ): ): 13505 - - 13516 . . DOI: 10.1364/OE.23.013505 http://doi.org/10.1364/OE.23.013505 . .

R Swanson , , , M Lamb , , , C Correia , , , 等 . . Wavefront reconstruction and prediction with convolutional neural networks . . Adaptive Optics Systems VI , , 2018 . . Article 10703F DOI: 10.1117/12.2312590 http://doi.org/10.1117/12.2312590 . .

JY Tang , , , RN Germain , , , M Cui . . Superpenetration optical microscopy by iterative multiphoton adaptive compensation technique . . Proc Nat Acad Sci , , 2012 . . 109 ( ( 22 ): ): 8434 - - 8439 . . DOI: 10.1073/pnas.1119590109 http://doi.org/10.1073/pnas.1119590109 . .

PO Vanberg , , , GO de Xivry , , , O Absil , , , 等 . . Machine learning for image-based wavefront sensing . . 33 rd Conf on Neural Information Processing Systems , , 2019 . . p. 1 - - 6 . . . .

BK Wang , , , M Barbiero , , , QM Zhang , , , 等 . . Super- resolution optical microscope: principle, instrumentation, and application . . Front Inform Technol Electron Eng , , 2019 . . 20 ( ( 5 ): ): 608 - - 630 . . DOI: 10.1631/FITEE.1800449 http://doi.org/10.1631/FITEE.1800449 . .

K Wang , , , DE Milkie , , , A Saxena , , , 等 . . Rapid adaptive optical recovery of optimal resolution over large volumes . . Nat Methods , , 2014 . . 11 ( ( 6 ): ): 625 - - 628 . . DOI: 10.1038/nmeth.2925 http://doi.org/10.1038/nmeth.2925 . .

K Wang , , , WZ Sun , , , CT Richie , , , 等 . . Direct wavefront sensing for high-resolution in vivo imaging in scattering tissue . . Nat Commun , , 2015 . . 6 7276 DOI: 10.1038/ncomms8276 http://doi.org/10.1038/ncomms8276 . .

J Yoon , , , M Lee , , , K Lee , , , 等 . . Optogenetic control of cell signaling pathway through scattering skull using wavefront shaping . . Sci Rep , , 2015 . . 5 13289 DOI: 10.1038/srep13289 http://doi.org/10.1038/srep13289 . .

ZP Yu , , , MY Xia , , , HH Li , , , 等 . . Implementation of digital optical phase conjugation with embedded calibration and phase rectification . . Sci Rep , , 2019 . . 9 ( ( 1 ): ): 1537 DOI: 10.1038/s41598-018-38326-4 http://doi.org/10.1038/s41598-018-38326-4 . .

ZP Zeng , , , H Xie , , , L Chen , , , 等 . . Computational methods in super-resolution microscopy . . Front Inform Technol Electron Eng , , 2017 . . 18 ( ( 9 ): ): 1222 - - 1235 . . DOI: 10.1631/FITEE.1601628 http://doi.org/10.1631/FITEE.1601628 . .

Views

105

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems

Related Author

Yusong ZHOU

Xiaoyu JIANG

Shu SUN

Xinmin ZHANG

Yuanqiu MO

Zhihuan SONG

Maokun ZHENG

Zhi LI

Related Institution

School of Mathematics, Southeast University

State Key Laboratory of Industrial Control Technology, Zhejiang University

College of Control Science and Engineering, Zhejiang University

State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰