FOLLOWUS
1.National Digital Switching System Engineering & Technological R&D Center, Zhengzhou 450003, China
2.Computation and Artificial Intelligence Innovative College, Fudan University, Shanghai 201203, China
3.School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China
‡ Corresponding authors
Received:14 July 2024,
Revised:11 October 2024,
Published:2025-05
Scan QR Code
Li CHEN, Fan ZHANG, Guangwei XIE, et al. S3Det: a fast object detector for remote sensing images based on artificial to spiking neural network conversion[J]. Frontiers of information technology & electronic engineering, 2025, 26(5): 713-727.
Li CHEN, Fan ZHANG, Guangwei XIE, et al. S3Det: a fast object detector for remote sensing images based on artificial to spiking neural network conversion[J]. Frontiers of information technology & electronic engineering, 2025, 26(5): 713-727. DOI: 10.1631/FITEE.2400594.
人工神经网络(ANN)在遥感影像目标检测方面取得显著进展。然而,低检测效率和高能耗一直是遥感领域的重要瓶颈。脉冲神经网络(SNN)以稀疏脉冲的形式处理信息,为计算机视觉任务带来高效能优势。不过,大部分研究工作集中在简单分类任务上,仅有少数研究者将其应用于自然图像的目标检测。本文考虑到生物大脑的简约特性,提出一种人工—脉冲神经网络快速转换方法,用于遥感影像检测。基于群组稀疏特征建立快速稀疏模型进行脉冲序列感知,并对原始图像进行变换域内的稀疏重采样,从而快速感知图像特征和编码的脉冲序列。此外,为满足相关遥感场景中的精度要求,从理论上分析了转换误差,提出通道自衰减加权归一化方法,以消除神经元过度激活。所提遥感影像目标检测模型被称作S3Det。基于一个大型公开遥感数据集的实验表明,S3Det实现了与ANN相似的精度。同时,我们的转换网络稀疏度为原始算法的24.32%;能耗仅为1.46 W,是原始算法的1/122。
Artificial neural networks (ANNs) have made great strides in the field of remote sensing image object detection. However
low detection efficiency and high power consumption have always been significant bottlenecks in remote sensing. Spiking neural networks (SNNs) process information in the form of sparse spikes
creating the advantage of high energy efficiency for computer vision tasks. However
most studies have focused on simple classification tasks
and only a few researchers have applied SNNs to object detection in natural images. In this study
we consider the parsimonious nature of biological brains and propose a fast ANN-to-SNN conversion method for remote sensing image detection. We establish a fast sparse model for pulse sequence perception based on group sparse features and conduct transform-domain sparse resampling of the original images to enable fast perception of image features and encoded pulse sequences. In addition
to meet accuracy requirements in relevant remote sensing scenarios
we theoretically analyze the transformation error and propose channel self-decaying weighted normalization (CSWN) to eliminate neuron overactivation. We propose S3Det
a remote sensing image object detection model. Our experiments
based on a large publicly available remote sensing dataset
show that S3Det achieves an accuracy performance similar to that of the ANN. Meanwhile
our transformed network is only 24.32% as sparse as the benchmark and consumes only 1.46 W
which is 1/122 of the original algorithm’s power consumption.
Azimi SM , Vig E , Bahmanyar R , et al. , 2018 . Towards multi-class object detection in unconstrained remote sensing imagery . Proc 14 th Asian Conf on Computer Vision , p. 150 - 165 . https://doi.org/10.1007/978-3-030-20893-6_10 https://doi.org/10.1007/978-3-030-20893-6_10
Chen GH , Pei GS , Tang Y , et al. , 2022 . A novel multi-sample data augmentation method for oriented object detection in remote sensing images . Proc IEEE 24 th Int Workshop on Multimedia Signal Processing , p. 1 - 7 . https://doi.org/10.1109/MMSP55362.2022.9949615 https://doi.org/10.1109/MMSP55362.2022.9949615
Chen K , Wang JQ , Pang JM , et al. , 2019 . MMDetection: open MMlab detection toolbox and benchmark . https://arxiv.org/abs/1906.07155 https://arxiv.org/abs/1906.07155
Chen L , Zhang F , Guo W , et al. , 2023 . SFTN: fast object detection for aerial images . IET Image Process , 17 ( 13 ): 3897 - 3907 . https://doi.org/10.1049/ipr2.12906 https://doi.org/10.1049/ipr2.12906
Cheng G , Zhou PC , Han JW , et al. , 2016a . Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images . IEEE Trans Geosci Remote Sens , 54 ( 12 ): 7405 - 7415 . https://doi.org/10.1109/TGRS.2016.2601622 https://doi.org/10.1109/TGRS.2016.2601622
Cheng G , Zhou PC , Han JW , 2016b . RIFD-CNN: rotation-invariant and Fisher discriminative convolutional neural networks for object detection . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 2884 - 2893 . https://doi.org/10.1109/CVPR.2016.315 https://doi.org/10.1109/CVPR.2016.315
Ding J , Xue N , Long Y , et al. , 2019 . Learning RoI Transformer for oriented object detection in aerial images . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 2844 - 2853 . https://doi.org/10.1109/CVPR.2019.00296 https://doi.org/10.1109/CVPR.2019.00296
Eldar YC , Kutyniok G , 2012 . Compressed Sensing: Theory and Applications . Cambridge University Press , Cambridge, UK . https://doi.org/10.1017/CBO9780511794308 https://doi.org/10.1017/CBO9780511794308
Everingham M , van Gool L , Williams CK , et al. , 2010 . The PASCAL Visual Object Classes (VOC) challenge . Int J Comput Vis , 88 ( 2 ): 303 - 338 . https://doi.org/10.1007/s11263-009-0275-4 https://doi.org/10.1007/s11263-009-0275-4
Gong MG , Li JZ , Zhang YR , et al. , 2022 . Two-path aggregation attention network with quad-patch data augmentation for few-shot scene classification . IEEE Trans Geosci Remote Sens , 60 : 4511616 . https://doi.org/10.1109/TGRS.2022.3197445 https://doi.org/10.1109/TGRS.2022.3197445
Han JM , Ding J , Li J , et al. , 2022 . Align deep features for oriented object detection . IEEE Trans Geosci Remote Sens , 60 : 5602511 . https://doi.org/10.1109/TGRS.2021.3062048 https://doi.org/10.1109/TGRS.2021.3062048
He X , Ma SP , He LY , et al. , 2022 . High-resolution polar network for object detection in remote sensing images . IEEE Geosci Remote Sens Lett , 19 : 6000605 . https://doi.org/10.1109/LGRS.2020.3039240 https://doi.org/10.1109/LGRS.2020.3039240
Horowitz M , 2014 . 1.1 Computing's energy problem (and what we can do about it) . Proc IEEE Int Solid-State Circuits Conf Digest of Technical Papers , p. 10 - 14 . https://doi.org/10.1109/ISSCC.2014.6757323 https://doi.org/10.1109/ISSCC.2014.6757323
Hu YF , Zheng Q , Jiang XD , et al. , 2023 . Fast-SNN: fast spiking neural network by converting quantized ANN . IEEE Trans Patt Anal Mach Intell , 45 ( 12 ): 14546 - 14562 . https://doi.org/10.1109/TPAMI.2023.3275769 https://doi.org/10.1109/TPAMI.2023.3275769
Huang ZC , Li W , Xia XG , et al. , 2022 . A general Gaussian heatmap label assignment for arbitrary-oriented object detection . IEEE Trans Image Process , 31 : 1895 - 1910 . https://doi.org/10.1109/TIP.2022.3148874 https://doi.org/10.1109/TIP.2022.3148874
Jiang YQ , Tan ZY , Wang JY , et al. , 2022 . GiraffeDet: a heavy-neck paradigm for object detection . https://arxiv.org/abs/2202.04256 https://arxiv.org/abs/2202.04256
Jiang YY , Zhu XY , Wang XB , et al. , 2018 . R 2 CNN: rotational region CNN for arbitrarily-oriented scene text detection . Proc 24 th Int Conf on Pattern Recognition , p. 3610 - 3615 . https://doi.org/10.1109/ICPR.2018.8545598 https://doi.org/10.1109/ICPR.2018.8545598
Kim S , Park S , Na B , et al. , 2020 . Spiking-YOLO: spiking neural network for energy-efficient object detection . Proc 34 th AAAI Conf on Artificial Intelligence , p. 11270 - 11277 . https://doi.org/10.1609/aaai.v34i07.6787 https://doi.org/10.1609/aaai.v34i07.6787
Komárek A , Lesaffre E , 2008 . Generalized linear mixed model with a penalized Gaussian mixture as a random effects distribution . Comput Stat Data Anal , 52 ( 7 ): 3441 - 3458 . https://doi.org/10.1016/j.csda.2007.10.024 https://doi.org/10.1016/j.csda.2007.10.024
LeCun Y , Bottou L , Bengio Y , et al. , 1998 . Gradient-based learning applied to document recognition . Proc IEEE , 86 ( 11 ): 2278 - 2324 . https://doi.org/10.1109/5.726791 https://doi.org/10.1109/5.726791
Li Y , He X , Dong YT , et al. , 2022 . Spike calibration: fast and accurate conversion of spiking neural network for object detection and segmentation . https://arxiv.org/abs/2207.02702 https://arxiv.org/abs/2207.02702
Lin TY , Maire M , Belongie S , et al. , 2014 . Microsoft COCO: common objects in context . Proc 13 th European Conf on Computer Vision , p. 740 - 755 . https://doi.org/10.1007/978-3-319-10602-1_48 https://doi.org/10.1007/978-3-319-10602-1_48
Lin TY , Goyal P , Girshick R , et al. , 2017 . Focal loss for dense object detection . Proc IEEE Int Conf on Computer Vision , p. 2999 - 3007 . https://doi.org/10.1109/ICCV.2017.324 https://doi.org/10.1109/ICCV.2017.324
Liu WX , Luo B , Liu J , et al. , 2022 . Synthetic data augmentation using multiscale attention CycleGAN for aircraft detection in remote sensing images . IEEE Geosci Remote Sens Lett , 19 : 4009205 . https://doi.org/10.1109/LGRS.2021.3052017 https://doi.org/10.1109/LGRS.2021.3052017
Liu ZK , Wang HZ , Weng LB , et al. , 2016 . Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds . IEEE Geosci Remote Sens Lett , 13 ( 8 ): 1074 - 1078 . https://doi.org/10.1109/LGRS.2016.2565705 https://doi.org/10.1109/LGRS.2016.2565705
Ma JQ , Shao WY , Ye H , et al. , 2018 . Arbitrary-oriented scene text detection via rotation proposals . IEEE Trans Multim , 20 ( 11 ): 3111 - 3122 . https://doi.org/10.1109/TMM.2018.2818020 https://doi.org/10.1109/TMM.2018.2818020
Maass W , 1997 . Networks of spiking neurons: the third generation of neural network models . Neur Netw , 10 ( 9 ): 1659 - 1671 . https://doi.org/10.1016/S0893-6080(97)00011-7 https://doi.org/10.1016/S0893-6080(97)00011-7
Merolla PA , Arthur JV , Alvarez-Icaza R , et al. , 2014 . A million spiking-neuron integrated circuit with a scalable communication network and interface . Science , 345 ( 6197 ): 668 - 673 . https://doi.org/10.1126/science.1254642 https://doi.org/10.1126/science.1254642
Ming Q , Zhou ZQ , Miao LJ , et al. , 2021 . Dynamic anchor learning for arbitrary-oriented object detection . Proc 35 th AAAI Conf on Artificial Intelligence, Electronic Network , p. 2355 - 2363 . https://doi.org/10.1609/aaai.v35i3.16336 https://doi.org/10.1609/aaai.v35i3.16336
Rathi N , Roy K , 2023 . DIET-SNN: a low-latency spiking neural network with direct input encoding and leakage and threshold optimization . IEEE Trans Neur Netw Learn Syst , 34 ( 6 ): 3174 - 3182 . https://doi.org/10.1109/TNNLS.2021.3111897 https://doi.org/10.1109/TNNLS.2021.3111897
Rueckauer B , Liu SC , 2021 . Temporal pattern coding in deep spiking neural networks . Proc Int Joint Conf on Neural Networks , p. 1 - 8 . https://doi.org/10.1109/IJCNN52387.2021.9533837 https://doi.org/10.1109/IJCNN52387.2021.9533837
Rueckauer B , Lungu IA , Hu YH , et al. , 2017 . Conversion of continuous-valued deep networks to efficient event-driven networks for image classification . Front Neurosci , 11 : 682 . https://doi.org/10.3389/fnins.2017.00682 https://doi.org/10.3389/fnins.2017.00682
Sinha D , El-Sharkawy M , 2019 . Thin MobileNet: an enhanced MobileNet architecture . Proc IEEE 10 th Annual Ubiquitous Computing, Electronics & Mobile Communication Conf , p. 280 - 285 . https://doi.org/10.1109/UEMCON47517.2019.8993089 https://doi.org/10.1109/UEMCON47517.2019.8993089
Vaswani A , Shazeer N , Parmar N , et al. , 2017 . Attention is all you need . Proc 31 st Int Conf on Neural Information Processing Systems , p. 6000 - 6010 .
Wang A , Chen H , Liu LH , et al. , 2024 . YOLOv10: real-time end-to-end object detection . https://arxiv.org/abs/2405.14458 https://arxiv.org/abs/2405.14458
Xia GS , Bai X , Ding J , et al. , 2018 . DOTA: a large-scale dataset for object detection in aerial images . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 3974 - 3983 . https://doi.org/10.1109/CVPR.2018.00418 https://doi.org/10.1109/CVPR.2018.00418
Xie XX , Lang CB , Miao SC , et al. , 2023 . Mutual-assistance learning for object detection . IEEE Trans Patt Anal Mach Intell , 45 ( 12 ): 15171 - 15184 . https://doi.org/10.1109/TPAMI.2023.3319634 https://doi.org/10.1109/TPAMI.2023.3319634
Xie XX , Cheng G , Li QY , et al. , 2024a . Fewer is more: efficient object detection in large aerial images . Sci China Inform Sci , 67 ( 1 ): 112106 . https://doi.org/10.1007/s11432-022-3718-5 https://doi.org/10.1007/s11432-022-3718-5
Xie XX , Cheng G , Rao CF , et al. , 2024b . Oriented object detection via contextual dependence mining and penalty-incentive allocation . IEEE Trans Geosci Remote Sens , 62 : 5618010 . https://doi.org/10.1109/TGRS.2024.3385985 https://doi.org/10.1109/TGRS.2024.3385985
Xie XX , Cheng G , Wang JB , et al. , 2024c . Oriented R-CNN and beyond . Int J Comput Vis , 132 ( 7 ): 2420 - 2442 . https://doi.org/10.1007/s11263-024-01989-w https://doi.org/10.1007/s11263-024-01989-w
Xiong YY , Liu HX , Gupta S , et al. , 2021 . MobileDets: searching for object detection architectures for mobile accelerators . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 3824 - 3833 . https://doi.org/10.1109/CVPR46437.2021.00382 https://doi.org/10.1109/CVPR46437.2021.00382
Yang X , Yan JC , 2022 . On the arbitrary-oriented object detection: classification based approaches revisited . Int J Comput Vis , 130 ( 5 ): 1340 - 1365 . https://doi.org/10.1007/s11263-022-01593-w https://doi.org/10.1007/s11263-022-01593-w
Yang X , Yang JR , Yan JC , et al. , 2019 . SCRDet: towards more robust detection for small, cluttered and rotated objects . Proc IEEE/CVF Int Conf on Computer Vision , p. 8231 - 8240 . https://doi.org/10.1109/ICCV.2019.00832 https://doi.org/10.1109/ICCV.2019.00832
Yang X , Yan JC , Feng ZM , et al. , 2021a . R3Det: refined single-stage detector with feature refinement for rotating object . Proc 35 th AAAI Conf on Artificial Intelligence , p. 3163 - 3171 . https://doi.org/10.1609/aaai.v35i4.16426 https://doi.org/10.1609/aaai.v35i4.16426
Yang X , Yan JC , Ming Q , et al. , 2021b . Rethinking rotated object detection with Gaussian Wasserstein distance loss . Proc 38 th Int Conf on Machine Learning , p. 11830 - 11841 .
Yao M , Zhao GS , Zhang HY , et al. , 2023 . Attention spiking neural networks . IEEE Trans Patt Anal Mach Intell , 45 ( 8 ): 9393 - 9410 . https://doi.org/10.1109/TPAMI.2023.3241201 https://doi.org/10.1109/TPAMI.2023.3241201
Zhang C , Lam KM , Wang Q , 2023 . CoF-Net: a progressive coarse-to-fine framework for object detection in remote-sensing imagery . IEEE Trans Geosci Remote Sens , 61 : 5600617 . https://doi.org/10.1109/TGRS.2022.3233881 https://doi.org/10.1109/TGRS.2022.3233881
Zhang GJ , Lu SJ , Zhang W , 2019 . CAD-Net: a context-aware detection network for objects in remote sensing imagery . IEEE Trans Geosci Remote Sens , 57 ( 12 ): 10015 - 10024 . https://doi.org/10.1109/TGRS.2019.2930982 https://doi.org/10.1109/TGRS.2019.2930982
Publicity Resources
Related Articles
Related Author
Related Institution