用改进的分数阶最速下降法训练分数阶全局最优反向传播机

蒲亦非; 王健

doi:10.1631/FITEE.1900593

Your Location：

Home >

Browse articles >

用改进的分数阶最速下降法训练分数阶全局最优反向传播机

用于信息科学的分数阶积分新趋势专题 | Updated：2022-05-19

- 用改进的分数阶最速下降法训练分数阶全局最优反向传播机
- Fractional-order global optimal backpropagation machine trained by an improved fractional-order steepest descent method
- 信息与电子工程前沿（英文） 2020年21卷第6期页码：809-833
- Affiliations：
  
  College of Computer Science, Sichuan University, Chengdu 610065, China
  College of Science, China University of Petroleum, Qingdao 266580, China
- Author bio：
  
  Yi-fei P, E-mail: puyifei@scu.edu.cn
  Jian WANG, E-mail: wangjiannl@upc.edu.cn
- Funds：
  
  Project supported by the National Key Research and Development Program of China (No. 2018YFC0830300) and the National Natural Science Foundation of China (No. 61571312)
- DOI：10.1631/FITEE.1900593
  中图分类号： O235;N93
- 收稿：2019-10-31，
  
  修回：2020-;3-31，
  
  纸质出版：2020-06
- Accepted：
Scan QR Code
蒲亦非, 王健. 用改进的分数阶最速下降法训练分数阶全局最优反向传播机[J]. 信息与电子工程前沿（英文）, 2020,21(6):809-833.

Yi-fei PU, Jian WANG. Fractional-order global optimal backpropagation machine trained by an improved fractional-order steepest descent method[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(6): 809-833.
蒲亦非, 王健. 用改进的分数阶最速下降法训练分数阶全局最优反向传播机[J]. 信息与电子工程前沿（英文）, 2020,21(6):809-833. DOI： 10.1631/FITEE.1900593.

Yi-fei PU, Jian WANG. Fractional-order global optimal backpropagation machine trained by an improved fractional-order steepest descent method[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(6): 809-833. DOI： 10.1631/FITEE.1900593.

摘要

本文介绍采用改进的分数阶最速下降法（FSDM）训练分数阶全局最优反向传播机。该反向传播机是一种分数阶反向传播神经网络（FBPNN）。分数阶反向传播神经网络是反向传播神经网络（BPNNs）大家族中一个先进的分数阶分支，它不同于绝大多数传统一阶最速下降法训练的经典一阶BPNNs。本文提出的FBPNN反向增量搜索在其均方误差近似分数阶偏导数的负方向进行。首先，从数学上描述用改进FSDM训练的FBPNN理论概念。然后，详细给出FBPNN分数阶全局最优收敛性的数学证明，分析神经网络结构构建以及分数阶多尺度全局寻优问题。最后，通过实验比较FBPNN和经典一阶BPNN的性能：包括函数逼近、分数阶多尺度全局寻优以及基于实际数据的全局搜索和误差拟合能力比对。相比经典一阶BPNN，FBPNN最主要优点是具有更高效的全局寻优能力，能够判定全局最优解。

Abstract

We introduce the fractional-order global optimal backpropagation machine

which is trained by an improved fractionalorder steepest descent method (FSDM). This is a fractional-order backpropagation neural network (FBPNN)

a state-of-the-art fractional-order branch of the family of backpropagation neural networks (BPNNs)

different from the majority of the previous classic first-order BPNNs which are trained by the traditional first-order steepest descent method. The reverse incremental search of the proposed FBPNN is in the negative directions of the approximate fractional-order partial derivatives of the square error. First

the theoretical concept of an FBPNN trained by an improved FSDM is described mathematically. Then

the mathematical proof of fractional-order global optimal convergence

an assumption of the structure

and fractional-order multi-scale global optimization of the FBPNN are analyzed in detail. Finally

we perform three (types of) experiments to compare the performances of an FBPNN and a classic first-order BPNN

i.e.

example function approximation

fractional-order multi-scale global optimization

and comparison of global search and error fitting abilities with real data. The higher optimal search ability of an FBPNN to determine the global optimal solution is the major advantage that makes the FBPNN superior to a classic first-order BPNN.

关键词

Keywords

references

M Andramonov , , , A Rubinov , , , B Glover . . Cutting angle methods in global optimization . . Appl Math Lett , , 1999 . . 12 ( ( 3 ): ): 95 - - 100 . . DOI: 10.1016/S0893-9659(98)00179-7 http://doi.org/10.1016/S0893-9659(98)00179-7 . .

E Barnard . . Optimization for training neural nets . . IEEE Trans Neur Netw , , 1992 . . 3 ( ( 2 ): ): 232 - - 240 . . DOI: 10.1109/72.125864 http://doi.org/10.1109/72.125864 . .

AR Barron . . Universal approximation bounds for superpositions of a sigmoidal function . . IEEE Trans Inform Theory , , 1993 . . 39 ( ( 3 ): ): 930 - - 945 . . DOI: 10.1109/18.256500 http://doi.org/10.1109/18.256500 . .

R Battiti . . First- and second-order methods for learning: between steepest descent and Newton's method . . Neur Comput , , 1992 . . 4 ( ( 2 ): ): 141 - - 166 . . DOI: 10.1162/neco.1992.4.2.141 http://doi.org/10.1162/neco.1992.4.2.141 . .

CB Browne , , , E Powley , , , D Whitehouse , , , 等 . . A survey of Monte Carlo tree search methods . . IEEE Trans Comput Intell AI Games , , 2012 . . 4 ( ( 1 ): ): 1 - - 43 . . DOI: 10.1109/tciaig.2012.2186810 http://doi.org/10.1109/tciaig.2012.2186810 . .

E Cantu-Paz , , , C Kamath . . An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems . . IEEE Trans Syst Man Cybern , , 2005 . . 35 ( ( 5 ): ): 915 - - 927 . . DOI: 10.1109/TSMCB.2005.847740 http://doi.org/10.1109/TSMCB.2005.847740 . .

C Charalambous . . Conjugate gradient algorithm for efficient training of artificial neural networks . . IEE Proc G , , 1992 . . 139 ( ( 3 ): ): 301 - - 310 . . DOI: 10.1049/ip-g-2.1992.0050 http://doi.org/10.1049/ip-g-2.1992.0050 . .

CC Chuang , , , SF Su , , , CC Hsiao . . The annealing robust backpropagation (ARBP) learning algorithm . . IEEE Trans Neur Netw , , 2000 . . 11 ( ( 5 ): ): 1067 - - 1077 . . DOI: 10.1109/72.870040 http://doi.org/10.1109/72.870040 . .

G Cybenko . . Approximation by superpositions of a sigmoidal function . . Math Contr Signals Syst , , 1989 . . 2 ( ( 4 ): ): 303 - - 314 . . DOI: 10.1007/bf02551274 http://doi.org/10.1007/bf02551274 . .

AS Elwakil . . Fractional-order circuits and systems: an emerging interdisciplinary research area . . IEEE Circ Syst Mag , , 2010 . . 10 ( ( 4 ): ): 40 - - 50 . . DOI: 10.1109/MCAS.2010.938637 http://doi.org/10.1109/MCAS.2010.938637 . .

MT Hagan , , , MB Menhaj . . Training feedforward networks with the Marquardt algorithm . . IEEE Trans Neur Netw , , 1994 . . 5 ( ( 6 ): ): 989 - - 993 . . DOI: 10.1109/72.329697 http://doi.org/10.1109/72.329697 . .

N Heymans , , , I Podlubny . . Physical interpretation of initial conditions for fractional differential equations with Riemann-Liouville fractional derivatives . . Rheol Acta , , 2006 . . 45 ( ( 5 ): ): 765 - - 771 . . DOI: 10.1007/s00397-005-0043-5 http://doi.org/10.1007/s00397-005-0043-5 . .

K Hornik , , , M Stinchcombe , , , H White . . Multilayer feedforward networks are universal approximators . . Neur Netw , , 1989 . . 2 ( ( 5 ): ): 359 - - 366 . . DOI: 10.1016/0893-6080(89)90020-8 http://doi.org/10.1016/0893-6080(89)90020-8 . .

RA Jacobs . . Increased rates of convergence through learning rate adaptation . . Neur Netw , , 1988 . . 1 ( ( 4 ): ): 295 - - 307 . . DOI: 10.1016/0893-6080(88)90003-2 http://doi.org/10.1016/0893-6080(88)90003-2 . .

E Kaslik , , , S Sivasundaram . . Dynamics of fractional-order neural networks . . Proc Int Joint Conf on Neural Networks , , 2011 . . p.1375 - - 1380 . . DOI: 10.1109/IJCNN.2011.6033277 http://doi.org/10.1109/IJCNN.2011.6033277 . .

RC Koeller . . Applications of fractional calculus to the theory of viscoelasticity . . J Appl Mech , , 1984 . . 51 ( ( 2 ): ): 299 - - 307 . . DOI: 10.1115/1.3167616 http://doi.org/10.1115/1.3167616 . .

Y LeCun . . Une procedure d'apprentissage pour reseau a seuil assymetrique . . Proc Cogn , , 1985 . . 85 599 - - 604 . . . .

FHF Leung , , , HK Lam , , , SH Ling , , , 等 . . Tuning of the structure and parameters of a neural network using an improved genetic algorithm . . IEEE Trans Neur Netw , , 2003 . . 14 ( ( 1 ): ): 79 - - 88 . . DOI: 10.1109/TNN.2002.804317 http://doi.org/10.1109/TNN.2002.804317 . .

TB Ludermir , , , A Yamazaki , , , C Zanchettin . . An optimization methodology for neural network weights and architectures . . IEEE Trans Neur Netw , , 2006 . . 17 ( ( 6 ): ): 1452 - - 1459 . . DOI: 10.1109/TNN.2006.881047 http://doi.org/10.1109/TNN.2006.881047 . .

S Manabe . . A suggestion of fractional-order controller for flexible spacecraft attitude control . . Nonl Dynam , , 2002 . . 29 ( ( 1-4 ): ): 251 - - 268 . . DOI: 10.1023/a:1016566017098 http://doi.org/10.1023/a:1016566017098 . .

V Maniezzo . . Genetic evolution of the topology and weight distribution of neural networks . . IEEE Trans Neur Netw , , 1994 . . 5 ( ( 1 ): ): 39 - - 53 . . DOI: 10.1109/72.265959 http://doi.org/10.1109/72.265959 . .

NY Nikolaev , , , H Iba . . Learning polynomial feedforward neural networks by genetic programming and backpropagation . . IEEE Trans Neur Netw , , 2003 . . 14 ( ( 2 ): ): 337 - - 350 . . DOI: 10.1109/TNN.2003.809405 http://doi.org/10.1109/TNN.2003.809405 . .

KB Oldham , , , J Spanier . . The Fractional Calculus: Integrations and Differentiations of Arbitrary Order , , : : New York Academic Press , , 1974 . . p.1 - - 234 . . . .

N Özdemir , , , D Karadeniz . . Fractional diffusion-wave problem in cylindrical coordinates . . Phys Lett A , , 2008 . . 372 ( ( 38 ): ): 5968 - - 5972 . . DOI: 10.1016/j.physleta.2008.07.054 http://doi.org/10.1016/j.physleta.2008.07.054 . .

PP Palmes , , , T Hayasaka , , , S Usui . . Mutation-based genetic neural network . . IEEE Trans Neur Netw , , 2005 . . 16 ( ( 3 ): ): 587 - - 600 . . DOI: 10.1109/tnn.2005.844858 http://doi.org/10.1109/tnn.2005.844858 . .

DB Parker . . Learning-Logic: Casting the Cortex of the Human Brain in Silicon . . Technical Report, No. TR-47, Center for Computational Research in Economics and Management Science, MIT, USA , , 1985 . . http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0224778092/ http://d.old.wanfangdata.com.cn/NSTLQK/NSTL_QKJJ0224778092/ , , . .

I Petrá . . Fractional-Order Nonlinear Systems: Modeling, Analysis and Simulation . . Springer Berlin Heidelberg, Berlin, Germany , , 2011 . . p.1 - - 218 . . http://d.old.wanfangdata.com.cn/Periodical/kzllyyy201606006 http://d.old.wanfangdata.com.cn/Periodical/kzllyyy201606006 , , . .

I Podlubny . . Fractional Differential Equations: an Introduction to Fractional Derivatives, Fractional Differential Equations, to Methods of Their Solution and Some of Their Applications , , : : San Diego, USA Academic Press , , 1998 . . p.1 - - 340 . . . .

I Podlubny , , , I Petrá , , , BM Vinagre , , , 等 . . Analogue realizations of fractional-order controllers . . Nonl Dynam , , 2002 . . 29 ( ( 1-4 ): ): 281 - - 296 . . DOI: 10.1023/a:1016556604320 http://doi.org/10.1023/a:1016556604320 . .

YF Pu , , , JL Zhou , , , X Yuan . . Fractional differential mask: a fractional differential-based approach for multiscale texture enhancement . . IEEE Trans Image Process , , 2010 . . 19 ( ( 2 ): ): 491 - - 511 . . DOI: 10.1109/TIP.2009.2035980 http://doi.org/10.1109/TIP.2009.2035980 . .

YF Pu , , , JL Zhou , , , Y Zhang , , , 等 . . Fractional extreme value adaptive training method: fractional steepest descent approach . . IEEE Trans Neur Netw Learn Syst , , 2015 . . 26 ( ( 4 ): ): 653 - - 662 . . DOI: 10.1109/TNNLS.2013.2286175 http://doi.org/10.1109/TNNLS.2013.2286175 . .

YF Pu , , , Z Yi , , , JL Zhou . . Defense against chip cloning attacks based on fractional Hopfield neural networks . . Int J Neur Syst , , 2016 . . 27 ( ( 4 ): ): 1750003 DOI: 10.1142/S0129065717500034 http://doi.org/10.1142/S0129065717500034 . .

YF Pu , , , Z Yi , , , JL Zhou . . Fractional Hopfield neural networks: fractional dynamic associative recurrent neural networks . . IEEE Trans Neur Netw Learn Syst , , 2017 . . 28 ( ( 10 ): ): 2319 - - 2333 . . DOI: 10.1109/TNNLS.2016.2582512 http://doi.org/10.1109/TNNLS.2016.2582512 . .

YF Pu , , , X Yuan , , , B Yu . . Analog circuit implementation of fractional-order memristor: arbitrary-order lattice scaling fracmemristor . . IEEE Trans Circ Syst I , , 2018a . . 65 ( ( 9 ): ): 2903 - - 2916 . . DOI: 10.1109/TCSI.2018.2789907 http://doi.org/10.1109/TCSI.2018.2789907 . .

YF Pu , , , P Siarry , , , A Chatterjee , , , 等 . . A fractional-order variational framework for retinex: fractional-order partial differential equation-based formulation for multi-scale nonlocal contrast enhancement with texture preserving . . IEEE Trans Image Process , , 2018b . . 27 ( ( 3 ): ): 1214 - - 1229 . . DOI: 10.1109/TIP.2017.2779601 http://doi.org/10.1109/TIP.2017.2779601 . .

AK Rigler , , , JM Irvine , , , TP Vogl . . Rescaling of variables in back propagation learning . . Neur Netw , , 1991 . . 4 ( ( 2 ): ): 225 - - 229 . . DOI: 10.1016/0893-6080(91)90006-q http://doi.org/10.1016/0893-6080(91)90006-q . .

YA Rossikhin , , , MV Shitikova . . Applications of fractional calculus to dynamic problems of linear and nonlinear hereditary mechanics of solids . . Appl Mech Rev , , 1997 . . 50 ( ( 1 ): ): 15 - - 67 . . DOI: 10.1115/1.3101682 http://doi.org/10.1115/1.3101682 . .

DE Rumelhart , , , GE Hinton , , , RJ Williams . . Learning representations by back-propagating errors . . Nature , , 1986a . . 323 ( ( 6088 ): ): 533 - - 536 . . DOI: 10.1038/323533a0 http://doi.org/10.1038/323533a0 . .

DE Rumelhart , , , JL McClelland , , , Research Group PDP . . Parallel Distributed Processing: Explorations in the Microstructure of Cognition , , : : Cambridge, USA Vol. 1 MIT Press , , 1986b . . p.547 - - 611 . . . .

DF Shanno . . Recent advances in numerical techniques for large-scale optimization . . In: Miller WT, Sutton RS, Werbos PJ (Eds.), Neural Networks for Control. MIT Press, Cambridge, US , , 1990 . . 171 - - 178 . . . .

ED Sontag . . Feedback stabilization using two-hidden- layer nets . . IEEE Trans Neur Netw , , 1992 . . 3 ( ( 6 ): ): 981 - - 990 . . DOI: 10.1109/72.165599 http://doi.org/10.1109/72.165599 . .

T Tollenaere . . SuperSAB: fast adaptive back propagation with good scaling properties . . Neur Netw , , 1990 . . 3 ( ( 5 ): ): 561 - - 573 . . DOI: 10.1016/0893-6080(90)90006-7 http://doi.org/10.1016/0893-6080(90)90006-7 . .

NK Treadgold , , , TD Gedeon . . Simulated annealing and weight decay in adaptive learning: the SARPROP algorithm . . IEEE Trans Neur Netw , , 1998 . . 9 ( ( 4 ): ): 662 - - 668 . . DOI: 10.1109/72.701179 http://doi.org/10.1109/72.701179 . .

TP Vogl , , , JK Mangis , , , AK Rigler , , , 等 . . Accelerating the convergence of the back-propagation method . . Biol Cybern , , 1988 . . 59 ( ( 4-5 ): ): 257 - - 263 . . DOI: 10.1007/bf00332914 http://doi.org/10.1007/bf00332914 . .

PJ Werbos . . Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences . . PhD Thesis, Harvard University, Cambridge, USA , , 1974 . . .

WC Yeh . . New parameter-free simplified swarm optimization for artificial neural network training and its application in the prediction of time series . . IEEE Trans Neur Netw Learn Syst , , 2013 . . 24 ( ( 4 ): ): 661 - - 665 . . DOI: 10.1109/TNNLS.2012.2232678 http://doi.org/10.1109/TNNLS.2012.2232678 . .

C Zanchettin , , , TB Ludermir , , , LM Almeida . . Hybrid training method for MLP: optimization of architecture and training . . IEEE Trans Syst Man Cybern , , 2011 . . 41 ( ( 4 ): ): 1097 - - 1109 . . DOI: 10.1109/TSMCB.2011.2107035 http://doi.org/10.1109/TSMCB.2011.2107035 . .

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Swarm intelligent computing of electric eel foraging heuristics for fractional Hammerstein autoregressive exogenous noise model identification

Fractional-order memristive neural synaptic weighting achieved by pulse-based fracmemristor bridge circuit

Design of a fractional

${\rm{P}}{{\rm{I}}^\lambda }{{\rm{D}}^\mu }$

controller using the cohort intelligence method