Negative effects of sufficiently small initial weights on back-propagation neural networks

Yan LIU; Jie YANG; Long LI; Wei WU

doi:10.1631/jzus.C1200008

Your Location：

Home >

Browse articles >

Negative effects of sufficiently small initial weights on back-propagation neural networks

计算机与自动化 | Updated：2022-12-09

- Negative effects of sufficiently small initial weights on back-propagation neural networks
- Negative effects of sufficiently small initial weights on back-propagation neural networks
- 信息与电子工程前沿（英文） 2012年13卷第8期页码：585-592
- Affiliations：
  
  1.School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, China
  2.School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China
  3.Department of Mathematics and Computational Science, Hengyang Normal University, Hengyang 421002, China
- Author bio：
  
  [ "Yan LIU, E-mail: liuyan@dlpu.edu.cn" ]
  [ "Jie YANG, E-mail: yangjiee@dlut.edu.cn" ]
  [ "Long LI, E-mail: long_li1982@163.com" ]
  Wei WU, E-mail: wuweiw@dlut.edu.cn
- Funds：
  
  Project supported by the National Natural Science Foundation of China (No. 11171367) and the Fundamental Research Funds for the Central Universities, China
- DOI：10.1631/jzus.C1200008
  中图分类号： TP18
- 收稿：2012-01-11，
  
  修回：2012-;7-;6，
  
  纸质出版：2012-08
- Accepted：
Scan QR Code
Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. 信息与电子工程前沿（英文）, 2012,13(8):585-592.

Yan LIU, Jie YANG, Long LI, et al. Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. Frontiers of Information Technology & Electronic Engineering, 2012, 13(8): 585-592.
Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. 信息与电子工程前沿（英文）, 2012,13(8):585-592. DOI： 10.1631/jzus.C1200008.

Yan LIU, Jie YANG, Long LI, et al. Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. Frontiers of Information Technology & Electronic Engineering, 2012, 13(8): 585-592. DOI： 10.1631/jzus.C1200008.

摘要

Abstract

In the training of feedforward neural networks

it is usually suggested that the initial weights should be small in magnitude in order to prevent premature saturation. The aim of this paper is to point out the other side of the story: In some cases

the gradient of the error functions is zero not only for infinitely large weights but also for zero weights. Slow convergence in the beginning of the training procedure is often the result of sufficiently small initial weights. Therefore

we suggest that

in these cases

the initial values of the weights should be neither too large

nor too small. For instance

a typical range of choices of the initial weights might be something like

$$(-0.4

-0.1)\cup (0.1

0.4)$$

rather than

$$(-0.1

0.1)$$

as suggested by the usual strategy. Our theory that medium size weights should be used has also been extended to a few commonly used transfer functions and error functions. Numerical experiments are carried out to support our theoretical findings.

关键词

Keywords

references

J. Bao , , , Y. Chen , , , J. Yu . . A regeneratable dynamic differential evolution algorithm for neural networks with integer weights . . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , , 2010 . . 11 ( ( 12 ): ): 939 - - 947 . . DOI: 10.1631/jzus.C1000137 http://doi.org/10.1631/jzus.C1000137 . .

P. Biswajeet , , , P. Saied . . Comparison between prediction capabilities of neural network and fuzzy logic techniques for landslide susceptibility mapping . . Dis. Adv. , , 2010 . . 3 ( ( 2 ): ): 26 - - 34 . . . .

P.A. Castillo , , , J. Carpio , , , J.J. Merelo , , , A. Prieto , , , V. Rivas , , , G. Romero . . Evolving multilayer perceptrons . . Neur. Process. Lett. , , 2000 . . 12 ( ( 2 ): ): 115 - - 128 . . DOI: 10.1023/A:1009684907680 http://doi.org/10.1023/A:1009684907680 . .

Y. Deng , , , X. He , , , J. Zhao , , , Y. Xiong , , , Y. Shen , , , J. Jiang . . Application of artificial neural network for switching loss modeling in power IGBTs . . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , , 2010 . . 11 ( ( 6 ): ): 435 - - 443 . . DOI: 10.1631/jzus.C0910442 http://doi.org/10.1631/jzus.C0910442 . .

G.P. Drago , , , S. Ridella . . Statistically controlled activation weight initialization . . IEEE Trans. Neur. Networks , , 1992 . . 3 ( ( 4 ): ): 627 - - 631 . . DOI: 10.1109/72.143378 http://doi.org/10.1109/72.143378 . .

J.L. Elman . . Learning and development in neural networks: the importance of starting small . . Cognition , , 1993 . . 48 ( ( 1 ): ): 71 - - 99 . . DOI: 10.1016/0010-0277(93)90058-4 http://doi.org/10.1016/0010-0277(93)90058-4 . .

M.T. Hagan , , , H.B. Demuth , , , M. Beale . . Neural Network Design . . PWS Publishing Company, Boston, MA. , , 1996 . . .

F.M. Ham , , , I. Kostanic . . Principles of Neurocomputing for Science and Engineering , , McGraw-Hill , , : : New York 2001 . . .

T. Kathirvalavakumar , , , P. Thangavel . . A new learning algorithm using simultaneous perturbation with weight initialization . . Neur. Process. Lett. , , 2003 . . 17 ( ( 1 ): ): 55 - - 68 . . DOI: 10.1023/A:1022919300793 http://doi.org/10.1023/A:1022919300793 . .

Z. Li , , , W. Wu , , , H. Zhang . . Convergence of on-line gradient methods for two-layer feedforward neural networks . . J. Math. Res. Exp. , , 2001 . . 21 219 - - 228 . . . .

M. Liu , , , M. Zhang , , , G. Yan . . A new neural network model for the feedback stabilization of nonlinear systems . . J. Zhejiang Univ.-Sci. A , , 2008 . . 9 ( ( 8 ): ): 1015 - - 1023 . . DOI: 10.1631/jzus.A0720122 http://doi.org/10.1631/jzus.A0720122 . .

T.B. Ludermir , , , A. Yamazaki , , , C. Zanchettin . . An optimization methodology for neural network weights and architectures . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 6 ): ): 1452 - - 1459 . . DOI: 10.1109/TNN.2006.881047 http://doi.org/10.1109/TNN.2006.881047 . .

B. Pradhan . . An assessment of the use of an advanced neural network model with five different training strategies for the preparation of landslide susceptibility maps . . J. Data Sci. , , 2011 . . 9 ( ( 1 ): ): 65 - - 81 . . . .

B. Pradhan , , , M.F. Buchroithner . . Comparison and validation of landslide susceptibility maps using an artificial neural network model for three test areas in Malaysia . . Envir. Eng. Geosci. , , 2010 . . 16 ( ( 2 ): ): 107 - - 126 . . DOI: 10.2113/gseegeosci.16.2.107 http://doi.org/10.2113/gseegeosci.16.2.107 . .

B. Pradhan , , , A.M. Youssef , , , R. Varathrajoo . . Approaches for delineating landslide hazard areas using different training sites in an advanced artificial neural network model . . Geo-spat. Inf. Sci. , , 2010 . . 13 ( ( 2 ): ): 93 - - 102 . . DOI: 10.1007/s11806-010-0236-7 http://doi.org/10.1007/s11806-010-0236-7 . .

H. Qi , , , H. Zhao , , , W. Liu , , , H. Zhang . . Parameters optimization and nonlinearity analysis of grating eddy current displacement sensor using neural network and genetic algorithm . . J. Zhejiang Univ.-Sci. A , , 2009 . . 10 ( ( 8 ): ): 1205 - - 1212 . . DOI: 10.1631/jzus.A0820564 http://doi.org/10.1631/jzus.A0820564 . .

J. Wang , , , W. Wu , , , J.M. Zurada . . Deterministic convergence of conjugate gradient method for feedforward neural networks . . Neurocomputing , , 2011 . . 74 ( ( 14-15 ): ): 2368 - - 2376 . . DOI: 10.1016/j.neucom.2011.03.016 http://doi.org/10.1016/j.neucom.2011.03.016 . .

W. Wu , , , G. Feng , , , Z. Li , , , Y. Xu . . Deterministic convergence of an online gradient method for BP neural networks . . IEEE Trans. Neur. Networks , , 2005 . . 16 ( ( 3 ): ): 533 - - 540 . . DOI: 10.1109/TNN.2005.844903 http://doi.org/10.1109/TNN.2005.844903 . .

Y. Xiong , , , W. Wu , , , X. Kang , , , C. Zhang . . Training pi-sigma network by online gradient algorithm with penalty for small weight update . . Neur. Comput. , , 2007 . . 19 ( ( 12 ): ): 3356 - - 3368 . . DOI: 10.1162/neco.2007.19.12.3356 http://doi.org/10.1162/neco.2007.19.12.3356 . .

J.Y.F. Yam , , , T.W.S. Chow . . Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients . . IEEE Trans. Neur. Networks , , 2001 . . 12 ( ( 2 ): ): 430 - - 434 . . DOI: 10.1109/72.914538 http://doi.org/10.1109/72.914538 . .

Y.F. Yam , , , T.W.S. Chow , , , C.T. Leung . . A new method in determining initial weights of feedforward neural networks for training enhancement . . Neurocomputing , , 1997 . . 16 ( ( 1 ): ): 23 - - 32 . . DOI: 10.1016/S0925-2312(96)00058-6 http://doi.org/10.1016/S0925-2312(96)00058-6 . .

S.S. Yang , , , S. Siu , , , C.L. Ho . . Analysis of the initial values in split-complex backpropagation algorithm . . IEEE Trans. Neur. Networks , , 2008 . . 19 ( ( 9 ): ): 1564 - - 1573 . . DOI: 10.1109/TNN.2008.2000805 http://doi.org/10.1109/TNN.2008.2000805 . .

X.Q. Zeng , , , Y.F. Wang , , , K. Zhang . . Computation of Adalineso sensitivity to weight perturbation . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 2 ): ): 515 - - 519 . . DOI: 10.1109/TNN.2005.863418 http://doi.org/10.1109/TNN.2005.863418 . .

N. Zhang , , , W. Wu , , , G. Zheng . . Convergence of gradient method with momentum for two-layer feedforward neural networks . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 2 ): ): 522 - - 525 . . DOI: 10.1109/TNN.2005.863460 http://doi.org/10.1109/TNN.2005.863460 . .

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Integrated communication and localization in millimeter-wave systems

A sparse matrix model-based optical proximity correction algorithm with model-based mapping between segments and control sites

A regeneratable dynamic differential evolution algorithm for neural networks with integer weights

Studying pressure sores through illuminant invariant assessment of digital color images