Negative effects of sufficiently small initial weights on back-propagation neural networks

Yan LIU; Jie YANG; Long LI; Wei WU

doi:10.1631/jzus.C1200008

Your Location：

Home >

Browse articles >

Negative effects of sufficiently small initial weights on back-propagation neural networks

Computer & Automation | Updated：2022-12-09

- Negative effects of sufficiently small initial weights on back-propagation neural networks
- Negative effects of sufficiently small initial weights on back-propagation neural networks
- Frontiers of Information Technology & Electronic Engineering Vol. 13, Issue 8, Pages: 585-592(2012)
- Affiliations：
  
  1.School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, China
  2.School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China
  3.Department of Mathematics and Computational Science, Hengyang Normal University, Hengyang 421002, China
- Author bio：
  
  [ "Yan LIU, E-mail: liuyan@dlpu.edu.cn" ]
  [ "Jie YANG, E-mail: yangjiee@dlut.edu.cn" ]
  [ "Long LI, E-mail: long_li1982@163.com" ]
  Wei WU, E-mail: wuweiw@dlut.edu.cn
- Funds：
- DOI：10.1631/jzus.C1200008
  CLC： TP18
- Received：11 January 2012，
  
  Revised：2012-;7-;6，
  
  Published：2012-08
- Accepted：
Scan QR Code
Yan LIU, Jie YANG, Long LI, et al. Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. Frontiers of Information Technology & Electronic Engineering, 2012, 13(8): 585-592.
DOI：

Yan LIU, Jie YANG, Long LI, et al. Negative effects of sufficiently small initial weights on back-propagation neural networks[J]. Frontiers of Information Technology & Electronic Engineering, 2012, 13(8): 585-592. DOI： 10.1631/jzus.C1200008.

摘要

Abstract

In the training of feedforward neural networks

it is usually suggested that the initial weights should be small in magnitude in order to prevent premature saturation. The aim of this paper is to point out the other side of the story: In some cases

the gradient of the error functions is zero not only for infinitely large weights but also for zero weights. Slow convergence in the beginning of the training procedure is often the result of sufficiently small initial weights. Therefore

we suggest that

in these cases

the initial values of the weights should be neither too large

nor too small. For instance

a typical range of choices of the initial weights might be something like

$$(-0.4

-0.1)\cup (0.1

0.4)$$

rather than

$$(-0.1

0.1)$$

as suggested by the usual strategy. Our theory that medium size weights should be used has also been extended to a few commonly used transfer functions and error functions. Numerical experiments are carried out to support our theoretical findings.

关键词

Keywords

references

J. Bao , , , Y. Chen , , , J. Yu . . A regeneratable dynamic differential evolution algorithm for neural networks with integer weights . . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , , 2010 . . 11 ( ( 12 ): ): 939 - - 947 . . DOI: 10.1631/jzus.C1000137 http://doi.org/10.1631/jzus.C1000137 . .

P. Biswajeet , , , P. Saied . . Comparison between prediction capabilities of neural network and fuzzy logic techniques for landslide susceptibility mapping . . Dis. Adv. , , 2010 . . 3 ( ( 2 ): ): 26 - - 34 . . . .

P.A. Castillo , , , J. Carpio , , , J.J. Merelo , , , A. Prieto , , , V. Rivas , , , G. Romero . . Evolving multilayer perceptrons . . Neur. Process. Lett. , , 2000 . . 12 ( ( 2 ): ): 115 - - 128 . . DOI: 10.1023/A:1009684907680 http://doi.org/10.1023/A:1009684907680 . .

Y. Deng , , , X. He , , , J. Zhao , , , Y. Xiong , , , Y. Shen , , , J. Jiang . . Application of artificial neural network for switching loss modeling in power IGBTs . . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , , 2010 . . 11 ( ( 6 ): ): 435 - - 443 . . DOI: 10.1631/jzus.C0910442 http://doi.org/10.1631/jzus.C0910442 . .

G.P. Drago , , , S. Ridella . . Statistically controlled activation weight initialization . . IEEE Trans. Neur. Networks , , 1992 . . 3 ( ( 4 ): ): 627 - - 631 . . DOI: 10.1109/72.143378 http://doi.org/10.1109/72.143378 . .

J.L. Elman . . Learning and development in neural networks: the importance of starting small . . Cognition , , 1993 . . 48 ( ( 1 ): ): 71 - - 99 . . DOI: 10.1016/0010-0277(93)90058-4 http://doi.org/10.1016/0010-0277(93)90058-4 . .

M.T. Hagan , , , H.B. Demuth , , , M. Beale . . Neural Network Design . . PWS Publishing Company, Boston, MA. , , 1996 . . .

F.M. Ham , , , I. Kostanic . . Principles of Neurocomputing for Science and Engineering , , McGraw-Hill , , : : New York 2001 . . .

T. Kathirvalavakumar , , , P. Thangavel . . A new learning algorithm using simultaneous perturbation with weight initialization . . Neur. Process. Lett. , , 2003 . . 17 ( ( 1 ): ): 55 - - 68 . . DOI: 10.1023/A:1022919300793 http://doi.org/10.1023/A:1022919300793 . .

Z. Li , , , W. Wu , , , H. Zhang . . Convergence of on-line gradient methods for two-layer feedforward neural networks . . J. Math. Res. Exp. , , 2001 . . 21 219 - - 228 . . . .

M. Liu , , , M. Zhang , , , G. Yan . . A new neural network model for the feedback stabilization of nonlinear systems . . J. Zhejiang Univ.-Sci. A , , 2008 . . 9 ( ( 8 ): ): 1015 - - 1023 . . DOI: 10.1631/jzus.A0720122 http://doi.org/10.1631/jzus.A0720122 . .

T.B. Ludermir , , , A. Yamazaki , , , C. Zanchettin . . An optimization methodology for neural network weights and architectures . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 6 ): ): 1452 - - 1459 . . DOI: 10.1109/TNN.2006.881047 http://doi.org/10.1109/TNN.2006.881047 . .

B. Pradhan . . An assessment of the use of an advanced neural network model with five different training strategies for the preparation of landslide susceptibility maps . . J. Data Sci. , , 2011 . . 9 ( ( 1 ): ): 65 - - 81 . . . .

B. Pradhan , , , M.F. Buchroithner . . Comparison and validation of landslide susceptibility maps using an artificial neural network model for three test areas in Malaysia . . Envir. Eng. Geosci. , , 2010 . . 16 ( ( 2 ): ): 107 - - 126 . . DOI: 10.2113/gseegeosci.16.2.107 http://doi.org/10.2113/gseegeosci.16.2.107 . .

B. Pradhan , , , A.M. Youssef , , , R. Varathrajoo . . Approaches for delineating landslide hazard areas using different training sites in an advanced artificial neural network model . . Geo-spat. Inf. Sci. , , 2010 . . 13 ( ( 2 ): ): 93 - - 102 . . DOI: 10.1007/s11806-010-0236-7 http://doi.org/10.1007/s11806-010-0236-7 . .

H. Qi , , , H. Zhao , , , W. Liu , , , H. Zhang . . Parameters optimization and nonlinearity analysis of grating eddy current displacement sensor using neural network and genetic algorithm . . J. Zhejiang Univ.-Sci. A , , 2009 . . 10 ( ( 8 ): ): 1205 - - 1212 . . DOI: 10.1631/jzus.A0820564 http://doi.org/10.1631/jzus.A0820564 . .

J. Wang , , , W. Wu , , , J.M. Zurada . . Deterministic convergence of conjugate gradient method for feedforward neural networks . . Neurocomputing , , 2011 . . 74 ( ( 14-15 ): ): 2368 - - 2376 . . DOI: 10.1016/j.neucom.2011.03.016 http://doi.org/10.1016/j.neucom.2011.03.016 . .

W. Wu , , , G. Feng , , , Z. Li , , , Y. Xu . . Deterministic convergence of an online gradient method for BP neural networks . . IEEE Trans. Neur. Networks , , 2005 . . 16 ( ( 3 ): ): 533 - - 540 . . DOI: 10.1109/TNN.2005.844903 http://doi.org/10.1109/TNN.2005.844903 . .

Y. Xiong , , , W. Wu , , , X. Kang , , , C. Zhang . . Training pi-sigma network by online gradient algorithm with penalty for small weight update . . Neur. Comput. , , 2007 . . 19 ( ( 12 ): ): 3356 - - 3368 . . DOI: 10.1162/neco.2007.19.12.3356 http://doi.org/10.1162/neco.2007.19.12.3356 . .

J.Y.F. Yam , , , T.W.S. Chow . . Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients . . IEEE Trans. Neur. Networks , , 2001 . . 12 ( ( 2 ): ): 430 - - 434 . . DOI: 10.1109/72.914538 http://doi.org/10.1109/72.914538 . .

Y.F. Yam , , , T.W.S. Chow , , , C.T. Leung . . A new method in determining initial weights of feedforward neural networks for training enhancement . . Neurocomputing , , 1997 . . 16 ( ( 1 ): ): 23 - - 32 . . DOI: 10.1016/S0925-2312(96)00058-6 http://doi.org/10.1016/S0925-2312(96)00058-6 . .

S.S. Yang , , , S. Siu , , , C.L. Ho . . Analysis of the initial values in split-complex backpropagation algorithm . . IEEE Trans. Neur. Networks , , 2008 . . 19 ( ( 9 ): ): 1564 - - 1573 . . DOI: 10.1109/TNN.2008.2000805 http://doi.org/10.1109/TNN.2008.2000805 . .

X.Q. Zeng , , , Y.F. Wang , , , K. Zhang . . Computation of Adalineso sensitivity to weight perturbation . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 2 ): ): 515 - - 519 . . DOI: 10.1109/TNN.2005.863418 http://doi.org/10.1109/TNN.2005.863418 . .

N. Zhang , , , W. Wu , , , G. Zheng . . Convergence of gradient method with momentum for two-layer feedforward neural networks . . IEEE Trans. Neur. Networks , , 2006 . . 17 ( ( 2 ): ): 522 - - 525 . . DOI: 10.1109/TNN.2005.863460 http://doi.org/10.1109/TNN.2005.863460 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Integrated communication and localization in millimeter-wave systems

A sparse matrix model-based optical proximity correction algorithm with model-based mapping between segments and control sites

A regeneratable dynamic differential evolution algorithm for neural networks with integer weights

Studying pressure sores through illuminant invariant assessment of digital color images

Related Author

Wei XIONG

Yaqi CUI

Pingliang XU

Jing XU

Bo GAO

Jie YANG

Xiao LI

Shi JIN

Related Institution

Institute of Information Fusion, Naval Aviation University

ZTE Corporation

State Key Laboratory of Mobile Network and Mobile Multimedia Technology

National Mobile Communications Research Laboratory, Southeast University

Institute of VLSI Design, Zhejiang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰