A pruning algorithm with <italic style="font-style: italic">L</italic><sub>1/2</sub> regularizer for extreme learning machine

Ye-tian FAN; Wei WU; Wen-yu YANG; Qin-wei FAN; Jian WANG

doi:10.1631/jzus.C1300197

Your Location：

Home >

Browse articles >

A pruning algorithm with L_1/2 regularizer for extreme learning machine

Computer & Automation | Updated：2022-05-19

- A pruning algorithm with L_1/2 regularizer for extreme learning machine
- 利用L1/2正则化进行极端学习机的网络修剪算法
- Frontiers of Information Technology & Electronic Engineering Vol. 15, Issue 2, Pages: 119-125(2014)
- Affiliations：
  
  1.School of Mathematical Sciences, Dalian University of Technology, Dalian 116023, China
  2.College of Science, Huazhong Agricultural University, Wuhan 430070, China
- Author bio：
  
  [ "Ye-tian FAN, fanyetian@mail.dlut.edu.cn" ]
  Wei WU, wuweiw@dlut.edu.cn
- Funds：
- DOI：10.1631/jzus.C1300197
  CLC： TP312
- Received：22 July 2013，
  
  Revised：2014-1-15，
  
  Accepted：14 October 2013，
  
  Published：2014-02
- Accepted：
Scan QR Code
Ye-tian FAN, Wei WU, Wen-yu YANG, et al. A pruning algorithm with L_1/2 regularizer for extreme learning machine[J]. Frontiers of Information Technology & Electronic Engineering, 2014, 15(2): 119-125.
DOI：

Ye-tian FAN, Wei WU, Wen-yu YANG, et al. A pruning algorithm with L_1/2 regularizer for extreme learning machine[J]. Frontiers of Information Technology & Electronic Engineering, 2014, 15(2): 119-125. DOI： 10.1631/jzus.C1300197.

摘要

研究背景

1. 神经网络有着广泛的应用，但收敛速度慢、精度低，影响了它的发展。相较于传统的神经网络，极端学习机克服了这些缺点，它不仅提供更快的学习速度，而且只需较少的人工干预，这些优点使得极端学习机得到了广泛应用。2. 相比于L1和L2正则化，L1/2正则化的解具有更好的稀疏性；而与L0正则化相比，它又更容易求解。

创新要点

将L1/2正则化方法与极端学习机结合，利用L1/2正则化较好的稀疏性，修剪极端学习机的网络结构。

方法提亮

极小化的目标函数中含有L1/2范数，当权值变得较小时，其导数值会较大。为了阻止权值过快增长，提出一个可变学习率。

重要结论

数据实验表明，相比于原始的极端学习机算法和带L2正则化的极端学习机算法，带L1/2正则化的极端学习机算法不仅拥有较少隐节点，并且拥有更好泛化能力。

Abstract

Compared with traditional learning methods such as the back propagation (BP) method

extreme learning machine provides much faster learning speed and needs less human intervention

and thus has been widely used. In this paper we combine the

1/2

regularization method with extreme learning machine to prune extreme learning machine. A variable learning coefficient is employed to prevent too large a learning increment. A numerical experiment demonstrates that a network pruned by

1/2

regularization has fewer hidden nodes but provides better performance than both the original network and the network pruned by

regularization.

关键词

Keywords

references

C.M. Bishop . . Neural Networks for Pattern Recognition , , : : UK Oxford University Press , , 1995 . . .

S. Chen , , , D.L. Donoho , , , M.A. Saunders . . Atomic decomposition by basis pursuit . . SIAM Rev. , , 2001 . . 43 ( ( 1 ): ): 129 - - 159 . . DOI: 10.1137/S003614450037906X http://doi.org/10.1137/S003614450037906X . .

D.L. Donoho , , , X. Huo . . Uncertainty principles and ideal atomic decomposition . . IEEE Trans. Inform. Theory , , 2001 . . 47 ( ( 7 ): ): 2845 - - 2862 . . DOI: 10.1109/18.959265 http://doi.org/10.1109/18.959265 . .

P. Horata , , , S. Chiewchanwattana , , , K. Sunat . . Robust extreme learning machine . . Neurocomputing , , 2013 . . 102 31 - - 44 . . DOI: 10.1016/j.neucom.2011.12.045 http://doi.org/10.1016/j.neucom.2011.12.045 . .

K. Hornik , , , M. Stinchcombe , , , H. White . . Multilayer feedforward networks are universal approximators . . Neur. Networks , , 1989 . . 2 ( ( 5 ): ): 359 - - 366 . . DOI: 10.1016/0893-6080(89)90020-8 http://doi.org/10.1016/0893-6080(89)90020-8 . .

G.B. Huang , , , C.K. Siew . . Extreme learning machine: RBF network case . . Proc. 8th Int. Conf. on Control, Automation, Robotics and Vision , , 2004 . . p.1029 - - 1036 . . DOI: 10.1109/ICARCV.2004.1468985 http://doi.org/10.1109/ICARCV.2004.1468985 . .

G.B. Huang , , , Q.Y. Zhu , , , C.K. Siew . . Extreme learning machine: a new learning scheme of feedforward neural networks . . Proc. IEEE Int. Joint Conf. on Neural Networks , , 2004 . . p.985 - - 990 . . DOI: 10.1109/IJCNN.2004.1380068 http://doi.org/10.1109/IJCNN.2004.1380068 . .

G.B. Huang , , , Q.Y. Zhu , , , C.K. Siew . . Extreme learning machine: theory and applications . . Neurocomputing , , 2006 . . 70 ( ( 1-3 ): ): 489 - - 501 . . DOI: 10.1016/j.neucom.2005.12.126 http://doi.org/10.1016/j.neucom.2005.12.126 . .

G.B. Huang , , , D.H. Wang , , , Y. Lan . . Extreme learning machines: a survey . . Int. J. Mach. Learn. Cybern. , , 2011 . . 2 ( ( 2 ): ): 107 - - 122 . . DOI: 10.1007/s13042-011-0019-y http://doi.org/10.1007/s13042-011-0019-y . .

Y. Miche , , , A. Sorjamaa , , , P. Bas , , , 等 . . OP-ELM: optimally pruned extreme learning machine . . IEEE Trans. Neur. Networks , , 2010 . . 21 ( ( 1 ): ): 158 - - 162 . . DOI: 10.1109/TNN.2009.2036259 http://doi.org/10.1109/TNN.2009.2036259 . .

Y. Miche , , , M. van Heeswijk , , , P. Bas , , , 等 . . TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization . . Neurocomputing , , 2011 . . 74 ( ( 16 ): ): 2413 - - 2421 . . DOI: 10.1016/j.neucom.2010.12.042 http://doi.org/10.1016/j.neucom.2010.12.042 . .

C.E. Rasmussen , , , C.K.I. Williams . . Gaussian Processes for Machine Learning , , : : Cambridge, MA MIT Press , , 2006 . . .

E. Soria-Olivas , , , J. Gomez-Sanchis , , , I.H. Jarman , , , 等 . . BELM: Bayesian extreme learning machine . . IEEE Trans. Neur. Networks , , 2011 . . 22 ( ( 3 ): ): 505 - - 509 . . DOI: 10.1109/TNN.2010.2103956 http://doi.org/10.1109/TNN.2010.2103956 . .

R. Tibshirani . . Regression shrinkage and selection via the Lasso . . J. R. Stat. Soc. B , , 1996 . . 58 ( ( 1 ): ): 267 - - 288 . . . .

W. Wu . . Computation of Neural Networks , , : : Beijing Higher Education Press , , 2003 . . .

G.B. Xu , , , D.H. Zhou . . Fault prediction for state-dependent fault based on online learning neural network . . J. Zhejiang Univ. (Eng. Sci.) , , 2010 . . 44 ( ( 7 ): ): 1251 - - 1254 . . . .

Z.B. Xu . . Data modeling: visual psychology approach and L 1/2 regularization theory . . Proc. Int. Congress of Mathmaticians , , 2010 . . p.3151 - - 3184 . . . .

Z.B. Xu , , , H. Zhang , , , Y. Wang , , , 等 . . L 1/2 regularization . . Sci. China Inform. Sci. , , 2010 . . 53 ( ( 6 ): ): 1159 - - 1169 . . DOI: 10.1007/s11432-010-0090-0 http://doi.org/10.1007/s11432-010-0090-0 . .

G.F. Yan , , , S.L. Zhang , , , M.Q. Liu . . Standard neural network model and its application . . J. Zhejiang Univ. (Eng. Sci.) , , 2004 . . 38 ( ( 3 ): ): 297 - - 301 . . . .

W.B. Zhang , , , H.B. Ji . . Fuzzy extreme learning machine for classification . . Electron. Lett. , , 2013 . . 49 ( ( 7 ): ): 448 - - 450 . . DOI: 10.1049/el.2012.3642 http://doi.org/10.1049/el.2012.3642 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰

A pruning algorithm with L1/2 regularizer for extreme learning machine

DOI：10.1631/jzus.C1300197

摘要

Abstract

关键词

Keywords

references

A pruning algorithm with L_1/2 regularizer for extreme learning machine