Topic modeling for large-scale text data

LI Xi-ming; OUYANG Ji-hong; LU You

doi:10.1631/FITEE.1400352

Your Location：

Home >

Browse articles >

Topic modeling for large-scale text data

Computer & Automation | Updated：2022-06-06

- Topic modeling for large-scale text data
- 大规模文本数据的主题建模
- Frontiers of Information Technology & Electronic Engineering Vol. 16, Issue 6, Pages: 457-465(2015)
- Affiliations：
  
  1.College of Computer Science and Technology, Jilin University, Changchun 130012, China
  2.MOE Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University, Changchun 130012, China
- Author bio：
  
  [ "Xi-ming LI, E-mail:liximing86@gmail.com" ]
  [ "Ji-hong OUYANG, E-mail:ouyj@jlu.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1400352
  CLC： TP391.1
- Received：15 October 2014，
  
  Revised：2015-05-07，
  
  Published：2015-06
- Accepted：
Scan QR Code
LI Xi-ming, OUYANG Ji-hong, LU You. Topic modeling for large-scale text data[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(6): 457-465.
DOI：

LI Xi-ming, OUYANG Ji-hong, LU You. Topic modeling for large-scale text data[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(6): 457-465. DOI： 10.1631/FITEE.1400352.

摘要

目的

研究大规模数据的主题模型在线推理算法，针对随机变分推理算法中随机梯度误差较大的问题，提出一种移动平均随机变分推理算法。

创新点

使用多次迭代的随机梯度移动平均值近似代替真实随机梯度，以此减小随机梯度和真实梯度间的误差。

方法

以主题模型的基础模型潜在狄利克雷分配为载体展开研究。考虑不同次迭代的文本子集具有不同的词汇（表1），使用不同次迭代的随机项移动平均值近似代替真实随机梯度的随机项。为尽可能保证算法的精度，使用最近

次迭代的随机项（图2）并验证所提算法的收敛性。

结论

在随机变分推理算法基础上，提出一种移动平均随机变分推理算法，实现更好的文本主题建模效果和更快的收敛速度。

Abstract

This paper develops a novel online algorithm

namely moving average stochastic variational inference (MASVI)

which applies the results obtained by previous iterations to smooth out noisy natural gradients. We analyze the convergence property of the proposed algorithm and conduct a set of experiments on two large-scale collections that contain millions of documents. Experimental results indicate that in contrast to algorithms named 'stochastic variational inference' and 'SGRLD'

our algorithm achieves a faster convergence rate and better performance.

关键词

Keywords

references

S. Amari . Natural gradient works efficiently in learning . Neur. Comput. , 1998 . 10 ( 2 ): 251 - 276 . DOI: 10.1162/089976698300017746 http://doi.org/10.1162/089976698300017746 .

C. Andrieu , N. de Freitas , A. Doucet , 等 . An introduction to MCMC for machine learning . Mach. Learn. , 2003 . 50 ( 1-2 ): 5 - 43 . DOI: 10.1023/A:1020281327116 http://doi.org/10.1023/A:1020281327116 .

D. Blatt , A.O. Hero , H. Gauchman . A convergent incremental gradient method with a constant step size . SIAM J. Optim. , 2007 . 18 ( 1 ): 29 - 51 . DOI: 10.1137/040615961 http://doi.org/10.1137/040615961 .

D.M. Blei . Probabilistic topic models . Commun. ACM , 2012 . 55 ( 4 ): 77 - 84 . DOI: 10.1145/2133806.2133826 http://doi.org/10.1145/2133806.2133826 .

D.M. Blei , A.Y. Ng , M.I. Jordan . Latent Dirichlet allocation . J. Mach. Learn. Res. , 2003 . 3 993 - 1022 . .

K.R. Canini , L. Shi , T.L. Griffiths . Online inference of topics with latent Dirichlet allocation . J. Mach. Learn. Res. , 2009 . 5 ( 2 ): 65 - 72 . .

T.L. Griffiths , M. Steyvers . Finding scientific topics . PNAS , 2004 . 101 ( suppl 1 ): 5228 - 5235 . DOI: 10.1073/pnas.0307752101 http://doi.org/10.1073/pnas.0307752101 .

M. Hoffman , F.R. Bach , D.M. Blei . Online learning for latent Dirichlet allocation . Advances in Neural Information Processing Systems , 2010 . p.856 - 864 . .

M. Hoffman , D.M. Blei , C. Wang , 等 . Stochastic variational inference . J. Mach. Learn. Res. , 2013 . 14 ( 1 ): 1303 - 1347 . .

Z. Liu , Y. Zhang , E.Y. Chang , 等 . PLDA+: parallel latent Dirichlet allocation with data placement and pipeline processing . ACM Trans. Intell. Syst. Technol. , 2011 . 2 ( 3 ): Article 26 .

D. Newman , A. Asuncion , P. Smyth , 等 . Distributed algorithms for topic models . J. Mach. Learn. Res. , 2009 . 10 1801 - 1828 . .

J. Ouyang , Y. Lu , X. Li . Momentum online LDA for large-scale datasets . Proc. 21st European Conf. on Artificial Intelligence , 2014 . p.1075 - 1076 . .

S. Patterson , Y.W. Teh . Stochastic gradient Riemannian Langevin dynamics on the probability simplex . Advances in Neural Information Processing Systems , 2013 . p.3102 - 3110 . .

R. Ranganath , C. Wang , D.M. Blei , 等 . An adaptive learning rate for stochastic variational inferencen . J. Mach. Learn. Res. , 2013 . 28 ( 2 ): 298 - 306 . .

T. Schaul , S. Zhang , Y. LeCun . No more pesky learning rates . arXiv preprint , 2013 . arXiv:1206 1106v2 .

X. Song , C.Y. Lin , B.L. Tseng , 等 . Modeling and predicting personal information dissemination behavior . Proc. 11th ACM SIGKDD Int. Conf. on Knowledge Discovery in Data Mining , 2005 . p.479 - 488 . DOI: 10.1145/1081870.1081925 http://doi.org/10.1145/1081870.1081925 .

V.B. Tadić . Convergence rate of stochastic gradient search in the case of multiple and non-isolated minima . arXiv preprint , 2009 . arXiv:0904 4229v2 .

Y.W. Teh , D. Newman , M. Welling . A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation . Advances in Neural Information Processing Systems , 2007 . p.1353 - 1360 . .

C. Wang , X. Chen , A.J. Smola , 等 . Variance reduction for stochastic gradient optimization . Advances in Neural Information Processing Systems , 2013 . p.181 - 189 . .

Y. Wang , H. Bai , M. Stanton , 等 . PLDA: parallel latent Dirichlet allocation for large-scale applications . Proc. 5th Int. Conf. on Algorithmic Aspects in Information and Management , 2009 . p.301 - 314 . DOI: 10.1007/978-3-642-02158-9_26 http://doi.org/10.1007/978-3-642-02158-9_26 .

F. Yan , N. Xu , Y. Qi . Parallel inference for latent Dirichlet allocation on graphics processing units . Advances in Neural Information Processing Systems , 2009 . p.2134 - 2142 . .

Y. Ye , S. Gong , C. Liu , 等 . Online belief propagation algorithm for probabilistic latent semantic analysis . Front. Comput. Sci. , 2013 . 7 ( 5 ): 526 - 535 . DOI: 10.1007/s11704-013-2360-7 http://doi.org/10.1007/s11704-013-2360-7 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Hybrid-driven Gaussian process online learning for highly maneuvering multi-target tracking

Related Author

Qiang GUO

Long TENG

Tianxiang YIN

Yunfei GUO

Xinliang WU

Wenming SONG

Related Institution

College of Information and Communication Engineering, Harbin Engineering University

China National Aeronautical Radio Electronics Research Institute

College of Automation, Hangzhou Dianzi University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰