基于依存关系和多义词分析的句法词嵌入

冶忠林; 赵海兴

doi:10.1631/FITEE.1601846

Your Location：

Home >

Browse articles >

基于依存关系和多义词分析的句法词嵌入

常规文章 | Updated：2022-05-19

- 基于依存关系和多义词分析的句法词嵌入
- Syntactic word embedding based on dependency syntax and polysemous analysis
- 信息与电子工程前沿（英文） 2018年19卷第4期页码：524-535
- Affiliations：
  
  School of Computer Science, Shaanxi Normal University, Xi'an 710119, China
  School of Computer, Qinghai Normal University, Xining 810800, China
- Author bio：
  
  Hai-xing ZHAO, E-mail: h.x.zhao@163.com
- Funds：
  
  Project supported by the National Natural Science Foundation of China (Nos. 61663041 and 61763041), the Program for Changjiang Scholars and Innovative Research Team in Universities, China (No. IRT_15R40), the Research Fund for the Chunhui Program of Ministry of Education of China (No. Z2014022), the Natural Science Foundation of Qinghai Province, China (No. 2014-ZJ-721), and the Fundamental Research Funds for the Central Universities, China (No. 2017TS045)
- DOI：10.1631/FITEE.1601846
  中图分类号： TP391
- 收稿：2016-12-21，
  
  修回：2018-;4-12，
  
  纸质出版：2018-04
- Accepted：
Scan QR Code
冶忠林, 赵海兴. 基于依存关系和多义词分析的句法词嵌入[J]. 信息与电子工程前沿（英文）, 2018,19(4):524-535.

Zhong-lin YE, Hai-xing ZHAO. Syntactic word embedding based on dependency syntax and polysemous analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(4): 524-535.
冶忠林, 赵海兴. 基于依存关系和多义词分析的句法词嵌入[J]. 信息与电子工程前沿（英文）, 2018,19(4):524-535. DOI： 10.1631/FITEE.1601846.

Zhong-lin YE, Hai-xing ZHAO. Syntactic word embedding based on dependency syntax and polysemous analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(4): 524-535. DOI： 10.1631/FITEE.1601846.

摘要

现有大多数词嵌入学习模型存在以下问题：（1）基于词袋上下文的模型完全忽略句子的句法结构关系；（2）每个词使用单个嵌入向量使多义词共享一个嵌入向量；（3）词嵌入往往趋向于句子上下文共性。为解决这些问题，提出一种基于依存关系和多义词分析的句法词嵌入（syntactic word embedding

SWE）。该算法主要处理：（1）基于主题模型，提出一个多义词识别算法；（2）采用符号"+"和"?"表示依存关系方向；（3）删除停用词及其依存关系；（4）引入"skip"依存关系表示依存关系之间的间接关系；（5）将基于依存关系的上下文输入到Word2Vec模型中训练语言模型。实验结果表明，SWE模型在词相似度评测任务中表现出优异性能。基于依存关系句法上下文捕获词语的语义和句法特征，使词语表现出较少的上下文主题相似性和更多的句法和语义相似性。综上，包含更多信息的SWE模型性能优于单一的词嵌入学习模型。

Abstract

Most word embedding models have the following problems: (1) In the models based on bag-of-words contexts

the structural relations of sentences are completely neglected; (2) Each word uses a single embedding

which makes the model indiscriminative for polysemous words; (3) Word embedding easily tends to contextual structure similarity of sentences. To solve these problems

we propose an easy-to-use representation algorithm of syntactic word embedding (SWE). The main procedures are: (1) A polysemous tagging algorithm is used for polysemous representation by the latent Dirichlet allocation (LDA) algorithm; (2) Symbols '+' and '−' are adopted to indicate the directions of the dependency syntax; (3) Stopwords and their dependencies are deleted; (4) Dependency skip is applied to connect indirect dependencies; (5) Dependency-based contexts are inputted to a word2vec model. Experimental results show that our model generates desirable word embedding in similarity evaluation tasks. Besides

semantic and syntactic features can be captured from dependency-based syntactic contexts

exhibiting less topical and more syntactic similarity. We conclude that SWE outperforms single embedding learning models.

关键词

Keywords

references

M Baroni , , , A Lenci . . Distributional memory: a general framework for corpus-based semantics . . Comput Ling , , 2010 . . 36 ( ( 4 ): ): 673 - - 721 . . DOI: 10.1162/coli_a_00016 http://doi.org/10.1162/coli_a_00016 . .

Y Bengio , , , R Ducharme , , , P Vincent , , , 等 . . A neural probabilistic language model . . J Mach Learn Res , , 2003 . . 3 ( ( 6 ): ): 1137 - - 1155 . . DOI: 10.1007/3-540-33486-6_6 http://doi.org/10.1007/3-540-33486-6_6 . .

JA Bullinaria , , , JP Levy . . Extracting semantic representations from word co-occurrence statistics: a computational study . . Behav Res Methods , , 2007 . . 39 ( ( 3 ): ): 510 - - 526 . . DOI: 10.3758/BF03193020 http://doi.org/10.3758/BF03193020 . .

L Finkelstein , , , E Gabrilovich , , , Y Matias , , , 等 . . Placing search in context: the concept revisited . . ACM Trans Inform Syst , , 2002 . . 20 ( ( 1 ): ): 116 - - 131 . . DOI: 10.1145/503104.503110 http://doi.org/10.1145/503104.503110 . .

JR Firth . . A synopsis of linguistic theory . . Stud Ling Anal , , 1957 . . 41 ( ( 4 ): ): 1 - - 32 . . . .

Y Goldberg , , , J Nivre . . A dynamic oracle for arc-eager dependency parsing . . Proc Coling , , 2012 . . p.959 - - 976 . . . .

Y Goldberg , , , J Nivre . . Training deterministic parsers with non-deterministic oracles . . Trans Assoc Comput Ling , , 2014 . . p.403 - - 414 . . . .

ZS Harris . . Distributional structure . . Word , , 1981 . . 10 ( ( 2-3 ): ): 146 - - 162 . . DOI: 10.1007/978-94-017-6059-1_36 http://doi.org/10.1007/978-94-017-6059-1_36 . .

F Hill , , , R Reichart , , , A Korhonen . . SimLex-999: evaluating semantic models with (genuine) similarity estimation . . Comput Ling , , 2015 . . 41 ( ( 2 ): ): 665 - - 695 . . DOI: 10.1162/COLI_a_00237 http://doi.org/10.1162/COLI_a_00237 . .

GE Hinton . . Learning distributed representations of concepts . . Proc 8 th Annual Conf of the Cognitive Science Society , , 1986 . . p.1 - - 12 . . . .

EH Huang , , , R Socher , , , CD Manning , , , 等 . . Improving word representations via global context and multiple word prototypes . . Proc 50 th Annual Meeting of Association for Computational Linguistics , , 2012 . . p.873 - - 882 . . . .

K Krishna , , , MN Murty . . Genetic K-means algorithm . . IEEE Trans Syst Man Cybern Part B , , 1999 . . 29 ( ( 3 ): ): 433 - - 439 . . DOI: 10.1109/3477.764879 http://doi.org/10.1109/3477.764879 . .

R Lebret , , , R Collobert . . Word embeddings through Hellinger PCA . . Proc 14 th Conf on European Chapter of the Association for Computational Linguistics , , 2014 . . p.482 - - 490 . . . .

R Lebret , , , R Collobert . . Rehabilitation of count-based models for word vector representations . . Int Conf on Intelligent Text Processing and Computational Linguistics , , 2015 . . p.417 - - 429 . . DOI: 10.1007/978-3-319-18111-0_31 http://doi.org/10.1007/978-3-319-18111-0_31 . .

O Levy , , , Y Goldberg . . Dependency-based word embeddings . . Proc 52 nd Annual Meeting of Association for Computational Linguistics , , 2014 . . p.302 - - 308 . . DOI: 10.3115/v1/P14-2050 http://doi.org/10.3115/v1/P14-2050 . .

Y Liu , , , ZY Liu , , , TS Chua , , , 等 . . Topical word embeddings . . Proc 29 th AAAI Conf on Artificial Intelligence , , 2015 . . p.2418 - - 2424 . . . .

MT Luong , , , R Socher , , , CD Manning . . Better word representations with recursive neural networks for morphology . . Proc 17 th Conf on Computational Natural Language Learning , , 2013 . . p.104 - - 113 . . . .

T Mikolov , , , I Sutskever , , , K Chen , , , 等 . . Distributed representations of words and phrases and their compositionality . . Int Conf on Neural Information Processing Systems , , 2013 . . p.3111 - - 3119 . . . .

A Mnih , , , GE Hinton . . A scalable hierarchical distributed language model . . Proc 21 st Int Conf on Neural Information Processing System , , 2008 . . p.1081 - - 1088 . . . .

KA Nguyen , , , SSI Walde , , , NT Vu . . Neural-based noise filtering from word embeddings . . Proc 26 th Int Conf on Computational Linguistics , , 2016 . . p.2699 - - 2707 . . . .

J Pennington , , , R Socher , , , CD Manning . . Glove: global vectors for word representation . . Proc Conf on Empirical Methods in Natural Language Processing , , 2014 . . p.1532 - - 1543 . . . .

YF Ren , , , RM Wang , , , DH Ji . . A topic-enhanced word embedding for Twitter sentiment classification . . Inform Sci , , 2016 . . 369 188 - - 198 . . DOI: 10.1016/j.ins.2016.06.040 http://doi.org/10.1016/j.ins.2016.06.040 . .

A Ritter , , , Mausam , , , O Etzioni . . A latent Dirichlet allocation method for selectional preferences . . Proc 48 th Annual Meeting of Association for Computational Linguistics , , 2010 . . p.424 - - 434 . . . .

H Rubenstein , , , JB Goodenough . . Contextual correlates of synonymy . . Commun ACM , , 1965 . . 8 ( ( 10 ): ): 627 - - 633 . . DOI: 10.1145/365628.365657 http://doi.org/10.1145/365628.365657 . .

F Tian , , , HJ Dai , , , J Bian , , , 等 . . A probabilistic model for learning multi-prototype word embeddings . . Proc 25 th Int Conf on Computational Linguistics , , 2014 . . p.151 - - 160 . . . .

PD Turney , , , P Pantel . . From frequency to meaning: vector space models of semantics . . J Artif Intell Res , , 2010 . . 37 ( ( 1 ): ): 141 - - 188 . . DOI: 10.1613/jair.2934 http://doi.org/10.1613/jair.2934 . .

P Wang , , , B Xu , , , JM Xu , , , 等 . . Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification . . Neurocomputing , , 2016 . . 174 ( ( B ): ): 806 - - 814 . . DOI: 10.1016/j.neucom.2015.09.096 http://doi.org/10.1016/j.neucom.2015.09.096 . .

W Xu , , , AI Rudnicky . . Can artificial neural networks learn language models . . Proc 6 th Int Conf on Spoken Language Processing , , 2000 . . p.202 - - 205 . . . .

M Zhai , , , J Tan , , , DJ Choi . . Intrinsic and extrinsic evaluations of word embeddings . . Proc 30 th AAAI Conf on Artificial Intelligence , , 2016 . . p.4282 - - 4283 . . . .

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Federated unsupervised representation learning

Learning embeddings of a heterogeneous behavior network for potential behavior prediction