Syntactic word embedding based on dependency syntax and polysemous analysis

Zhong-lin YE; Hai-xing ZHAO

doi:10.1631/FITEE.1601846

Your Location：

Home >

Browse articles >

Syntactic word embedding based on dependency syntax and polysemous analysis

Regular Papers | Updated：2022-05-19

- Syntactic word embedding based on dependency syntax and polysemous analysis
- 基于依存关系和多义词分析的句法词嵌入
- Frontiers of Information Technology & Electronic Engineering Vol. 19, Issue 4, Pages: 524-535(2018)
- Affiliations：
  
  School of Computer Science, Shaanxi Normal University, Xi'an 710119, China
  School of Computer, Qinghai Normal University, Xining 810800, China
- Author bio：
  
  Hai-xing ZHAO, E-mail: h.x.zhao@163.com
- Funds：
- DOI：10.1631/FITEE.1601846
  CLC： TP391
- Received：21 December 2016，
  
  Revised：2018-;4-12，
  
  Published：2018-04
- Accepted：
Scan QR Code
Zhong-lin YE, Hai-xing ZHAO. Syntactic word embedding based on dependency syntax and polysemous analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(4): 524-535.
DOI：

Zhong-lin YE, Hai-xing ZHAO. Syntactic word embedding based on dependency syntax and polysemous analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(4): 524-535. DOI： 10.1631/FITEE.1601846.

摘要

现有大多数词嵌入学习模型存在以下问题：（1）基于词袋上下文的模型完全忽略句子的句法结构关系；（2）每个词使用单个嵌入向量使多义词共享一个嵌入向量；（3）词嵌入往往趋向于句子上下文共性。为解决这些问题，提出一种基于依存关系和多义词分析的句法词嵌入（syntactic word embedding

SWE）。该算法主要处理：（1）基于主题模型，提出一个多义词识别算法；（2）采用符号"+"和"?"表示依存关系方向；（3）删除停用词及其依存关系；（4）引入"skip"依存关系表示依存关系之间的间接关系；（5）将基于依存关系的上下文输入到Word2Vec模型中训练语言模型。实验结果表明，SWE模型在词相似度评测任务中表现出优异性能。基于依存关系句法上下文捕获词语的语义和句法特征，使词语表现出较少的上下文主题相似性和更多的句法和语义相似性。综上，包含更多信息的SWE模型性能优于单一的词嵌入学习模型。

Abstract

Most word embedding models have the following problems: (1) In the models based on bag-of-words contexts

the structural relations of sentences are completely neglected; (2) Each word uses a single embedding

which makes the model indiscriminative for polysemous words; (3) Word embedding easily tends to contextual structure similarity of sentences. To solve these problems

we propose an easy-to-use representation algorithm of syntactic word embedding (SWE). The main procedures are: (1) A polysemous tagging algorithm is used for polysemous representation by the latent Dirichlet allocation (LDA) algorithm; (2) Symbols '+' and '−' are adopted to indicate the directions of the dependency syntax; (3) Stopwords and their dependencies are deleted; (4) Dependency skip is applied to connect indirect dependencies; (5) Dependency-based contexts are inputted to a word2vec model. Experimental results show that our model generates desirable word embedding in similarity evaluation tasks. Besides

semantic and syntactic features can be captured from dependency-based syntactic contexts

exhibiting less topical and more syntactic similarity. We conclude that SWE outperforms single embedding learning models.

关键词

Keywords

references

M Baroni , , , A Lenci . . Distributional memory: a general framework for corpus-based semantics . . Comput Ling , , 2010 . . 36 ( ( 4 ): ): 673 - - 721 . . DOI: 10.1162/coli_a_00016 http://doi.org/10.1162/coli_a_00016 . .

Y Bengio , , , R Ducharme , , , P Vincent , , , 等 . . A neural probabilistic language model . . J Mach Learn Res , , 2003 . . 3 ( ( 6 ): ): 1137 - - 1155 . . DOI: 10.1007/3-540-33486-6_6 http://doi.org/10.1007/3-540-33486-6_6 . .

JA Bullinaria , , , JP Levy . . Extracting semantic representations from word co-occurrence statistics: a computational study . . Behav Res Methods , , 2007 . . 39 ( ( 3 ): ): 510 - - 526 . . DOI: 10.3758/BF03193020 http://doi.org/10.3758/BF03193020 . .

L Finkelstein , , , E Gabrilovich , , , Y Matias , , , 等 . . Placing search in context: the concept revisited . . ACM Trans Inform Syst , , 2002 . . 20 ( ( 1 ): ): 116 - - 131 . . DOI: 10.1145/503104.503110 http://doi.org/10.1145/503104.503110 . .

JR Firth . . A synopsis of linguistic theory . . Stud Ling Anal , , 1957 . . 41 ( ( 4 ): ): 1 - - 32 . . . .

Y Goldberg , , , J Nivre . . A dynamic oracle for arc-eager dependency parsing . . Proc Coling , , 2012 . . p.959 - - 976 . . . .

Y Goldberg , , , J Nivre . . Training deterministic parsers with non-deterministic oracles . . Trans Assoc Comput Ling , , 2014 . . p.403 - - 414 . . . .

ZS Harris . . Distributional structure . . Word , , 1981 . . 10 ( ( 2-3 ): ): 146 - - 162 . . DOI: 10.1007/978-94-017-6059-1_36 http://doi.org/10.1007/978-94-017-6059-1_36 . .

F Hill , , , R Reichart , , , A Korhonen . . SimLex-999: evaluating semantic models with (genuine) similarity estimation . . Comput Ling , , 2015 . . 41 ( ( 2 ): ): 665 - - 695 . . DOI: 10.1162/COLI_a_00237 http://doi.org/10.1162/COLI_a_00237 . .

GE Hinton . . Learning distributed representations of concepts . . Proc 8 th Annual Conf of the Cognitive Science Society , , 1986 . . p.1 - - 12 . . . .

EH Huang , , , R Socher , , , CD Manning , , , 等 . . Improving word representations via global context and multiple word prototypes . . Proc 50 th Annual Meeting of Association for Computational Linguistics , , 2012 . . p.873 - - 882 . . . .

K Krishna , , , MN Murty . . Genetic K-means algorithm . . IEEE Trans Syst Man Cybern Part B , , 1999 . . 29 ( ( 3 ): ): 433 - - 439 . . DOI: 10.1109/3477.764879 http://doi.org/10.1109/3477.764879 . .

R Lebret , , , R Collobert . . Word embeddings through Hellinger PCA . . Proc 14 th Conf on European Chapter of the Association for Computational Linguistics , , 2014 . . p.482 - - 490 . . . .

R Lebret , , , R Collobert . . Rehabilitation of count-based models for word vector representations . . Int Conf on Intelligent Text Processing and Computational Linguistics , , 2015 . . p.417 - - 429 . . DOI: 10.1007/978-3-319-18111-0_31 http://doi.org/10.1007/978-3-319-18111-0_31 . .

O Levy , , , Y Goldberg . . Dependency-based word embeddings . . Proc 52 nd Annual Meeting of Association for Computational Linguistics , , 2014 . . p.302 - - 308 . . DOI: 10.3115/v1/P14-2050 http://doi.org/10.3115/v1/P14-2050 . .

Y Liu , , , ZY Liu , , , TS Chua , , , 等 . . Topical word embeddings . . Proc 29 th AAAI Conf on Artificial Intelligence , , 2015 . . p.2418 - - 2424 . . . .

MT Luong , , , R Socher , , , CD Manning . . Better word representations with recursive neural networks for morphology . . Proc 17 th Conf on Computational Natural Language Learning , , 2013 . . p.104 - - 113 . . . .

T Mikolov , , , I Sutskever , , , K Chen , , , 等 . . Distributed representations of words and phrases and their compositionality . . Int Conf on Neural Information Processing Systems , , 2013 . . p.3111 - - 3119 . . . .

A Mnih , , , GE Hinton . . A scalable hierarchical distributed language model . . Proc 21 st Int Conf on Neural Information Processing System , , 2008 . . p.1081 - - 1088 . . . .

KA Nguyen , , , SSI Walde , , , NT Vu . . Neural-based noise filtering from word embeddings . . Proc 26 th Int Conf on Computational Linguistics , , 2016 . . p.2699 - - 2707 . . . .

J Pennington , , , R Socher , , , CD Manning . . Glove: global vectors for word representation . . Proc Conf on Empirical Methods in Natural Language Processing , , 2014 . . p.1532 - - 1543 . . . .

YF Ren , , , RM Wang , , , DH Ji . . A topic-enhanced word embedding for Twitter sentiment classification . . Inform Sci , , 2016 . . 369 188 - - 198 . . DOI: 10.1016/j.ins.2016.06.040 http://doi.org/10.1016/j.ins.2016.06.040 . .

A Ritter , , , Mausam , , , O Etzioni . . A latent Dirichlet allocation method for selectional preferences . . Proc 48 th Annual Meeting of Association for Computational Linguistics , , 2010 . . p.424 - - 434 . . . .

H Rubenstein , , , JB Goodenough . . Contextual correlates of synonymy . . Commun ACM , , 1965 . . 8 ( ( 10 ): ): 627 - - 633 . . DOI: 10.1145/365628.365657 http://doi.org/10.1145/365628.365657 . .

F Tian , , , HJ Dai , , , J Bian , , , 等 . . A probabilistic model for learning multi-prototype word embeddings . . Proc 25 th Int Conf on Computational Linguistics , , 2014 . . p.151 - - 160 . . . .

PD Turney , , , P Pantel . . From frequency to meaning: vector space models of semantics . . J Artif Intell Res , , 2010 . . 37 ( ( 1 ): ): 141 - - 188 . . DOI: 10.1613/jair.2934 http://doi.org/10.1613/jair.2934 . .

P Wang , , , B Xu , , , JM Xu , , , 等 . . Semantic expansion using word embedding clustering and convolutional neural network for improving short text classification . . Neurocomputing , , 2016 . . 174 ( ( B ): ): 806 - - 814 . . DOI: 10.1016/j.neucom.2015.09.096 http://doi.org/10.1016/j.neucom.2015.09.096 . .

W Xu , , , AI Rudnicky . . Can artificial neural networks learn language models . . Proc 6 th Int Conf on Spoken Language Processing , , 2000 . . p.202 - - 205 . . . .

M Zhai , , , J Tan , , , DJ Choi . . Intrinsic and extrinsic evaluations of word embeddings . . Proc 30 th AAAI Conf on Artificial Intelligence , , 2016 . . p.4282 - - 4283 . . . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Federated unsupervised representation learning

Learning embeddings of a heterogeneous behavior network for potential behavior prediction

Related Author

Fengda ZHANG

Kun KUANG

Long CHEN

Zhaoyang YOU

Tao SHEN

Jun XIAO

Yin ZHANG

Chao WU

Related Institution

College of Computer Science and Technology, Zhejiang University

School of Public Affairs, Zhejiang University

Tongdun Technology

Institute of Basic Medicine and Cancer, Chinese Academy of Sciences

ElasticMind.AI Technology Inc

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰