Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension

Xiang-zhou HUANG; Si-liang TANG; Yin ZHANG; Bao-gang WEI

doi:10.1631/FITEE.1900571

Your Location：

Home >

Browse articles >

Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension

Regular Papers | Updated：2022-05-19

- Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension
- 观点型问题机器阅读理解中混合词向量和层叠循环神经网络联合训练方法的应用
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 9, Pages: 1346-1355(2020)
- Affiliations：
  
  College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
- Author bio：
  
  [ "Xiang-zhou HUANG, E-mail: huangxiangzhou@zju.edu.cn" ]
  [ "Si-liang TANG, E-mail: siliang@zju.edu.cn" ]
  Yin ZHANG, E-mail: yinzh@zju.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900571
  CLC： TP391.1
- Received：19 October 2019，
  
  Revised：2020-;8-10，
  
  Published：2020-09
- Accepted：
Scan QR Code
Xiang-zhou HUANG, Si-liang TANG, Yin ZHANG, et al. Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1346-1355.
DOI：

Xiang-zhou HUANG, Si-liang TANG, Yin ZHANG, et al. Hybrid embedding and joint training of stacked encoder for opinion question machine reading comprehension[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1346-1355. DOI： 10.1631/FITEE.1900571.

摘要

观点型问题机器阅读理解指计算机通过分析相应段落回答问题。相比于传统机器阅读理解任务的答案是在相关段落中的某一段文本，观点型问题因其答案可能不出现在相关段落中而需由多个句子推理得出，其对应的机器阅读理解任务更具挑战性。针对这个任务，提出一种新颖的基于神经网络的解决方案，其中使用了一种结合文本特征的混合词向量训练方法。此外，引入额外的注意力网络和输出层，产生多个辅助损失函数用于联合训练层叠循环神经网络。针对数据集样本分布不平衡的问题，引入问题和段落的不相关性实现数据增强。实验结果验证了所提方法的有效性。该方案获得了AIC2018观点型问题机器阅读理解赛道的双周赛冠军。

Abstract

Opinion question machine reading comprehension (MRC) requires a machine to answer questions by analyzing corresponding passages. Compared with traditional MRC tasks where the answer to every question is a segment of text in corresponding passages

opinion question MRC is more challenging because the answer to an opinion question may not appear in corresponding passages but needs to be deduced from multiple sentences. In this study

a novel framework based on neural networks is proposed to address such problems

in which a new hybrid embedding training method combining text features is used. Furthermore

extra attention and output layers which generate auxiliary losses are introduced to jointly train the stacked recurrent neural networks. To deal with imbalance of the dataset

irrelevancy of question and passage is used for data augmentation. Experimental results show that the proposed method achieves state-of-the-art performance. We are the biweekly champion in the opinion question MRC task in Artificial Intelligence Challenger 2018 (AIC 2018).

关键词

Keywords

references

P Bajaj , , , D Campos , , , N Craswell , , , 等 . . MS MARCO: a human generated MAchine Reading COmprehension dataset , , 2016 . . https://arxiv.org/abs/1611.09268 https://arxiv.org/abs/1611.09268 , , . .

J Devlin , , , MW Chang , , , K Lee , , , 等 . . BERT: pre-training of deep bidirectional transformers for language understanding , , 2018 . . https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805 , , . .

KM He , , , XY Zhang , , , SQ Ren , , , 等 . . Deep residual learning for image recognition . . IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.770 - - 778 . . DOI: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90 . .

W He , , , K Liu , , , J Liu , , , 等 . . DuReader: a Chinese machine reading comprehension dataset from real-world applications . . Proc Workshop on Machine Reading for Question Answering , , 2018 . . p.37 - - 46 . . DOI: 10.18653/v1/W18-2605 http://doi.org/10.18653/v1/W18-2605 . .

KM Hermann , , , T Kočiský , , , E Grefenstette , , , 等 . . Teaching machines to read and comprehend . . Proc 28 th Int Conf on Neural Information Processing Systems , , 2015 . . p.1693 - - 1701 . . . .

S Hochreiter , , , J Schmidhuber . . Long short-term memory . . Neur Comput , , 1997 . . 9 ( ( 8 ): ): 1735 - - 1780 . . DOI: 10.1162/neco.1997.9.8.1735 http://doi.org/10.1162/neco.1997.9.8.1735 . .

M Joshi , , , E Choi , , , DS Weld , , , 等 . . TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension , , 2017 . . https://arxiv.org/abs/1705.03551 https://arxiv.org/abs/1705.03551 , , . .

DP Kingma , , , J Ba . . Adam: a method for stochastic optimization , , 2014 . . https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 , , . .

JH Liu , , , W Wei , , , MS Sun , , , 等 . . A multi-answer multi-task framework for real-world machine reading comprehension . . Proc Conf on Empirical Methods in Natural Language Processing , , 2018 . . p.2109 - - 2118 . . DOI: 10.18653/v1/D18-1235 http://doi.org/10.18653/v1/D18-1235 . .

T Mikolov , , , I Sutskever , , , K Chen , , , 等 . . Distributed representations of words and phrases and their compositionality . . Proc 26 th Int Conf on Neural Information Processing Systems , , 2013a . . p.3111 - - 3119 . . . .

T Mikolov , , , K Chen , , , G Corrado , , , 等 . . Efficient estimation of word representations in vector space , , 2013b . . https://arxiv.org/abs/1301.3781 https://arxiv.org/abs/1301.3781 , , . .

YH Pan . . Heading toward artificial intelligence 2.0 . . Engineering , , 2016 . . 2 ( ( 4 ): ): 409 - - 413 . . DOI: 10.1016/J.ENG.2016.04.018 http://doi.org/10.1016/J.ENG.2016.04.018 . .

R Pascanu , , , T Mikolov , , , Y Bengio . . Understanding the exploding gradient problem , , 2012 . . https://arxiv.org/abs/1211.5063v1 https://arxiv.org/abs/1211.5063v1 , , . .

P Rajpurkar , , , J Zhang , , , K Lopyrev , , , 等 . . SQuAD: 100 000+ questions for machine comprehension of text . . Proc Conf on Empirical Methods in Natural Language Processing , , 2016 . . p.2383 - - 2392 . . DOI: 10.18653/v1/D16-1264 http://doi.org/10.18653/v1/D16-1264 . .

M Richardson , , , CJC Burges , , , E Renshaw . . MCTest: a challenge dataset for the open-domain machine comprehension of text . . Proc Conf on Empirical Methods in Natural Language Processing , , 2013 . . p.193 - - 203 . . . .

M Seo , , , A Kembhavi , , , A Farhadi , , , 等 . . Bidirectional attention flow for machine comprehension , , 2016 . . https://arxiv.org/abs/1611.01603 https://arxiv.org/abs/1611.01603 , , . .

N Srivastava , , , G Hinton , , , A Krizhevsky , , , 等 . . Dropout: a simple way to prevent neural networks from overfitting . . J Mach Learn Res , , 2014 . . 15 ( ( 1 ): ): 1929 - - 1958 . . . .

I Sutskever , , , O Vinyals , , , QV Le . . Sequence to sequence learning with neural networks . . Advances in Neural Information Processing Systems , , 2014 . . p.3104 - - 3112 . . . .

CQ Tan , , , FR Wei , , , WH Wang , , , 等 . . Multiway attention networks for modeling sentence pairs . . Proc 27 th Int Joint Conf on Artificial Intelligence , , 2018 . . p.4411 - - 4417 . . DOI: 10.24963/ijcai.2018/613 http://doi.org/10.24963/ijcai.2018/613 . .

O Vinyals , , , M Fortunato , , , N Jaitly . . Pointer networks . . Advances in Neural Information Processing Systems , , 2015 . . p.2692 - - 2700 . . . .

SH Wang , , , J Jiang . . Learning natural language inference with LSTM . . Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , , 2016 . . p.1442 - - 1451 . . DOI: 10.18653/v1/N16-1170 http://doi.org/10.18653/v1/N16-1170 . .

W Wang , , , N Yang , , , F Wei , , , 等 . . R-NET: Machine Reading Comprehension with Self-matching Networks . . Technical Report, Natural Language Computing Group, Microsoft Research Asia, Beijing, China , , 2017 . . https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf , , . .

HC Wu , , , RWP Luk , , , KF Wong , , , 等 . . Interpreting TF-IDF term weights as making relevance decisions . . ACM Trans Inform Syst , , 2008 . . 26 ( ( 3 ): ): 13 DOI: 10.1145/1361684.1361686 http://doi.org/10.1145/1361684.1361686 . .

YH Wu , , , M Schuster , , , ZF Chen , , , 等 . . Google's neural machine translation system: bridging the gap between human and machine translation , , 2016 . . https://arxiv.org/abs/1609.08144 https://arxiv.org/abs/1609.08144 , , . .

Y Yang , , , WT Yih , , , C Meek . . WikiQA: a challenge dataset for open-domain question answering . . Proc Conf on Empirical Methods in Natural Language Processing , , 2015 . . p.2013 - - 2018 . . DOI: 10.18653/v1/D15-1237 http://doi.org/10.18653/v1/D15-1237 . .

AW Yu , , , D Dohan , , , MT Luong , , , 等 . . QANet: combining local convolution with global self-attention for reading comprehension , , 2018 . . https://arxiv.org/abs/1804.09541 https://arxiv.org/abs/1804.09541 , , . .

YT Zhuang , , , F Wu , , , C Chen , , , 等 . . Challenges and opportunities: from big data to knowledge in AI 2.0 . . Front Inform Technol Electron Eng , , 2017 . . 18 ( ( 1 ): ): 3 - - 14 . . DOI: 10.1631/FITEE.1601883 http://doi.org/10.1631/FITEE.1601883 . .

Views

101

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Significance extraction based on data augmentation for reinforcement learning

Detecting compromised accounts caused by phone number recycling on e-commerce platforms: taking Meituan as an example

A graph-based two-stage classification network for mobile screen defect inspection

Visual recognition of cardiac pathology based on 3D parametric model reconstruction

A novel color image encryption algorithm based on a fractional-order discrete chaotic neural network and DNA sequence operations

Related Author

Yang YANG

Dequan LI

Yuxi HAN

Zhenhua ZHANG

Min GAO

Shutong CHEN

Yangbo GAO

Yu CHEN

Related Institution

Faculty of Artificial Intelligence, Anhui University of Science and Technology

Department of Interactive Media, Hong Kong Baptist University

Meituan

Shanghai Key Lab of Intelligent Information Processing, Fudan University

School of Computer Science, Fudan University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰