Learning to select pseudo labels: a semi-supervised method for named entity recognition

LI Zhen-zhen; FENG Da-wei; LI Dong-sheng; LU Xi-cheng

doi:10.1631/FITEE.1800743

Your Location：

Home >

Browse articles >

Learning to select pseudo labels: a semi-supervised method for named entity recognition

Regular Papers | Updated：2022-05-19

- Learning to select pseudo labels: a semi-supervised method for named entity recognition
- 学习挑选伪标签：一种用于命名实体识别的半监督学习方法
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 6, Pages: 903-916(2020)
- Affiliations：
  
  College of Computer, National University of Defense Technology, Changsha 410073, China
- Author bio：
  
  [ "Zhen-zhen LI, E-mail: lizhenzhen14@nudt.edu.cn" ]
  [ "Da-wei FENG, E-mail: davyfeng.c@gmail.com" ]
  [ "Dong-sheng LI, E-mail: dsli@nudt.edu.cn" ]
  [ "Xi-cheng LU, E-mail: xclu@nudt.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1800743
  CLC： TP391.1
- Received：22 November 2018，
  
  Revised：2019-;8-19，
  
  Published Online：27 December 2019，
  
  Published：2020-06
- Accepted：
Scan QR Code
LI Zhen-zhen, FENG Da-wei, LI Dong-sheng, et al. Learning to select pseudo labels: a semi-supervised method for named entity recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(6): 903-916.
DOI：

LI Zhen-zhen, FENG Da-wei, LI Dong-sheng, et al. Learning to select pseudo labels: a semi-supervised method for named entity recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(6): 903-916. DOI： 10.1631/FITEE.1800743.

摘要

深度学习模型在命名实体识别（NER）中实现了最先进的性能；然而，其良好性能很大程度上依赖于大量标记数据。在某些特定领域，例如医学、金融和军事领域，标记数据非常稀缺，而未标记数据则很容易获得。过往研究使用未标记数据丰富词的表示，却忽略了未标记数据中对NER任务很可能有帮助的大量实体信息。本文提出一种用于NER任务的半监督方法，其通过学习一个判别模块筛除错误伪标签，以创建高质量标注数据。伪标签是为未标记数据自动生成的标签，并被当作真实标签用来训练模型。该半监督框架包括3个步骤：为特定NER任务构建最佳单神经网络模型，学习一个评价伪标签的模块，以及迭代创建新的标记数据和改进NER模型。两个英语NER任务和一个中文医疗命名实体识别任务的实验结果表明，该方法进一步提高了最佳单神经模型的性能。当仅使用预训练的静态词嵌入且不依赖任何外部知识时，该方法可获得与CoNLL-2003和OntoNotes 5.0英语NER任务上最先进模型相当的性能。

Abstract

Deep learning models have achieved state-of-the-art performance in named entity recognition (NER); the good performance

however

relies heavily on substantial amounts of labeled data. In some specific areas such as medical

financial

and military domains

labeled data is very scarce

while unlabeled data is readily available. Previous studies have used unlabeled data to enrich word representations

but a large amount of entity information in unlabeled data is neglected

which may be beneficial to the NER task. In this study

we propose a semi-supervised method for NER tasks

which learns to create high-quality labeled data by applying a pre-trained module to filter out erroneous pseudo labels. Pseudo labels are automatically generated for unlabeled data and used as if they were true labels. Our semi-supervised framework includes three steps: constructing an optimal single neural model for a specific NER task

learning a module that evaluates pseudo labels

and creating new labeled data and improving the NER model iteratively. Experimental results on two English NER tasks and one Chinese clinical NER task demonstrate that our method further improves the performance of the best single neural model. Even when we use only pre-trained static word embeddings and do not rely on any external knowledge

our method achieves comparable performance to those state-of-the-art models on the CoNLL-2003 and OntoNotes 5.0 English NER tasks.

关键词

Keywords

references

A Akbik , , , D Blythe , , , R Vollgraf . . Contextual string embeddings for sequence labeling . . Proc 27 th Int Conf on Computational Linguistics , , 2018 . . p.1638 - - 1649 . . . .

CC Chang , , , CJ Lin . . LIBSVM-a library for support vector machines . . ACM Trans Intell Syst Technol , , 2011 . . 2 Article 27 DOI: 10.1145/1961189.1961199 http://doi.org/10.1145/1961189.1961199 . .

NV Chawla , , , KW Bowyer , , , LO Hall , , , 等 . . SMOTE: synthetic minority over-sampling technique . . J Artif Intell Res , , 2002 . . 16 321 - - 357 . . DOI: 10.1613/jair.953 http://doi.org/10.1613/jair.953 . .

JPC Chiu , , , E Nichols . . Named entity recognition with bidirectional LSTM-CNNs . . Trans Assoc Comput Ling , , 2016 . . 4 357 - - 370 . . DOI: 10.1162/tacl_a_00104 http://doi.org/10.1162/tacl_a_00104 . .

R Collobert , , , J Weston , , , L Bottou , , , 等 . . Natural language processing (almost) from scratch . . J Mach Learn Res , , 2011 . . 12 2493 - - 2537 . . . .

C Cortes , , , V Vapnik . . Support-vector networks . . Mach Learn , , 1995 . . 20 ( ( 3 ): ): 273 - - 297 . . DOI: 10.1007/BF00994018 http://doi.org/10.1007/BF00994018 . .

J Devlin , , , MW Chang , , , K Lee , , , 等 . . BERT: pre-training of deep bidirectional transformers for language understanding . . 2018 . . https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805 , , . .

A Ghaddar , , , P Langlais . . Robust lexical features for improved neural network named-entity recognition . . Proc 27 th Int Conf on Computational Linguistics , , 2018 . . p.1896 - - 1907 . . . .

Y Grandvalet , , , Y Bengio . . Entropy regularization , , : : Cambridge, Mass In: Chapelle O, Sch lkopf B, Zien A (Eds.), Semi-supervised Learning. MIT Press , , 2006 . . p.151 - - 168 . . DOI: 10.7551/mitpress/9780262033589.001.0001 http://doi.org/10.7551/mitpress/9780262033589.001.0001 . .

S Hochreiter , , , J Schmidhuber . . Long short-term memory . . Neur Comput , , 1997 . . 9 ( ( 8 ): ): 1735 - - 1780 . . DOI: 10.1162/neco.1997.9.8.1735 http://doi.org/10.1162/neco.1997.9.8.1735 . .

J Hu , , , X Shi , , , Z Liu , , , 等 . . HITSZ_CNER: a hybrid system for entity recognition from Chinese clinical text . . China Conf on Knowledge Graph and Semantic Computing , , 2017 . . p.1 - - 6 . . . .

Z Huang , , , W Xu , , , K Yu . . Bidirectional LSTM-CRF models for sequence tagging . . 2015 . . https://arxiv.org/abs/1508.01991 https://arxiv.org/abs/1508.01991 , , . .

AN Jagannatha , , , H Yu . . Structured prediction models for RNN based sequence labeling in clinical text . . Proc Conf on Empirical Methods in Natural Language Processing , , 2016 . . p.856 DOI: 10.18653/v1/D16-1082 http://doi.org/10.18653/v1/D16-1082 . .

JD Lafferty , , , A McCallum , , , FCN Pereira . . Conditional random fields: probabilistic models for segmenting and labeling sequence data . . Proc 18 th Int Conf on Machine Learning , , 2001 . . p.282 - - 289 . . . .

G Lample , , , M Ballesteros , , , S Subramanian , , , 等 . . Neural architectures for named entity recognition . . North American Chapter of the Association for Computational Linguistics , , 2016 . . p.260 - - 270 . . DOI: 10.18653/v1/N16-1030 http://doi.org/10.18653/v1/N16-1030 . .

DH Lee . . Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks . . Work Shop on Challenges in Representation Learning , , 2013 . . p.1 - - 6 . . . .

PH Li , , , RP Dong , , , YS Wang , , , 等 . . Leveraging linguistic structures for named entity recognition with bidirectional recursive neural networks . . Proc Conf on Empirical Methods in Natural Language Processing , , 2017 . . p.2664 - - 2669 . . DOI: 10.18653/v1/D17-1282 http://doi.org/10.18653/v1/D17-1282 . .

WH Liao , , , S Veeramachaneni . . A simple semi-supervised algorithm for named entity recognition . . Proc NAACL HLT Workshop on Semi-supervised Learning for Natural Language Processing , , 2009 . . p.58 - - 65 . . . .

XZ Ma , , , E Hovy . . End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF . . Proc 54 th Annual Meeting of the Association for Computational Linguistics , , 2016 . . p.1064 - - 1074 . . DOI: 10.13140/RG.2.1.2182.5685 http://doi.org/10.13140/RG.2.1.2182.5685 . .

G Mesnil , , , X He , , , L Deng , , , 等 . . Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding . . Interspeech , , 2013 . . p.1 - - 5 . . . .

T Mikolov , , , I Sutskever , , , K Chen , , , 等 . . Distributed representations of words and phrases and their compositionality . . Proc 26 th Int Conf on Neural Information Processing Systems , , 2013 . . p.3111 - - 3119 . . . .

J Pennington , , , R Socher , , , CD Manning . . Glove: global vectors for word representation . . Proc Empirical Methods in Natural Language Processing , , 2014 . . p.1532 - - 1543 . . . .

ME Peters , , , W Ammar , , , C Bhagavatula , , , 等 . . Semi-supervised sequence tagging with bidirectional language models . . Proc 55 th Annual Meeting of the Association for Computational Linguistics , , 2017 . . p.1756 - - 1765 . . DOI: 10.18653/v1/P17-1161 http://doi.org/10.18653/v1/P17-1161 . .

ME Peters , , , M Neumann , , , M Iyyer , , , 等 . . Deep contextualized word representations . . 2018 . . https://arxiv.org/abs/1802.05365 https://arxiv.org/abs/1802.05365 , , . .

S Pradhan , , , A Moschitti , , , N Xue , , , 等 . . Towards robust linguistic analysis using ontonotes . . Proc 7 th Conf on Computational Natural Language Learning , , 2013 . . p.143 - - 152 . . . .

YJ Qi , , , R Collobert , , , P Kuksa , , , 等 . . Combining labeled and unlabeled data with word-class distribution learning . . Proc 18 th ACM Conf on Information and Knowledge Management , , 2009 . . p.1737 - - 1740 . . DOI: 10.1145/1645953.1646218 http://doi.org/10.1145/1645953.1646218 . .

M Rei . . Semi-supervised multitask learning for sequence labeling . . 55 th Annual Meeting of the Association for Computational Linguistics , , 2017 . . p.2121 - - 2130 . . DOI: 10.18653/v1/P17-1194 http://doi.org/10.18653/v1/P17-1194 . .

J Schmidhuber . . Deep learning in neural networks: an overview . . Neur Netw , , 2015 . . 61 85 - - 117 . . DOI: 10.1016/j.neunet.2014.09.003 http://doi.org/10.1016/j.neunet.2014.09.003 . .

YY Shen , , , H Yun , , , ZC Lipton , , , 等 . . Deep active learning for named entity recognition . . 2017 . . https://arxiv.org/abs/1707.05928 https://arxiv.org/abs/1707.05928 , , . .

E Strubell , , , P Verga , , , D Belanger , , , 等 . . Fast and accurate entity recognition with iterated dilated mboxconvolutions . . Proc Conf on Empirical Methods in Natural Language Processing , , 2017 . . p.2670 - - 2680 . . . .

YQ Sun , , , L Li , , , ZW Xie , , , 等 . . Co-training an improved recurrent neural network with probability statistic models for named entity recognition . . Int Conf on Database Systems for Advanced Applications , , 2017 . . p.545 - - 555 . . DOI: 10.1007/978-3-319-55699-4_33 http://doi.org/10.1007/978-3-319-55699-4_33 . .

EF Tjong Kim Sang , , , F de Meulder . . Introduction to the CoNLL-2003 shared task: language-independent named entity recognition . . Proc 7 th Conf on Natural Language Learning at HLT-NAACL , , 2003 . . p.142 - - 147 . . DOI: 10.3115/1119176.1119195 http://doi.org/10.3115/1119176.1119195 . .

H Wu , , , S Prasad . . Semi-supervised deep learning using pseudo labels for hyperspectral image classification . . IEEE Trans Image Process , , 2018 . . 27 ( ( 3 ): ): 1259 - - 1270 . . DOI: 10.1109/TIP.2017.2772836 http://doi.org/10.1109/TIP.2017.2772836 DOI: 10.1109/TIP.2017.2772836 http://doi.org/10.1109/TIP.2017.2772836 . .

Y Xia , , , Q Wang . . Clinical named entity recognition: ECUST in the CCKS-2017 shared task 2 . . CEUR Workshop Proc , , 2017 . . p.43 - - 48 . . . .

Y Xiao , , , Z Wang . . Clinical Named Entity Recognition Evaluation Tasks at CCKS 2017 . . 2017 . . .

J Yang , , , Y Zhang . . NCRF++: an open-source neural sequence labeling toolkit . . Proc 56 th Annual Meeting of the Association for Computational Linguistics , , 2018 . . p.74 - - 79 . . . .

F Zhai , , , S Potdar , , , B Xiang , , , 等 . . Neural models for sequence chunking . . Proc 31 st AAAI Conf on Artificial Intelligence , , 2017 . . p.3365 - - 3371 . . . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems

Related Author

Yusong ZHOU

Xiaoyu JIANG

Shu SUN

Xinmin ZHANG

Yuanqiu MO

Zhihuan SONG

Maokun ZHENG

Zhi LI

Related Institution

School of Mathematics, Southeast University

State Key Laboratory of Industrial Control Technology, Zhejiang University

College of Control Science and Engineering, Zhejiang University

State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰