Tandem hidden Markov models using deep belief networks for offline handwriting recognition

Partha Pratim ROY; Guoqiang ZHONG; Mohamed CHERIET

doi:10.1631/FITEE.1600996

Your Location：

Home >

Browse articles >

Tandem hidden Markov models using deep belief networks for offline handwriting recognition

Regular Papers | Updated：2022-05-19

- Tandem hidden Markov models using deep belief networks for offline handwriting recognition
- 融合深度置信网络的串联隐马尔科夫模型及其在脱机手写识别中的应用
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 7, Pages: 978-988(2017)
- Affiliations：
  
  Department of Computer Science & Engineering, Indian Institute of Technology Roorkee, Roorkee 247667, India
  Department of Computer Science and Technology, Ocean University of China, Qingdao 266100, China
  Synchromedia Laboratory, cole de Technologie Suprieure, Montreal H3C 1K3, Canada
- Author bio：
  
  Partha Pratim ROY, E-mail:proy.fcs@iitr.ac.in
  [ "Guoqiang ZHONG, E-mail: gqzhong@ouc.edu.cn" ]
  [ "Mohamed CHERIET, E-mail:mohamed.cheriet@etsmtl.ca" ]
- Funds：
- DOI：10.1631/FITEE.1600996
  CLC： TP391
- Received：15 February 2016，
  
  Revised：2017-;6-16，
  
  Published：2017-07
- Accepted：
Scan QR Code
Partha Pratim ROY, Guoqiang ZHONG, Mohamed CHERIET. Tandem hidden Markov models using deep belief networks for offline handwriting recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(7): 978-988.
DOI：

Partha Pratim ROY, Guoqiang ZHONG, Mohamed CHERIET. Tandem hidden Markov models using deep belief networks for offline handwriting recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(7): 978-988. DOI： 10.1631/FITEE.1600996.

摘要

在文档分析和模式识别领域，自由书写的脱机手写识别是一个非常具有挑战性的研究课题。近年来，为了充分探索隐藏在文档图像中的监督信息，许多研究工作试图将多层感知机以一种混合或串联的形式嵌入隐马尔科夫模型当中。然而，因为多层感知机学习能力的不足，学习到的特征对于后续的识别任务不一定是最优的。在本文中，我们针对自由书写的脱机手写识别提出一种基于深度结构的串联方法。在提出的模型中，深度置信网络被用于学习序列数据的紧致表示，隐马尔科夫模型被用于（子-）词的识别。我们在两个公开的数据集上验证了所提出的模型，这两个数据集是分别基于拉丁和阿拉伯语的RIMES和IFN/ENIT；我们还在Devanagari数据集上验证了所提出的模型，这个数据集是基于印度语的。大量的实验展示了所提出模型的优势，特别是相对于多层感知机-隐马尔科夫模型的串联方法。

Abstract

Unconstrained offline handwriting recognition is a challenging task in the areas of document analysis and pattern recognition. In recent years

to sufficiently exploit the supervisory information hidden in document images

much effort has been made to integrate multi-layer perceptrons (MLPs) in either a hybrid or a tandem fashion into hidden Markov models (HMMs). However

due to the weak learnability of MLPs

the learnt features are not necessarily optimal for subsequent recognition tasks. In this paper

we propose a deep architecture-based tandem approach for unconstrained offline handwriting recognition. In the proposed model

deep belief networks are adopted to learn the compact representations of sequential data

while HMMs are applied for (sub-)word recognition. We evaluate the proposed model on two publicly available datasets

i.e.

RIMES and IFN/ENIT

which are based on Latin and Arabic languages respectively

and one dataset collected by ourselves called Devanagari (an Indian script). Extensive experiments show the advantage of the proposed model

especially over the MLP-HMMs tandem approaches.

关键词

Keywords

references

E. Augustin , , , M. Carré , , , E. Grosicki , , , 等 . . RIMES evaluation campaign for handwritten mail processing . . Proc. Int. Workshop on Frontiers in Handwriting Recognition , , 2006 . . p.231 - - 235 . . . .

L.E. Baum , , , T. Petrie , , , G. Soules , , , 等 . . A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains . . Ann. Math. Statist. , , 1970 . . 41 ( ( 1 ): ): 164 - - 171 . . DOI: 10.1214/aoms/1177697196 http://doi.org/10.1214/aoms/1177697196 . .

R. Bertolami , , , H. Bunke . . Hidden Markov modelbased ensemble methods for offline handwritten text line recognition . . Patt. Recog. , , 2008 . . 41 ( ( 11 ): ): 3452 - - 3460 . . DOI: 10.1016/j.patcog.2008.04.003 http://doi.org/10.1016/j.patcog.2008.04.003 . .

A.L. Bianne-Bernard , , , F. Menasri , , , R.A.H. Mohamad , , , 等 . . Dynamic and contextual information in HMM modeling for handwritten word recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2011 . . 33 ( ( 10 ): ): 2066 - - 2080 . . DOI: 10.1109/TPAMI.2011.22 http://doi.org/10.1109/TPAMI.2011.22 . .

H.A. Bourlard , , , N. Morgan . . Connectionist Speech Recognition: a Hybrid Approach , , : : USA Springer US , , 1994 . . .

H. Bunke . . Recognition of cursive Roman handwriting: past, present and future . . Proc. 7th Int. Conf. on Document Analysis and Recognition , , 2003 . . p.448 - - 459 . . DOI: 10.1109/ICDAR.2003.1227707 http://doi.org/10.1109/ICDAR.2003.1227707 . .

G. Dahl , , , D. Yu , , , L. Deng , , , 等 . . Large vocabulary continuous speech recognition with context-dependent DBN-HMMs . . Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing , , 2011 . . p.4688 - - 4691 . . . .

T. Deselaers , , , S. Hasan , , , O. Bender , , , 等 . . A deep learning approach to machine transliteration . . Proc. 4th Workshop on Statistical Machine Translation , , 2009 . . p.233 - - 241 . . . .

P. Dreuw , , , G. Heigold , , , H. Ney . . Confidence-based discriminative training for model adaptation in offline Arabic handwriting recognition . . Proc. 10th Int. Conf. on Document Analysis and Recognition , , 2009 . . p.596 - - 600 . . DOI: 10.1109/ICDAR.2009.116 http://doi.org/10.1109/ICDAR.2009.116 . .

P. Dreuw , , , P. Doetsch , , , C. Plahl , , , 等 . . Hierarchical hybrid MLP/HMM or rather MLP features for a discriminatively trained Gaussian HMM: a comparison for offline handwriting recognition . . Proc. 18th Int. Conf. on Image Processing , , 2011a . . p.3541 - - 3544 . . DOI: 10.1109/ICIP.2011.6116480 http://doi.org/10.1109/ICIP.2011.6116480 . .

P. Dreuw , , , G. Heigold , , , H. Ney . . Confidence-and margin-based MMI/MPE discriminative training for offline handwriting recognition . . Int. J. Doc. Anal. Recog. , , 2011b . . 14 273 - - 288 . . DOI: 10.1007/s10032-011-0160-x http://doi.org/10.1007/s10032-011-0160-x . .

A. El-Yacoubi , , , M. Gilloux , , , R. Sabourin , , , 等 . . An HMM-based approach for off-line unconstrained handwritten word modeling and recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 1999 . . 21 ( ( 8 ): ): 752 - - 760 . . DOI: 10.1109/34.784288 http://doi.org/10.1109/34.784288 . .

S. Espana-Boquera , , , M.J. Castro-Bleda , , , J. Gorbe-Moya , , , 等 . . Improving offline handwritten text recognition with hybrid HMM/ANN models . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2011 . . 33 ( ( 4 ): ): 767 - - 779 . . DOI: 10.1109/TPAMI.2010.141 http://doi.org/10.1109/TPAMI.2010.141 . .

H. Fujisawa . . Forty years of research in character and document recognition-an industrial perspective . . Patt. Recog. , , 2008 . . 41 2435 - - 2446 . . DOI: 10.1016/j.patcog.2008.03.015 http://doi.org/10.1016/j.patcog.2008.03.015 . .

A. Graves , , , J. Schmidhuber . . Offline handwriting recognition with multidimensional recurrent neural networks . . Proc. 21st Int. Conf. on Neural Information Processing Systems , , 2008 . . p.545 - - 552 . . . .

A. Graves , , , M. Liwicki , , , S. Fernández , , , 等 . . A novel connectionist system for unconstrained handwriting recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2009 . . 31 ( ( 5 ): ): 855 - - 868 . . DOI: 10.1109/TPAMI.2008.137 http://doi.org/10.1109/TPAMI.2008.137 . .

E. Grosicki , , , H. El Abed . . ICDAR 2009 handwriting recognition competition . . Proc. 10th Int. Conf. on Document Analysis and Recognition , , 2009 . . p.1398 - - 1402 . . DOI: 10.1109/ICDAR.2009.184 http://doi.org/10.1109/ICDAR.2009.184 . .

S. Haykin . . Neural Networks: a Comprehensive Foundation , , : : USA Prentice Hall , , 1998 . . .

H. Hermansky , , , D.P.W. Ellis , , , S. Sharma . . Tandem connectionist feature extraction for conventional HMM systems . . Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing , , 2000 . . p.1 - - 4 . . DOI: 10.1109/ICASSP.2000.862024 http://doi.org/10.1109/ICASSP.2000.862024 . .

G.E. Hinton . . Training products of experts by minimizing contrastive divergence . . Neur. Comput. , , 2002 . . 14 ( ( 8 ): ): 1771 - - 1800 . . DOI: 10.1162/089976602760128018 http://doi.org/10.1162/089976602760128018 . .

G.E. Hinton , , , S. Osindero , , , Y.W. Teh . . A fast learning algorithm for deep belief nets . . Neur. Comput. , , 2006 . . 18 ( ( 7 ): ): 1527 - - 1554 . . DOI: 10.1162/neco.2006.18.7.1527 http://doi.org/10.1162/neco.2006.18.7.1527 . .

Y. Kessentini , , , T. Paquet , , , A. Benhamadou . . A multistream HMM-based approach for off-line multi-script handwritten word recognition . . Proc. Int. Conf. on Frontiers in Handwriting Recognition , , 2008 . . p.1 - - 6 . . . .

J. Kittler , , , P.C. Young . . A new approach to feature selection based on the Karhunen-Loeve expansion . . Patt. Recog. , , 1973 . . 5 ( ( 4 ): ): 335 - - 352 . . DOI: 10.1016/0031-3203(73)90025-3 http://doi.org/10.1016/0031-3203(73)90025-3 . .

M. Kozielski , , , P. Doetsch , , , H. Ney . . Improvements in RWTH's system for off-line handwriting recognition . . Proc. 12th Int. Conf. on Document Analysis and Recognition , , 2013 . . p.935 - - 939 . . DOI: 10.1109/ICDAR.2013.190 http://doi.org/10.1109/ICDAR.2013.190 . .

V. Margner , , , H. El Abed . . ICFHR 2010-Arabic handwriting recognition competition . . Proc. Int. Conf. on Frontiers in Handwriting Recognition , , 2010 . . p.709 - - 714 . . DOI: 10.1109/ICFHR.2010.115 http://doi.org/10.1109/ICFHR.2010.115 . .

S. Marinai , , , M. Gori , , , G. Soda . . Artificial neural networks for document analysis and recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2005 . . 27 ( ( 1 ): ): 23 - - 35 . . DOI: 10.1109/TPAMI.2005.4 http://doi.org/10.1109/TPAMI.2005.4 . .

U.V. Marti , , , H. Bunke . . Using a statistical language model to improve the performance of an HMM-based cursive handwriting recognition system . . Int. J. Patt. Recog. Artif. Intell. , , 2001 . . 15 ( ( 1 ): ): 65 - - 90 . . DOI: 10.1142/S0218001401000848 http://doi.org/10.1142/S0218001401000848 . .

R.A.H. Mohamad , , , L. Likforman-Sulem , , , C. Mokbel . . Combining slanted-frame classifiers for improved HMMbased Arabic handwriting recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2009 . . 31 ( ( 7 ): ): 1165 - - 1177 . . DOI: 10.1109/TPAMI.2008.136 http://doi.org/10.1109/TPAMI.2008.136 . .

A.R. Mohamed , , , G. Dahl , , , G. Hinton . . Deep belief networks for phone recognition . . Proc. NIPS Workshop on Deep Learning for Speech Recognition and Related Applications , , 2009 . . p.1 - - 9 . . . .

A.R. Mohamed , , , G. Dahl , , , G. Hinton . . Acoustic modeling using deep belief networks . . IEEE Trans. Audio Speech Lang. Process. , , 2012 . . 20 ( ( 1 ): ): 14 - - 22 . . DOI: 10.1109/TASL.2011.2109382 http://doi.org/10.1109/TASL.2011.2109382 . .

N. Otsu . . A threshold selection method from gray-level histograms . . IEEE Trans. Syst. Man Cybern. , , 1979 . . 9 ( ( 1 ): ): 62 - - 66 . . DOI: 10.1109/TSMC.1979.4310076 http://doi.org/10.1109/TSMC.1979.4310076 . .

U. Pal , , , B.B. Chaudhuri . . Indian script character recognition: a survey . . Patt. Recog. , , 2004 . . 37 ( ( 9 ): ): 1887 - - 1899 . . DOI: 10.1016/j.patcog.2004.02.003 http://doi.org/10.1016/j.patcog.2004.02.003 . .

L.R. Rabiner . . A tutorial on hidden Markov models and selected applications in speech recognition . . Proc. IEEE , , 1989 . . 77 ( ( 2 ): ): 257 - - 286 . . DOI: 10.1109/5.18626 http://doi.org/10.1109/5.18626 . .

S. Renals , , , N. Morgan , , , H. Bourlard , , , 等 . . Connectionist probability estimators in HMM speech recognition . . IEEE Trans. Speech Audio Process. , , 1994 . . 2 ( ( 1 ): ): 161 - - 174 . . DOI: 10.1109/89.260359 http://doi.org/10.1109/89.260359 . .

J.A. Rodríguez , , , F. Perronnin . . Local gradient histogram features for word spotting in unconstrained handwritten documents . . Proc. Int. Conf. on Frontiers in Handwriting Recognition , , 2008 . . p.7 - - 12 . . . .

J. Schenk , , , G. Rigoll . . Novel hybrid NN/HMM modelling techniques for on-line handwriting recognition . . Proc. 10th Int. Workshop on Frontiers in Handwriting Recognition , , 2006 . . p.1 - - 5 . . . .

A. Senior , , , A.J. Robinson . . An off-line cursive handwriting recognition system . . IEEE Trans. Patt. Anal. Mach. Intell. , , 1998 . . 20 ( ( 3 ): ): 309 - - 321 . . DOI: 10.1109/34.667887 http://doi.org/10.1109/34.667887 . .

A. Senior , , , G. Heigold , , , M. Bacchiani , , , 等 . . GMMfree DNN training . . Proc. Int. Conf. on Acoustics, Speech, and Signal Processing , , 2014 . . p.1 - - 5 . . . .

S. Sharma , , , D. Ellis , , , S. Kajarekar , , , 等 . . Feature extraction using non-linear transformation for robust speech recognition on the Aurora database . . Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing , , 2000 . . p.1117 - - 1120 . . DOI: 10.1109/ICASSP.2000.859160 http://doi.org/10.1109/ICASSP.2000.859160 . .

B. Shaw , , , U. Bhattacharya , , , S.K. Parui . . Combination of features for efficient recognition of offline handwritten Devanagari words . . Proc. 14th Int. Conf. on Frontiers in Handwriting Recognition , , 2014 . . p.240 - - 245 . . DOI: 10.1109/ICFHR.2014.48 http://doi.org/10.1109/ICFHR.2014.48 . .

S. Thomas , , , C. Chatelain , , , L. Heutte , , , 等 . . A deep HMM model for multiple keywords spotting in handwritten documents . . Patt. Anal. Appl. , , 2015 . . 18 ( ( 4 ): ): 1003 - - 1015 . . DOI: 10.1007/s10044-014-0433-3 http://doi.org/10.1007/s10044-014-0433-3 . .

A. Vinciarelli . . A survey on off-line cursive word recognition . . Patt. Recog. , , 2002 . . 35 ( ( 7 ): ): 1433 - - 1446 . . DOI: 10.1016/S0031-3203(01)00129-7 http://doi.org/10.1016/S0031-3203(01)00129-7 . .

A. Vinciarelli , , , S. Bengio , , , H. Bunke . . Offline recognition of unconstrained handwritten texts using HMMs and statistical language models . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2004 . . 26 ( ( 6 ): ): 709 - - 720 . . DOI: 10.1109/TPAMI.2004.14 http://doi.org/10.1109/TPAMI.2004.14 . .

S. Young , , , G. Evermann , , , M.J.F. Gales . . The HTK Book (Version 34) . . Engineering Department, Cambridge University, UK. , , 2006 . . .

M. Zimmermann , , , J.C. Chappelier , , , H. Bunke . . Offline grammar-based recognition of handwritten sentences . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2006 . . 28 ( ( 5 ): ): 818 - - 821 . . DOI: 10.1109/TPAMI.2006.103 http://doi.org/10.1109/TPAMI.2006.103 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems

Related Author

Yusong ZHOU

Xiaoyu JIANG

Shu SUN

Xinmin ZHANG

Yuanqiu MO

Zhihuan SONG

Maokun ZHENG

Zhi LI

Related Institution

School of Mathematics, Southeast University

State Key Laboratory of Industrial Control Technology, Zhejiang University

College of Control Science and Engineering, Zhejiang University

State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰