Binary neural networks for speech recognition

Yan-min QIAN; Xu XIANG

doi:10.1631/FITEE.1800469

Your Location：

Home >

Browse articles >

Binary neural networks for speech recognition

Regular Papers | Updated：2022-05-19

- Binary neural networks for speech recognition
- 用于语音识别的二值神经网络
- Frontiers of Information Technology & Electronic Engineering Vol. 20, Issue 5, Pages: 701-715(2019)
- Affiliations：
  
  Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
  SpeechLab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
- Author bio：
  
  Yan-min QIAN, E-mail: yanminqian@sjtu.edu.cn
  [ "Xu XIANG, E-mail: chinoiserie@sjtu.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1800469
  CLC： TP391.4
- Received：06 August 2018，
  
  Revised：2019-;5-;3，
  
  Published：2019-05
- Accepted：
Scan QR Code
Yan-min QIAN, Xu XIANG. Binary neural networks for speech recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(5): 701-715.
DOI：

Yan-min QIAN, Xu XIANG. Binary neural networks for speech recognition[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(5): 701-715. DOI： 10.1631/FITEE.1800469.

摘要

近年来，在语音识别的声学建模中，深度神经网络(DNNs)明显优于高斯混合模型。然而，推断阶段巨大的计算量使其难以部署在低功耗的嵌入式模型上。为此，稀疏性和低精度定点量化技术被广泛使用。为降低推理阶段计算量，本文开发了用于语音识别的二进制神经网络，并实现了高速的二值矩阵乘法。在中央处理器(CPU)和图形处理单元(GPU)上，二值矩阵乘法的运行速度是浮点矩阵乘法的5–7倍。针对大规模连续语音识别的声学建模，提出多种二值神经网络及相关模型优化算法。为提高二值模型的精度，探索了从浮点模型到二值模型的知识蒸馏技术。在标准的Switchboard语音识别任务上，该二值神经网络模型比浮点神经网络模型速度提高3–4倍。借助知识蒸馏技术，二值深度神经网络或卷积神经网络相对其浮点神经网络的词错误率增加可以保持在15%以内。若只二值化卷积神经网络的卷积层，词错误率增加几乎可忽略。

Abstract

Recently

deep neural networks (DNNs) significantly outperform Gaussian mixture models in acoustic modeling for speech recognition. However

the substantial increase in computational load during the inference stage makes deep models difficult to directly deploy on low-power embedded devices. To alleviate this issue

structure sparseness and low precision fixed-point quantization have been applied widely. In this work

binary neural networks for speech recognition are developed to reduce the computational cost during the inference stage. A fast implementation of binary matrix multiplication is introduced. On modern central processing unit (CPU) and graphics processing unit (GPU) architectures

a 5--7 times speedup compared with full precision floating-point matrix multiplication can be achieved in real applications. Several kinds of binary neural networks and related model optimization algorithms are developed for large vocabulary continuous speech recognition acoustic modeling. In addition

to improve the accuracy of binary models

knowledge distillation from the normal full precision floating-point model to the compressed binary model is explored. Experiments on the standard Switchboard speech recognition task show that the proposed binary neural networks can deliver 3--4 times speedup over the normal full precision deep models. With the knowledge distillation from the normal floating-point models

the binary DNNs or binary convolutional neural networks (CNNs) can restrict the word error rate (WER) degradation to within 15.0%

compared to the normal full precision floating-point DNNs or CNNs

respectively. Particularly for the binary CNN with binarization only on the convolutional layers

the WER degradation is very small and is almost negligible with the proposed approach.

关键词

Keywords

references

Y Bengio , , , N Lonard , , , A Courville . . Estimating orpropagating gradients through stochastic neurons forconditional computation , , 2013 . . https://doi.org/abs/1308.3432 https://doi.org/abs/1308.3432 , , . .

MX Bi , , , YM Qian , , , K Yu . . Very deep convolutionalneural networks for LVCSR . . 16 th Annual Conf of Int Speech Communication Association , , 2015 . . p.3259 - - 3263 . . . .

ZH Chen , , , YM Zhuang , , , YM Qian , , , 等 . . Phonesynchronous speech recognition with CTC lattices . . IEEE/ACM Trans Audio Speech Lang Process , , 2017 . . 25 ( ( 1 ): ): 90 - - 101 . . DOI: 10.1109/TASLP.2016.2625459 http://doi.org/10.1109/TASLP.2016.2625459 . .

ZH Chen , , , J Luitjens , , , HN Xu , , , 等 . . A GPU-based WFST decoder with exact lattice generation , , 2018a . . https://doi.org/abs/1804.03243 https://doi.org/abs/1804.03243 , , . .

ZH Chen , , , Q Liu , , , H Li , , , 等 . . On modular trainingof neural acoustics-to-word model for LVCSR . . IEEE Int Conf on Acoustics, Speech, and Signal Processing , , 2018b . . p.4754 - - 4758 . . DOI: 10.1109/ICASSP.2018.8461361 http://doi.org/10.1109/ICASSP.2018.8461361 . .

ZH Chen , , , J Droppo , , , JY Li , , , 等 . . Progressivejoint modeling in unsupervised single-channel overlapped speech recognition . . IEEE/ACM Trans Audio Speech Lang Process , , 2018c . . 26 ( ( 1 ): ): 184 - - 196 . . DOI: 10.1109/TASLP.2017.2765834 http://doi.org/10.1109/TASLP.2017.2765834 . .

R Collobert , , , K Kavukcuoglu , , , C Farabet . . Torch7:a Matlab-like environment for machine learning . . BigLearn NIPS Workshop , , 2011 . . .

M Courbariaux , , , I Hubara , , , D Soudry , , , 等 . . Binarizedneural networks: training deep neural networks withweights and activations constrained to +1 or -1 , , 2016 . . https://doi.org/abs/1602.02830 https://doi.org/abs/1602.02830 , , . .

GE Dahl , , , D Yu , , , L Deng , , , 等 . . Context-dependentpre-trained deep neural networks for large-vocabularyspeech recognition . . IEEE Trans Audio Speech Lang Process , , 2012 . . 20 ( ( 1 ): ): 30 - - 42 . . DOI: 10.1109/tasl.2011.2134090 http://doi.org/10.1109/tasl.2011.2134090 . .

M Denil , , , B Shakibi , , , L Dinh , , , 等 . . Predicting parameters in deep learning . . 26 th Int Conf on Neural Information Processing Systems , , 2013 . . p.2148 - - 2156 . . . .

J Duchi , , , E Hazan , , , Y Singer . . Adaptive subgradientmethods for online learning and stochastic optimization . . J Mach Learn Res , , 2011 . . 12 2121 - - 2159 . . . .

K Goto , , , RA van de Geijn . . Anatomy of highperformance matrix multiplication . . ACM Trans Math Softw , , 2008 . . 34 ( ( 3 ): ): Article 12 DOI: 10.1145/1356052.1356053 http://doi.org/10.1145/1356052.1356053 . .

S Gupta , , , A Agrawal , , , K Gopalakrishnan , , , 等 . . Deeplearning with limited numerical precision . . Proc 32nd Int Conf on Machine Learning , , 2015 . . p.1737 - - 1746 . . . .

P Hammarlund , , , AJ Martinez , , , AA Bajwa , , , 等 . . Haswell: the fourth-generation Intel core processor . . IEEE Micro , , 2014 . . 34 ( ( 2 ): ): 6 - - 20 . . DOI: 10.1109/MM.2014.10 http://doi.org/10.1109/MM.2014.10 . .

S Han , , , J Pool , , , J Tran , , , 等 . . Learning both weightsand connections for efficient neural network . . Proc 28 th Int Conf on Neural Information Processing Systems , , 2015 . . p.1135 - - 1143 . . . .

S Han , , , JL Kang , , , HZ Mao , , , 等 . . ESE: efficient speechrecognition engine with sparse LSTM on FPGA . . Proc ACM/SIGDA Int Symp on Field-Programmable Gate Arrays , , 2017 . . p.75 - - 84 . . DOI: 10.1145/3020078.3021745 http://doi.org/10.1145/3020078.3021745 . .

TX He , , , YC Fan , , , YM Qian , , , 等 . . Reshaping deepneural network for fast decoding by node-pruning . . Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing , , 2014 . . p.245 - - 249 . . DOI: 10.1109/ICASSP.2014.6853595 http://doi.org/10.1109/ICASSP.2014.6853595 . .

G Hinton , , , L Deng , , , D Yu , , , 等 . . Deep neural networksfor acoustic modeling in speech recognition: the sharedviews of four research groups . . IEEE Signal Process Mag , , 2012 . . 29 ( ( 6 ): ): 82 - - 97 . . DOI: 10.1109/msp.2012.2205597 http://doi.org/10.1109/msp.2012.2205597 . .

G Hinton , , , O Vinyals , , , J Dean . . Distilling the knowledgein a neural network , , 2015 . . https://doi.org/abs/1503.02531 https://doi.org/abs/1503.02531 , , . .

I Hubara , , , M Courbariaux , , , D Soudry , , , 等 . . Quantizedneural networks: training neural networks with low precision weights and activations , , 2016 . . https://doi.org/abs/1609.07061 https://doi.org/abs/1609.07061 , , . .

S Ioffe , , , C Szegedy . . Batch normalization: acceleratingdeep network training by reducing internal covariateshift . . 32nd Int Conf on Machine Learning , , 2015 . . p.448 - - 456 . . . .

N Jaitly , , , P Nguyen , , , A Senior . . Applicationof pretrained deep neural networks to large vocabularyspeech recognition . . Proc 13 th Annual Conf of the Int Speech Communication Association , , 2012 . . .

D Kingma , , , J Ba . . Adam: a method for stochasticoptimization , , 2014 . . https://doi.org/abs/1412.6980 https://doi.org/abs/1412.6980 , , . .

JY Li , , , ML Seltzer , , , X Wang , , , 等 . . Large-scale domainadaptation via teacher-student learning . . Proc 18 th Annual Conf of Int Speech Communication Association , , 2017 . . p.2386 - - 2390 . . DOI: 10.21437/Interspeech.2017-519 http://doi.org/10.21437/Interspeech.2017-519 . .

TM Low , , , FD Igual , , , TM Smith . . Analyticalmodeling is enough for high-performance BLIS . . ACM Trans Math Softw , , 2016 . . 43 ( ( 2 ): ): Article 12 DOI: 10.1145/2925987 http://doi.org/10.1145/2925987 . .

L Lu , , , S Renals . . Small-footprint highway deep neuralnetworks for speech recognition . . IEEE/ACM Trans Audio Speech Lang Process , , 2017 . . 25 ( ( 7 ): ): 1502 - - 1511 . . DOI: 10.1109/TASLP.2017.2698723 http://doi.org/10.1109/TASLP.2017.2698723 . .

L Lu , , , M Guo , , , S Renals . . Knowledge distillation forsmall-footprint highway networks . . Proc IEEE Int Confon Acoustics, Speech and Signal Processing , , 2017 . . p.4820 - - 4824 . . DOI: 10.1109/ICASSP.2017.7953072 http://doi.org/10.1109/ICASSP.2017.7953072 . .

AR Mohamed , , , GE Dahl , , , GE Hinton . . Acoustic modeling using deep belief networks . . IEEE Trans Audio Speech Lang Process , , 2012 . . 20 ( ( 1 ): ): 14 - - 22 . . DOI: 10.1109/TASL.2011.2109382 http://doi.org/10.1109/TASL.2011.2109382 . .

A Novikov , , , D Podoprikhin , , , A Osokin , , , 等 . . Tensorizing neural networks . . Advances in Neural Information Processing Systems , , 2015 . . p.442 - - 450 . . . .

D Povey , , , A Ghoshal , , , G Boulianne , , , 等 . . The Kaldispeech recognition toolkit . . Proc IEEE Workshop on Automatic Speech Recognition and Understanding , , 2011 . . .

YM Qian , , , PC Woodland . . Very deep convolutionalneural networks for robust speech recognition . . Proc IEEE Spoken Language Technology Workshop , , 2016 . . p.481 - - 488 . . DOI: 10.1109/SLT.2016.7846307 http://doi.org/10.1109/SLT.2016.7846307 . .

YM Qian , , , TX He , , , W Deng , , , 等 . . Automatic modelredundancy reduction for fast back-propagation for deepneural networks in speech recognition . . Proc Int Joint Conf on Neural Networks , , 2015 . . p.1 - - 6 . . DOI: 10.1109/IJCNN.2015.7280335 http://doi.org/10.1109/IJCNN.2015.7280335 . .

YM Qian , , , MX Bi , , , T Tan , , , 等 . . Very deep convolutional neural networks for noise robust speech recognition . . IEEE/ACM Trans Audio Speech Lang Process , , 2016 . . 24 ( ( 12 ): ): 2263 - - 2276 . . DOI: 10.1109/TASLP.2016.2602884 http://doi.org/10.1109/TASLP.2016.2602884 . .

M Rastegari , , , V Ordonez , , , J Redmon , , , 等 . . XNORNet: ImageNet classification using binary convolutionalneural networks . . Proc 14 th European Conf on Computer Vision , , 2016 . . p.525 - - 542 . . DOI: 10.1007/978-3-319-46493-0_32 http://doi.org/10.1007/978-3-319-46493-0_32 . .

TN Sainath , , , AR Mohamed , , , B Kingsbury , , , 等 . . Deepconvolutional neural networks for LVCSR . . Proc IEEE Int Conf on Acoustics, Speech and Signal Processing , , 2013 . . p.8614 - - 8618 . . DOI: 10.1109/ICASSP.2013.6639347 http://doi.org/10.1109/ICASSP.2013.6639347 . .

H Sak , , , A Senior , , , F Beaufays . . Long short-term memoryrecurrent neural network architectures for large scaleacoustic modeling . . Proc 15 th Annual Conf of Int Speech Communication Association , , 2014 . . p.338 - - 342 . . . .

G Saon , , , G Kurata , , , T Sercu , , , 等 . . English conversational telephone speech recognition by humans andmachines , , 2017 . . https://doi.org/abs/1703.02136 https://doi.org/abs/1703.02136 , , . .

T Sercu , , , C Puhrsch , , , B Kingsbury , , , 等 . . Very deepmultilingual convolutional neural networks for LVCSR . . Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing , , 2016 . . p.4955 - - 4959 . . DOI: 10.1109/icassp.2016.7472620 http://doi.org/10.1109/icassp.2016.7472620 . .

YQ Wang , , , JY Li , , , YF Gong . . Small-footprint highperformance deep neural network-based speech recognition using split-VQ . . Proc IEEE Int Conf on Acoustics, Speech and Signal Processing , , 2015 . . p.4984 - - 4988 . . DOI: 10.1109/ICASSP.2015.7178919 http://doi.org/10.1109/ICASSP.2015.7178919 . .

W Xiong , , , J Droppo , , , X Huang , , , 等 . . Achieving humanparity in conversational speech recognition , , 2016 . . https://doi.org/abs/1610.05256 https://doi.org/abs/1610.05256 , , . .

W Xiong , , , J Droppo , , , X Huang , , , 等 . . The Microsoft2016 conversational speech recognition system . . Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing , , 2017 . . p.5255 - - 5259 . . DOI: 10.1109/icassp.2017.7953159 http://doi.org/10.1109/icassp.2017.7953159 . .

J Xue , , , JY Li , , , YF Gong . . Restructuring of deep neuralnetwork acoustic models with singular value decomposition . . Proc 14 th Annual Conf of Int Speech Communication Association , , 2013 . . p.2365 - - 2369 . . . .

S Young , , , G Evermann , , , M Gales , , , 等 . . The HTK Book . . Cambridge University Engineering Department, Cambridge, UK , , 2006 . . .

D Yu , , , F Seide , , , G Li , , , 等 . . Exploiting sparsenessin deep neural networks for large vocabulary speechrecognition . . Proc IEEE Int Conf on Acoustics, Speech, and Signal Processing , , 2012 . . p.4409 - - 4412 . . DOI: 10.1109/ICASSP.2012.6288897 http://doi.org/10.1109/ICASSP.2012.6288897 . .

D Yu , , , W Xiong , , , J Droppo , , , 等 . . Deep convolutionalneural networks with layer-wise context expansion andattention . . Proc 17 th Annual Conf of Int Speech Communication Association , , 2016 . . p.17 - - 21 . . DOI: 10.21437/Interspeech.2016-251 http://doi.org/10.21437/Interspeech.2016-251 . .

SC Zhou , , , YX Wu , , , ZK Ni , , , 等 . . DoReFa-Net: traininglow bitwidth convolutional neural networks with lowbitwidth gradients , , 2016 . . https://doi.org/abs/1606.06160 https://doi.org/abs/1606.06160 , , . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Knowledge distillation for financial large language models: a systematic review of strategies, applications, and evaluation

Prototype-guided cross-task knowledge distillation

Federated learning on non-IID and long-tailed data via dual-decoupling

Multi-exit self-distillation with appropriate teachers

Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives

Related Author

Jianzong WANG

Junfei XIE

Xiaoyang QU

Xulong ZHANG

Jiaqi SHI

Deng LI

Peng LI

Aming WU

Related Institution

Institute of Advanced Technology, University of Science and Technology of China

Ping An Technology (Shenzhen) Co., Ltd.

School of Electronic Engineering, Xidian University

Songshan Laboratory

College of Intelligence and Computing, Tianjin University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰