管理面向任务的虚拟助手软件系统的经验性研究

李姝玥; 郭家琪; 高妍; 楼建光; 杨德建; 肖炎; 周亚东; 刘烃

doi:10.1631/FITEE.2100467

Your Location：

Home >

Browse articles >

管理面向任务的虚拟助手软件系统的经验性研究

软件数据智能分析专题 | Updated：2022-05-23

- 管理面向任务的虚拟助手软件系统的经验性研究
- How to manage a task-oriented virtual assistant software project: an experience report
- 信息与电子工程前沿（英文） 2022年23卷第5期页码：749-762
- Affiliations：
  
  1.Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an710049, China
  2.Microsoft Research Asia, Beijing100080, China
- Author bio：
  
  E-mail: lishuyue1221@stu.xjtu.edu.cn;
  E-mail: jasperguo2013@stu.xjtu.edu.cn;
  E-mail: yan.gao@microsoft.com;
  E-mail: jlou@microsoft.com;
  E-mail: dejian.yang@microsoft.com;
  E-mail: yan.xiao@microsoft.com;
  E-mail: ydzhou@xjtu.edu.cn;
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2100467
  中图分类号： TP311.5
- 收稿：2021-09-30，
  
  录用：2022-01-30，
  
  纸质出版：2022-05-0
- Accepted：
Scan QR Code
李姝玥, 郭家琪, 高妍, 等. 管理面向任务的虚拟助手软件系统的经验性研究[J]. 信息与电子工程前沿（英文）, 2022,23(5):749-762.

Shuyue LI, Jiaqi GUO, Yan GAO, et al. How to manage a task-oriented virtual assistant software project: an experience report[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(5): 749-762.
李姝玥, 郭家琪, 高妍, 等. 管理面向任务的虚拟助手软件系统的经验性研究[J]. 信息与电子工程前沿（英文）, 2022,23(5):749-762. DOI： 10.1631/FITEE.2100467.

Shuyue LI, Jiaqi GUO, Yan GAO, et al. How to manage a task-oriented virtual assistant software project: an experience report[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(5): 749-762. DOI： 10.1631/FITEE.2100467.

摘要

面向任务的虚拟助手是为用户提供自然语言接口以完成特定领域任务的软件系统。随着近年来自然语言处理和机器学习技术的发展，越来越多面向任务的虚拟助手产品开始涌现。由于自然语言理解这一问题的复杂性和困难性，管理一个面向任务的虚拟助手软件项目具有挑战性。同时，据我们所知，与虚拟助手开发相关的管理和经验在学术界和工业界都少有研究或分享。为填补这空白，本文分享了我们在微软开发一项虚拟助手产品过程中的管理经验和教训。相信我们的经验和教训能为研究人员和相关从业者提供宝贵参考。最后，设计了一个需求管理工具SpecSpace，对我们虚拟助手项目的管理有很大帮助。

Abstract

Task-oriented virtual assistants are software systems that provide users with a natural language interface to complete domain-specific tasks. With the recent technological advances in natural language processing and machine learning

an increasing number of task-oriented virtual assistants have been developed. However

due to the well-known complexity and difficulties of the natural language understanding problem

it is challenging to manage a task-oriented virtual assistant software project. Meanwhile

the management and experience related to the development of virtual assistants are hardly studied or shared in the research community or industry

to the best of our knowledge. To bridge this knowledge gap

in this paper

we share our experience and the lessons that we have learned at managing a task-oriented virtual assistant software project at Microsoft. We believe that our practices and the lessons learned can provide a useful reference for other researchers and practitioners who aim to develop a virtual assistant system. Finally

we have developed a requirement management tool

named SpecSpace

which can facilitate the management of virtual assistant projects.

关键词

Keywords

references

Amershi S , Begel A , Bird C , et al. , 2019 . Software engineering for machine learning: a case study . Proc IEEE/ACM 41 st Int Conf on Software Engineering: Software Engineering in Practice , p. 291 - 300 . doi: 10.1109/ICSE-SEIP.2019.00042 http://doi.org/10.1109/ICSE-SEIP.2019.00042

Arpteg A , Brinne B , Crnkovic-Friis L , et al. , 2018 . Software engineering challenges of deep learning . Proc 44 th Euromicro Conf on Software Engineering and Advanced Applications , p. 50 - 59 . doi: 10.1109/SEAA.2018.00018 http://doi.org/10.1109/SEAA.2018.00018

Bender EM , Koller A , 2020 . Climbing towards NLU: on meaning, form, and understanding in the age of data . Proc 58 th Annual Meeting of the Association for Computational Linguistics , p. 5185 - 5198 . doi: 10.18653/v1/2020.acl-main.463 http://doi.org/10.18653/v1/2020.acl-main.463

Bonwell CC , Eison JA , 1991 . Active Learning: Creating Excitement in the Classroom. ERIC Number ED336049 . The George Washington University , Washington, USA .

Bradley AJ , 2020 . Brace Yourself for an Explosion of Virtual Assistants . https://blogs.gartner.com/anthony_bradley/2020/08/10/brace-yourself-for-an-explosion-of-virtual-assistants/ https://blogs.gartner.com/anthony_bradley/2020/08/10/brace-yourself-for-an-explosion-of-virtual-assistants/ [Accessed on Aug. 10, 2020 ].

Breck E , Cai SQ , Nielsen E , et al. , 2017 . The ML test score: a rubric for ML production readiness and technical debt reduction . Proc IEEE Int Conf on Big Data , p. 1123 - 1132 . doi: 10.1109/BigData.2017.8258038 http://doi.org/10.1109/BigData.2017.8258038

Campagna G , Xu SL , Moradshahi M , et al. , 2019 . Genie: a generator of natural language semantic parsers for virtual assistant commands . Proc 40 th ACM SIGPLAN Conf on Programming Language Design and Implementation , p. 394 - 410 . doi: 10.1145/3314221.3314594 http://doi.org/10.1145/3314221.3314594

Dhamdhere K , McCurley KS , Nahmias R , et al. , 2017 . Analyza: exploring data with conversation . Proc 22 nd Int Conf on Intelligent User Interfaces , p. 493 - 504 . doi: 10.1145/3025171.3025227 http://doi.org/10.1145/3025171.3025227

Facebook , 2020 . Surveybot . https://surveybot.io/ https://surveybot.io/ [Accessed on Aug. 10, 2020 ].

Gao Y , Lou JG , Zhang DM , 2019 . A hybrid semantic parsing approach for tabular data analysis . https://arxiv.org/abs/1910.10363

Hains G , Jakobsson A , Khmelevsky Y , 2018 . Towards formal methods and software engineering for deep learning: security, safety and productivity for DL systems development . Proc Annual IEEE Int Systems Conf , p. 1 - 5 . doi: 10.1109/SYSCON.2018.8369576 http://doi.org/10.1109/SYSCON.2018.8369576

Horkoff J , 2019 . Non-functional requirements for machine learning: challenges and new directions . Proc 27 th Int Requirements Engineering Conf , p. 386 - 391 . doi: 10.1109/RE.2019.00050 http://doi.org/10.1109/RE.2019.00050

Huang XW , Kroening D , Kwiatkowska M , et al. , 2018 . Safety and trustworthiness of deep neural networks: a survey . https://arxiv.org/abs/1812.08342v1

Islam J , Nguyen HA , Pan R , et al. , 2019 . What do developers ask about ML libraries? A large-scale study using stack overflow . https://arxiv.org/abs/1906.11940

Krishnan S , Wang JN , Wu E , et al. , 2016 . ActiveClean: interactive data cleaning for statistical modeling . Proc VLDB Endow , 9 ( 12 ): 948 - 959 . doi: 10.14778/2994509.2994514 http://doi.org/10.14778/2994509.2994514

Krishnan S , Franklin MJ , Goldberg K , et al. , 2017 . BoostClean: automated error detection and repair for machine learning . https://arxiv.org/abs/1711.01299

Lee DTS , Zhou ZQ , Tse TH , 2020 . Metamorphic robustness testing of Google Translate . Proc 42 nd Int Conf on Software Engineering Workshops , p. 388 - 395 . doi: 10.1145/3387940.3391484 http://doi.org/10.1145/3387940.3391484

Marijan D , Gotlieb A , Ahuja MK , 2019 . Challenges of testing machine learning based systems . Proc IEEE Int Conf on Artificial Intelligence Testing , p. 101 - 102 . doi: 10.1109/AITest.2019.00010 http://doi.org/10.1109/AITest.2019.00010

Mason L , Baxter J , Bartlett PL , et al. , 1999 . Boosting algorithms as gradient descent . Proc 12 th Int Conf on Neural Information Processing Systems , p. 512 - 518 .

Masuda S , Ono K , Yasue T , et al. , 2018 . A survey of software quality for machine learning applications . IEEE Int Conf on Software Testing, Verification and Validation Workshops , p. 279 - 284 . doi: 10.1109/ICSTW.2018.00061 http://doi.org/10.1109/ICSTW.2018.00061

Microsoft , 2020 . Power BI . https://powerbi.microsoft.com/ https://powerbi.microsoft.com/ [Accessed on Aug. 10, 2020 ].

Oram R , 2019 . Meeting Edward: Chatbots and the Changing Face of the Hotel Guest Experience . https://blogs.oracle.com/hospitality/chatbots-and-the-changing-the-face-of-the-hotel-guest-experience https://blogs.oracle.com/hospitality/chatbots-and-the-changing-the-face-of-the-hotel-guest-experience [Accessed on Aug. 10, 2020 ].

Polyzotis N , Roy S , Whang SE , et al. , 2018 . Data lifecycle challenges in production machine learning: a survey . ACM SIGMOD Rec , 47 ( 2 ): 17 - 28 . doi: 10.1145/3299887.3299891 http://doi.org/10.1145/3299887.3299891

Radford A , Wu J , Child R , et al. , 2019 . Language Models are Unsupervised Multitask Learners . https://openai.com/blog/ https://openai.com/blog/ [Accessed on Jan. 1, 2020 ].

Schapire RE , 1990 . The strength of weak learnability . Mach Learn , 5 ( 2 ): 197 - 227 . doi: 10.1007/BF00116037 http://doi.org/10.1007/BF00116037

Schelter S , Lange D , Schmidt P , et al. , 2018a . Automating large-scale data quality verification . Proc VLDB Endow , 11 ( 12 ): 1781 - 1794 . doi: 10.14778/3229863.3229867 http://doi.org/10.14778/3229863.3229867

Schelter S , Biessmann F , Januschowski T , et al. , 2018b . On challenges in machine learning model management . IEEE Data Eng Bull , 41 ( 4 ): 5 - 15 .

Sculley D , Holt G , Golovin D , et al. , 2015 . Hidden technical debt in machine learning systems . Proc 28 th Int Conf on Neural Information Processing Systems , p. 2503 - 2511 .

Sun NY , Yang XF , Liu YF , 2020 . TableQA: a large-scale Chinese text-to-SQL dataset for table-aware SQL generation . https://arxiv.org/abs/2006.06434v1

Tableau , 2020a . Ask Data . https://www.tableau.com/products/new-features/ask-data/ https://www.tableau.com/products/new-features/ask-data/ [Accessed on Aug. 10, 2020 ].

Tableau , 2020b . Tableau . https://www.tableau.com/ https://www.tableau.com/ [Accessed on Aug. 10, 2020 ].

Task Virtual , 2020 . TaskVirtual . https://taskvirtual.com/ https://taskvirtual.com/ [Accessed on Aug. 10, 2020 ].

Thrun S , 1998 . Lifelong learning algorithms . In: Thrun S , Pratt L (Eds.), Learning to Learn . Kluwer Academic Publishers , Norwell, USA , p. 181 - 209 .

Vogelsang A , Borg M , 2019 . Requirements engineering for machine learning: perspectives from data scientists . Proc 27 th Int Requirements Engineering Conf Workshops , p. 245 - 251 . doi: 10.1109/REW.2019.00050 http://doi.org/10.1109/REW.2019.00050

Voigt P , von dem Bussche A , 2017 . The EU General Data Protection Regulation (GDPR): a Practical Guide . Springer , Cham, Germany . doi: 10.1007/978-3-319-57959-7 http://doi.org/10.1007/978-3-319-57959-7

Wikipedia , 2021 . Virtual Assistant . https://en.wikipedia.org/wiki/Virtual_assistant https://en.wikipedia.org/wiki/Virtual_assistant [Accessed on Aug. 10, 2020 ].

Yao ZY , Su Y , Sun H , et al. , 2019 . Model-based interactive semantic parsing: a unified framework and a text-to-SQL case study . Proc Conf on Empirical Methods in Natural Language Processing and the 9 th Int Joint Conf on Natural Language Processing , p. 5447 - 5458 . doi: 10.18653/v1/D19-1547 http://doi.org/10.18653/v1/D19-1547

Young SWH , 2014 . Improving library user experience with A/B testing: principles and process . Weave J Libr User Exper . doi: 10.3998/weave.12535642.0001.101 http://doi.org/10.3998/weave.12535642.0001.101

Zhang JM , Harman M , Ma L , et al. , 2022 . Machine learning testing: survey, landscapes and horizons . IEEE Trans Softw Eng , 48 ( 1 ): 1 - 36 . doi: 10.1109/TSE.2019.2962027 http://doi.org/10.1109/TSE.2019.2962027

Zhang TY , Gao CY , Ma L , et al. , 2019 . An empirical study of common challenges in developing deep learning applications . Proc 30 th Int Symp on Software Reliability Engineering , p. 104 - 115 . doi: 10.1109/ISSRE.2019.00020 http://doi.org/10.1109/ISSRE.2019.00020

Zhong V , Xiong CM , Socher R , 2017 . Seq2SQL: generating structured queries from natural language using reinforcement learning . https://arxiv.org/abs/1709.00103v5

浏览量

178

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Towards sustainable adversarial training with successive perturbation generation

Advances and challenges in artificial intelligence text generation

A disk failure prediction model for multiple issues

Data-driven soft sensors in blast furnace ironmaking: a survey