Devising optimal integration test orders using cost–benefit analysis

Fanyi MENG; Ying WANG; Hai YU; Zhiliang ZHU

doi:10.1631/FITEE.2100466

Your Location：

Home >

Browse articles >

Devising optimal integration test orders using cost–benefit analysis

Special Feature on Intelligent Analysis for Software Data | Updated：2022-05-23

- Devising optimal integration test orders using cost–benefit analysis
- 基于成本收益分析的集成测试序列生成优化方法
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 5, Pages: 692-714(2022)
- Affiliations：
  
  1.Software College, Northeastern University, Shenyang110169, China
  2.State Key Lab for Novel Software Technology, Nanjing University, Nanjing210023, China
- Author bio：
  
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2100466
  CLC： TP311
- Received：30 September 2021，
  
  Accepted：11 January 2022，
  
  Published：0 May 2022
- Accepted：
Scan QR Code
Fanyi MENG, Ying WANG, Hai YU, et al. Devising optimal integration test orders using cost–benefit analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(5): 692-714.
DOI：

Fanyi MENG, Ying WANG, Hai YU, et al. Devising optimal integration test orders using cost–benefit analysis[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(5): 692-714. DOI： 10.1631/FITEE.2100466.

摘要

集成测试是面向对象软件测试的重要组成部分。传统的类级集成测试顺序研究策略大多围绕如何降低测试成本开展工作，并未考虑赋予可靠性风险较大的节点较高测试优先级，从而影响软件测试效率。本文提出一种兼顾测试成本与测试效率的方法生成集成测试序列。根据软件在不同场景下的运行状态，将其映射成多层动态执行网络（multi-layer dynamic execution network，MDEN）。借助该网络模型与概率风险评估方法为软件中每一个类赋予风险权重。利用成本收益分析方法，在生成测试用例的过程中保证两条原则：为高风险的类赋予较高权重，同时最小化测试桩复杂度。在此基础上，分析测试序列对软件系统总体运行风险的影响，从而提出评估测试序列优劣的度量方案。通过与现有算法的实验对比分析，证明所提算法生成的类级集成测试序列能有效降低测试代价。最后，将所提算法实现为自动生成集成测试序列的开源工具ITOsolution，。

Abstract

Integration testing is an integral part of software testing. Prior studies have focused on reducing test cost in integration test order generation. However

there are no studies concerning the testing priorities of critical classes when generating integration test orders. Such priorities greatly affect testing efficiency. In this study

we propose an effective strategy that considers both test cost and efficiency when generating test orders. According to a series of dynamic execution scenarios

the software is mapped into a multi-layer dynamic execution network (MDEN) model. By analyzing the dynamic structural complexity

an evaluation scheme is proposed to quantify the class testing priority with the defined class risk index. Cost–benefit analysis is used to perform cycle-breaking operations

satisfying two principles: assigning higher priorities to higher-risk classes and minimizing the total complexity of test stubs. We also present a strategy to evaluate the effectiveness of integration test order algorithms by calculating the reduction of software risk during their testing process. Experiment results show that our approach performs better across software of different scales

in comparison with the existing algorithms that aim only to minimize test cost. Finally

we implement a tool

ITOsolution

to help practitioners automatically generate test orders.

关键词

Keywords

references

Abdurazik A , Offutt J , 2006 . Coupling-based class integration and test order . Proc Int Workshop on Automation of Software Test , p. 50 - 56 . doi: 10.1145/1138929.1138940 http://doi.org/10.1145/1138929.1138940

Al Mannai WI , Lewis TG , 2008 . A general defender-attacker risk model for networks . J Risk Finance , 9 ( 3 ): 244 - 261 . doi: 10.1108/15265940810875577 http://doi.org/10.1108/15265940810875577

Amland S , 2000 . Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study . J Syst Softw , 53 ( 3 ): 287 - 295 . doi: 10.1016/S0164-1212(00)00019-4 http://doi.org/10.1016/S0164-1212(00)00019-4

Assunção WKG , Colanzi TE , Vergilio SR , et al. , 2014 . A multi-objective optimization approach for the integration and test order problem . Inform Sci , 267 : 119 - 139 . doi: 10.1016/j.ins.2013.12.040 http://doi.org/10.1016/j.ins.2013.12.040

Bang L , Aydin A , Bultan T , 2015 . Automatically computing path complexity of programs . Proc 10 th Joint Meeting on Foundations of Software Engineering , p. 61 - 72 . doi: 10.1145/2786805.2786863 http://doi.org/10.1145/2786805.2786863

Bansal P , Sabharwal S , Sidhu P , 2009 . An investigation of strategies for finding test order during integration testing of object oriented applications . Proc Int Conf on Methods and Models in Computer Science , p. 1 - 8 . doi: 10.1109/ICM2CS.2009.5397936 http://doi.org/10.1109/ICM2CS.2009.5397936

Binder RV , 1996 . Testing object-oriented software: a survey . Softw Test Verif Reliab , 6 ( 3-4 ): 125 - 252 . doi: 10.1002/(SICI)1099-1689(199609/12)6:3/4<125::AID-STVR121>3.0.CO;2-X http://doi.org/10.1002/(SICI)1099-1689(199609/12)6:3/4<125::AID-STVR121>3.0.CO;2-X

Bowring JF , Rehg JM , Harrold MJ , 2004 . Active learning for automatic classification of software behavior . ACM SIGSOFT Softw Eng Notes , 29 ( 4 ): 195 - 205 . doi: 10.1145/1013886.1007539 http://doi.org/10.1145/1013886.1007539

Briand LC , Jie F , Labiche Y , 2002 . Experimenting with genetic algorithms to devise optimal integration test orders . Technical Report, No. SCE-02-03 , Carleton University , Ottawa, Canada .

Briand LC , Labiche Y , Wang YH , 2003 . An investigation of graph-based class integration test order strategies . IEEE Trans Softw Eng , 29 ( 7 ): 594 - 607 . doi: 10.1109/TSE.2003.1214324 http://doi.org/10.1109/TSE.2003.1214324

Cai KY , Yin BB , 2009 . Software execution processes as an evolving complex network . Inform Sci , 179 ( 12 ): 1903 - 1928 . doi: 10.1016/j.ins.2009.01.011 http://doi.org/10.1016/j.ins.2009.01.011

da Veiga Cabral R , Pozo A , Vergilio SR , 2010 . A Pareto ant colony algorithm applied to the class integration and test order problem . Proc 22 nd IFIP WG 6.1 Int Conf on Testing Software and Systems , p. 16 - 29 . doi: 10.1007/978-3-642-16573-3_3 http://doi.org/10.1007/978-3-642-16573-3_3

Deb K , Pratap A , Agarwal S , et al. , 2002 . A fast and elitist multiobjective genetic algorithm: NSGA-II . IEEE Trans Evol Comput , 6 ( 2 ): 182 - 197 . doi: 10.1109/4235.996017 http://doi.org/10.1109/4235.996017

Floyd RW , 1962 . Algorithm 97: shortest path . Commun ACM , 5 ( 6 ): 345 . doi: 10.1145/367766.368168 http://doi.org/10.1145/367766.368168

Frankl PG , Weyuker EJ , 2000 . Testing software to detect and reduce risk . J Syst Softw , 53 ( 3 ): 275 - 286 . doi: 10.1016/S0164-1212(00)00018-2 http://doi.org/10.1016/S0164-1212(00)00018-2

Goseva-Popstojanova K , Hassan A , Guedem A , et al. , 2003 . Architectural-level risk analysis using UML . IEEE Trans Softw Eng , 29 ( 10 ): 946 - 960 . doi: 10.1109/TSE.2003.1237174 http://doi.org/10.1109/TSE.2003.1237174

Hao D , Zhang L , Zang L , et al. , 2016 . To be optimal or not in test-case prioritization . IEEE Trans Softw Eng , 42 ( 5 ): 490 - 505 . doi: 10.1109/TSE.2015.2496939 http://doi.org/10.1109/TSE.2015.2496939

Henry S , Kafura D , 1981 . Software structure metrics based on information flow . IEEE Trans Softw Eng , 7 ( 5 ): 510 - 518 . doi: 10.1109/TSE.1981.231113 http://doi.org/10.1109/TSE.1981.231113

Huang CY , Lyu MR , 2005 . Optimal testing resource allocation, and sensitivity analysis in software development . IEEE Trans Reliab , 54 ( 4 ): 592 - 603 . doi: 10.1109/TR.2005.858099 http://doi.org/10.1109/TR.2005.858099

Jiang SJ , Zhang YM , Li HY , et al. , 2011 . An approach for inter-class integration test order determination based on coupling measures . Chin J Comput , 34 ( 6 ): 1062 - 1074 . doi: 10.3724/SP.J.1016.2011.01062 http://doi.org/10.3724/SP.J.1016.2011.01062

Jiang SJ , Zhang M , Zhang YM , et al. , 2021 . An integration test order strategy to consider control coupling . IEEE Trans Softw Eng , 47 ( 7 ): 1350 - 1367 . doi: 10.1109/TSE.2019.2921965 http://doi.org/10.1109/TSE.2019.2921965

Jorgensen PC , Erickson C , 1994 . Object-oriented integration testing . Commun ACM , 37 ( 9 ): 30 - 38 . doi: 10.1145/182987.182989 http://doi.org/10.1145/182987.182989

Knowles JD , Corne DW , 2000 . Approximating the nondominated front using the Pareto archived evolution strategy . Evol Comput , 8 ( 2 ): 149 - 172 . doi: 10.1162/106365600568167 http://doi.org/10.1162/106365600568167

Kung D , Gao J , Hsia P , et al. , 1995 . A test strategy for object-oriented programs . Proc 19 th Annual Int Computer Software and Applications Conf , p. 239 - 244 . doi: 10.1109/CMPSAC.1995.524786 http://doi.org/10.1109/CMPSAC.1995.524786

Le Traon Y , Jeron T , Jezequel JM , et al. , 2000 . Efficient object-oriented integration and regression testing . IEEE Trans Reliab , 49 ( 1 ): 12 - 25 . doi: 10.1109/24.855533 http://doi.org/10.1109/24.855533

Lipow M , 1982 . Number of faults per line of code . IEEE Trans Softw Eng , 8 ( 4 ): 437 - 439 . doi: 10.1109/TSE.1982.235579 http://doi.org/10.1109/TSE.1982.235579

McCabe TJ , 1976 . A complexity measure . IEEE Trans Softw Eng , 2 ( 4 ): 308 - 320 . doi: 10.1109/TSE.1976.233837 http://doi.org/10.1109/TSE.1976.233837

Monden A , Hayashi T , Shinoda S , et al. , 2013 . Assessing the cost effectiveness of fault prediction in acceptance testing . IEEE Trans Reliab , 39 ( 10 ): 1345 - 1357 . doi: 10.1109/TSE.2013.21 http://doi.org/10.1109/TSE.2013.21

Myers CR , 2003 . Software systems as complex networks: structure, function, and evolvability of software collaboration graphs . Phys Rev E , 68 ( 2 ): 046116 . doi: 10.1103/PhysRevE.68.046116 http://doi.org/10.1103/PhysRevE.68.046116

NASA , 1999 . Pyroshock Test Crite . Technical Report, No. NASA-STD-7003A . NASA , Washington, USA .

Sharma C , Sibal R , 2013 . Application of different metaheuristic techniques for finding optimal test order during integration testing of object oriented systems and their comparative study . Int J Soft Comput Eng , 3 ( 12 ): 1 - 19 .

Tai KC , Daniels JF , 1999 . Interclass test order for object-oriented software . J Obj-Orient Progr , 12 ( 4 ): 18 - 25 .

Vergilio SR , Pozo A , Árias JCG , et al. , 2012 . Multi-objective optimization algorithms applied to the class integration and test order problem . Int J Softw Tools Technol Transf , 14 ( 4 ): 461 - 475 . doi: 10.1007/s10009-012-0226-1 http://doi.org/10.1007/s10009-012-0226-1

Walters C , Ludwig D , 1994 . Calculation of Bayes posterior probability distributions for key population parameters . Can J Fish Aquat Sci , 51 ( 3 ): 713 - 722 . doi: 10.1139/f94-071 http://doi.org/10.1139/f94-071

Wang Y , Zhu ZL , Yang B , et al. , 2018a . Using reliability risk analysis to prioritize test cases . J Syst Softw , 139 : 14 - 31 . doi: 10.1016/j.jss.2018.01.033 http://doi.org/10.1016/j.jss.2018.01.033

Wang Y , Zhu ZL , Yu H , et al. , 2018b . Risk analysis on multi-granular flow network for software integration testing . IEEE Trans Circ Syst II Expr Briefs , 65 ( 8 ): 1059 - 1063 . doi: 10.1109/TCSII.2017.2775442 http://doi.org/10.1109/TCSII.2017.2775442

Wang ZS , Li BX , Wang LL , et al. , 2011 . A brief survey on automatic integration test order generation . Proc 23 rd Int Conf on Software Engineering & Knowledge Engineering , p. 254 - 257 .

Weyuker EJ , 1988 . Evaluating software complexity measures . IEEE Trans Softw Eng , 14 ( 9 ): 1357 - 1365 . doi: 10.1109/32.6178 http://doi.org/10.1109/32.6178

Xu C , Qin Y , Yu P , et al. , 2020 . Theories and techniques for growing software: paradigm and beyond . Sci Sin Inform , 50 ( 11 ): 1595 - 1611 . doi: 10.1360/SSI-2020-0079 http://doi.org/10.1360/SSI-2020-0079

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰