RESEARCH PAPER
A novel test case prioritization method based on problems of numerical software code statement defect prediction
1
School of Reliability and Systems Engineering Beihang University No.37 Xueyuan RD. Haidian, 100191, Beijing, China
2
Ji'an Municipal Industry and Information Technology Bureau 11/F, Block B, Administration Center Building Jizhou District, Ji'an, Jiangxi, 343000, China
Publication date: 2020-09-30
Eksploatacja i Niezawodność – Maintenance and Reliability 2020;22(3):419-431
KEYWORDS
software defect predictiontest case prioritizationcode statement metricsmachine learningsoftware testing
ABSTRACT
Test case prioritization (TCP) has been considerably utilized to arrange the implementation order of test cases, which contributes
to improve the efficiency and resource allocation of software regression testing. Traditional coverage-based TCP techniques, such
as statement-level, method/function-level and class-level, only leverages program code coverage to prioritize test cases without
considering the probable distribution of defects. However, software defect data tends to be imbalanced following Pareto principle.
Instinctively, the more vulnerable the code covered by the test case is, the higher the priority it is. Besides, statement-level coverage is a more fine-grained method than function-level coverage or class-level coverage, which can more accurately formulate
test strategies. Therefore, we present a test case prioritization approach based on statement software defect prediction to tame
the limitations of current coverage-based techniques in this paper. Statement metrics in the source code are extracted and data
pre-processing is implemented to train the defect predictor. And then the defect detection rate of test cases is calculated by combining the prioritization strategy and prediction results. Finally, the prioritization performance is evaluated in terms of average
percentage faults detected in four open source datasets. We comprehensively compare the performance of the proposed method
under different prioritization strategies and predictors. The experimental results show it is a promising technique to improve the
prevailing coverage-based TCP methods by incorporating statement-level defect-proneness. Moreover, it is also concluded that
the performance of the additional strategy is better than that of max and total, and the choice of the defect predictor affects the
efficiency of the strategy.
REFERENCES (51)
1.
Arafeen M J, Do H. Test case prioritization using requirements-based clustering. Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation 2013: 312-321, https://doi.org/10.1109/ICST.2....
2.
Arar O F, Ayan K. A feature dependent naive bayes approach and its application to the software defect prediction problem. Applied Soft Computing 2017; 59: 197-209, https://doi.org/10.1016/j.asoc....
3.
Arar O F, Ayan K. Software defect prediction using cost-sensitive neural network. Applied Soft Computing 2015; 33(C): 263-277, https://doi.org/10.1016/j.asoc....
4.
Bertolino A, Miranda B, Pietrantuono R, et al. Adaptive coverage and operational profile-based testing for reliability improvement. Proceedings of the 39th International Conference on Software Engineering 2017: 541-551.
5.
Boehm B, Basili V R. Software defect reduction top 10 list. Computer 2001; 34(1): 135-137, https://doi.org/10.1109/2.9629....
6.
Catal C, Diri B. Investigating the effect of dataset size, metrics sets, and feature selection techniques on software fault prediction problem. Information Sciences 2009; 179(8): 1040-1058, https://doi.org/10.1016/j.ins.....
7.
Catal C, Mishra D. Test case prioritization: A systematic mapping study. Software Quality Journal 2013; 21(3): 445-478, https://doi.org/10.1007/s11219....
8.
Di Nardo D, Alshahwan N, Briand L, et al. Coverage-based regression test case selection, minimization and prioritization: A case study on an industrial system. Software Testing, Verification and Reliability 2015; 25(4): 371-396, https://doi.org/10.1002/stvr.1....
9.
Do H, Mirarab S, Tahvildari L, et al. The effects of time constraints on test case prioritization: A series of controlled experiments. IEEE Transactions on Software Engineering 2010; 36(5): 593-617, https://doi.org/10.1109/TSE.20....
10.
Durelli V H S, Durelli R S, Borges S S, et al. Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 2019; 68(3): 1189-1212, https://doi.org/10.1109/TR.201....
11.
Elbaum S, Malishevsky A G, Rothermel G. Test case prioritization: A family of empirical studies. IEEE Transactions on Software Engineering 2002; 28(2): 159-182, https://doi.org/10.1109/32.988....
12.
Friedman J H. Stochastic gradient boosting. Computational statistics & data analysis 2002; 38(4): 367-378.
13.
Garg D, Datta A. Test case prioritization due to database changes in web applications. Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation 2012: 726-730.
14.
Hall M, Frank E, Holmes G, et al. The weka data mining software: An update. ACM SIGKDD explorations newsletter 2009; 11(1): 10-18.
15.
Harrold M J. Testing: A roadmap. Proceedings of the Future of Software Engineering 2000: 61-72.
16.
He P, Li B, Liu X, et al. An empirical study on software defect prediction with a simplified metric set. Information and Software Technology 2015; 59(C): 170-190, https://doi.org/10.1016/j.infs....
17.
Henard C, Papadakis M, Harman M, et al. Comparing white-box and black-box test prioritization. Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) 2016: 523-534, https://doi.org/10.1145/288478....
18.
Hosseini S, Turhan B, Mäntylä M. A benchmark study on the effectiveness of search-based data selection and feature selection for cross project defect prediction. Information and Software Technology 2017; 95: 296-312,https://doi.org/10.1016/j.infs....
19.
Hovemeyer D, Pugh W. Finding bugs is easy. Acm sigplan notices 2004; 39(12): 92-106, https://doi.org/10.1145/105288....
20.
Huang C Y, Chang J R, Chang Y H. Design and analysis of gui test-case prioritization using weight-based methods. Journal of Systems and Software 2010; 83(4): 646-659, https://doi.org/10.1016/j.jss.....
21.
Khatibsyarbini M, Isa M A, Jawawi D N A, et al. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology 2018; 93: 74-93, https://doi.org/ 10.1016/j.infsof.2017.08.014.
22.
Krishnamoorthi R, Sahaaya Arul Mary S A. Requirement based system test case prioritization of new and regression test cases. International Journal of Software Engineering and Knowledge Engineering 2009; 19(3): 453-475, https://doi.org/10.1142/S02181....
23.
Lachmann R, Schulze S, Nieke M, et al. System-level test case prioritization using machine learning. Proceedings of the 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA) 2016: 361-368, https://doi.org/10.1109/ICMLA.....
24.
Laradji I H, Alshayeb M, Ghouti L. Software defect prediction using ensemble learning on selected features. Information and Software Technology 2015; 58: 388-402, https://doi.org/10.1016/j.infs....
25.
Lessmann S, Baesens B, Mues C, et al. Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Transactions on Software Engineering 2008; 34(4): 485-496, https://doi.org/10.1109/TSE.20....
26.
Li Z, Harman M, Hierons R M. Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering 2007; 33(4): 225-237, https://doi.org/10.1109/TSE.20....
27.
Mei H, Hao D, Zhang L, et al. A static approach to prioritizing junit test cases. IEEE Transactions on Software Engineering 2012; 38(6): 1258-1275, https://doi.org/10.1109/TSE.20....
28.
Menzies T, Greenwald J, Frank A. Data mining static code attributes to learn defect predictors. IEEE Transactions on Software Engineering 2006; 33(1): 2-13, https://doi.org/10.1109/TSE.20....
29.
Mirarab S, Tahvildari L. A prioritization approach for software test cases based on bayesian networks. Proceedings of the International Conference on Fundamental Approaches to Software Engineering 2007: 279-290.
30.
Nam J, Kim S. Clami: Defect prediction on unlabeled datasets. Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) 2015: 452-463, https://doi.org/10.1109/ASE.20....
31.
Ohlsson N, Alberg H. Predicting fault-prone software modules in telephone switches. IEEE Transactions on Software Engineering 1996; 22(12): 886-894, https://doi.org/10.1109/32.553....
32.
Paterson D, Campos J, Abreu R, et al. An empirical study on the use of defect prediction for test case prioritization. Proceedings of the 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) 2019: 346-357, https://doi.org/10.1109/ICST.2....
33.
Peng R, Zhai Q. Modeling of software fault detection and correction processes with fault dependency. Eksploatacja i Niezawodnosc – Maintenance and Reliability 2017; 19(3): 467-475, https://doi.org/10.17531/ein.2....
34.
Peng X, Liu B, Wang S. Feedback-based integrated prediction: Defect prediction based on feedback from software testing process. Journal of Systems and Software 2018; 143: 159–171, https://doi.org/10.1016/j.jss.....
35.
Rothermel G, Untch R H, Chu C, et al. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering 2001; 27(10): 929-948, https://doi.org/10.1109/32.962....
36.
Rothermel G, Untch R H, Chu C, et al. Test case prioritization: An empirical study. Proceedings of the IEEE International Conference on Software Maintenance-1999 (ICSM'99)'Software Maintenance for Business Change'(Cat No 99CB36360) 1999: 179-188.
37.
Shao Y, Liu B, Wang S, et al. A novel software defect prediction based on atomic class-association rule mining. Expert Systems with Applications 2018; 114: 237-254, https://doi.org/10.1016/j.eswa....
38.
Shatnawi O. Measuring commercial software operational reliability: An interdisciplinary modelling approach. Eksploatacja i Niezawodnosc – Maintenance and Reliability, 2014; 16(4): 585-594.
39.
Song Q, Jia Z, Shepperd M, et al. A general software defect-proneness prediction framework. IEEE Transactions on Software Engineering 2011; 37(3): 356-370, https://doi.org/10.1109/TSE.20....
40.
Spieker H, Gotlieb A, Marijan D, et al. Reinforcement learning for automatic test case prioritization and selection in continuous integration. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis 2017: 12-22, https://doi. org/10.1145/3092703.3092709.
41.
Srikanth H, Hettiarachchi C, Do H. Requirements based test prioritization using risk factors: An industrial study. Information and Software Technology 2016; 69: 71-83, https://doi.org/10.1016/j.infs....
42.
Sun Z, Song Q, Zhu X. Using coding-based ensemble learning to improve software defect prediction. IEEE Transactions on Systems Man and Cybernetics Part C 2012; 42(6): 1806-1817, https://doi.org/10.1109/TSMCC.....
43.
Tonella P, Avesani P, Susi A. Using the case-based ranking methodology for test case prioritization. Proceedings of the 2006 22nd IEEE International Conference on Software Maintenance 2006: 123-133.
44.
Turhan B, Menzies T, Bener A c, ,e B., et al. On the relative value of cross-company and within-company data for defect prediction. Empirical Software Engineering 2009; 14(5): 540-578, https://doi.org/10.1007/s10664....
45.
Wang H, Khoshgoftaar T M, Napolitano A. Software measurement data reduction using ensemble techniques. Neurocomputing 2012; 92(3):124-132, https://doi.org/10.1016/j.neuc....
46.
Wang S, Nam J, Tan L. Qtep: Quality-aware test case prioritization. Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering 2017: 523-534, https://doi.org/10.1145/310623....
47.
Wang S, Yao X. Using class imbalance learning for software defect prediction. IEEE Transactions on Reliability 2013; 62(2): 434-443, https://doi.org/10.1109/TR.201....
48.
Xiao L, Miao H, Zhuang W, et al. An empirical study on clustering approach combining fault prediction for test case prioritization. Proceedings of the 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS) 2017: 815-820.
49.
Xu J, Meng Z, Xu L. Integrated system of health management-oriented reliability prediction for a spacecraft software system with an adaptive genetic algorithm support vector machine. Eksploatacja i Niezawodnosc – Maintenance and Reliability 2014; 16(4): 571-578.
50.
Yang X, Tang K, Yao X. A learning-to-rank approach to software defect prediction. IEEE Transactions on Reliability 2015; 64(1): 234-246, https://doi.org/10.1109/TR.201....
51.
Yoo S, Harman M. Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability 2012; 22(2): 67-120, https://doi.org/10.1002/stv.43....
CITATIONS (3):
1.
Role of Machine Learning in Software Testing
Vedpal, Naresh Chauhan
2021 5th International Conference on Information Systems and Computer Networks (ISCON)
Vedpal, Naresh Chauhan
2021 5th International Conference on Information Systems and Computer Networks (ISCON)
2.
Software Defect Prediction and Analysis Using Enhanced Random Forest (extRF) Technique: A Business Process Management and Improvement Concept in IOT-Based Application Processing Environment
Fahad Alshammari, Sikandar Ali
Mobile Information Systems
Fahad Alshammari, Sikandar Ali
Mobile Information Systems
Share
RELATED ARTICLE
| eISSN: | 2956-3860 |
| ISSN: | 1507-2711 |
We process personal data collected when visiting the website. The function of obtaining information about users and their behavior is carried out by voluntarily entered information in forms and saving cookies in end devices. Data, including cookies, are used to provide services, improve the user experience and to analyze the traffic in accordance with the Privacy policy. Data are also collected and processed by Google Analytics tool (more).
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
