How Gamekins Changed the Way Students Approach Software Testing

Table of Links

Abstract and 1 Introduction

2.1 Software Testing

2.2 Gamification of Software Testing

3 Gamifying Continuous Integration and 3.1 Challenges in Teaching Software Testing

3.2 Gamification Elements of Gamekins

3.3 Gamified Elements and the Testing Curriculum

4 Experiment Setup and 4.1 Software Testing Course

4.2 Integration of Gamekins and 4.3 Participants

4.4 Data Analysis

4.5 Threats to Validity

5.1 RQ1: How did the students use Gamekins during the course?

5.2 RQ2: What testing behavior did the students exhibit?

5.3 RQ3: How did the students perceive the integration of Gamekins into their projects?

6 Related Work

7 Conclusions, Acknowledgments, and References

6 RELATED WORK

Gamification has been explored for incentivizing practitioners to write tests and adopt good testing practices [24]. For this, gamification has been integrated into integrated development environments [47], browsers [23], and continuous integration [3], with results suggesting that gamification can improve test automation. There have also been attempts to gamify aspects of testing such as test-to-code traceability [37] or acceptance testing [40]. While we also address continuous integration, our gamification approach aims to influence testing behavior directly, rather than the adoption of tools, and we use more different gamification elements.

Gamification has also been applied to software testing education, incorporating different gamification elements and concepts. Unplugged approaches use tools like Lego or dice to teach testing concepts [33]. Others utilize virtual races [7], quizzes [12], business trainings [51], or board games [36]. Gamification has also been applied to tutorial systems for teaching software testing [16, 22] and to mutation testing [21]. Additionally, serious games have been developed to teach software testing [5, 46]. Most of these studies report that the use of gamification had a positive impact on testing, and our approach adds further evidence that this is the case. In contrast to prior work, our approach differs as gamification is integrated into the development process via CI, and it uses different gamification elements. However, gamification elements do not universally resolve the lack of motivation; a recent systematic mapping study identified negative effects such as lack of motivation and effectiveness in some cases [1]. In our study, such effects mainly occurred when there were technical issues with Gamekins.

7 CONCLUSIONS

Overall, the integration of the Gamekins plugin for Jenkins into the software testing course proved to be successful in engaging students in testing activities that go beyond formal requirements for the course using gamification elements such as challenges, quests, achievements, leaderboards, and avatars. This resulted in a significant improvement in the correctness of the programs they wrote as part of their coursework, and generally very positive feedback.

Our study revealed several opportunities to further improve Gamekins and its integration into a testing course. Specifically, the quests, challenges, and achievements should be reworked and refined to address issues such as when they are unsolvable with a better tracking of modified code. Additionally, an interesting suggestion made by students is to incorporate an IDE plugin to display challenges directly within the development environment. Ongoing work on Gamekins also includes the adoption of new programming languages and testing frameworks. While Gamekins has been effective in a university setting, evaluating its potential in an industry setting would also be valuable. Additionally, an evaluation using more complex student projects could provide valuable insights into team behavior and longer-term effects of Gamekins.

To support replications, all source code and experiment materials used in our study are available at: https://doi.org/10.6084/m9.figshare.23600925

Gamekins is available at: https://gamekins.org

ACKNOWLEDGMENTS

This work is supported by the DFG under grant FR 2955/2-1.

REFERENCES

[1] Cláuvin Almeida, Marcos Kalinowski, Anderson Uchôa, and Bruno Feijó. 2023. Negative effects of gamification in education software: Systematic mapping and practitioner perceptions. Information and Software Technology 156 (2023). https://doi.org/10.1016/j.infsof.2022.107142

[2] Mark Ardis, David Budgen, Gregory W. Hislop, Jeff Offutt, Mark Sebern, and Willem Visser. 2015. SE 2014: Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering. Computer 48, 11 (2015). https://doi.org/10. 1109/MC.2015.345

[3] Patrick Ayoup, Diego Elias Costa, and Emad Shihab. 2022. Achievement unlocked: a case study on gamifying DevOps practices in industry. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. https://doi.org/10.1145/3540250.3558948

[4] Kent L. Beck. 2003. Test-driven Development - by example. Addison-Wesley.

[5] Jonathan Bell, Swapneel Sheth, and Gail Kaiser. 2011. Secret ninja testing with HALO software engineering. In Proceedings of the 4th international workshop on Social software engineering.

[6] Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. 2015. When, how, and why developers (do not) test in their IDEs. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE, Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (Eds.). https://doi.org/ 10.1145/2786805.2786843

[7] Raquel Blanco, Manuel Trinidad, María José Suárez-Cabal, Alejandro Calderón, Mercedes Ruiz, and Javier Tuya. 2023. Can gamification help in software testing education? Findings from an empirical study. Journal of Systems and Software 200 (2023). https://doi.org/10.1016/j.jss.2023.111647

[8] G. Booch, R.A. Maksimchuk, M.W. Engle, J. Conallen, B.J. Young, and K.A. Houston. 1991. Object-oriented Analysis and Design with Applications. https: //books.google.de/books?id=3NQgAQAAIAAJ

[9] David A. Carrington. 1997. Teaching software testing. In Proceedings of the ACM SIGCSE 2nd Australasian Conference on Computer Science Education, ACSE 1997, Melbourne, Victoria, Australia, 1997 (ACM International Conference Proceeding Series, Vol. 2), Harald Søndergaard and A. John Hurst (Eds.). https://doi.org/10. 1145/299359.299369

[10] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing (2009).

[11] Gabriela Martins de Jesus, Fabiano Cutigi Ferrari, Daniel de Paula Porto, and Sandra Camargo Pinto Ferraz Fabbri. 2018. Gamification in Software Testing: A Characterization Study. In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing, SAST. https://doi.org/10.1145/3266003.3266007

[12] Gabriela Martins de Jesus, Leo Natan Paschoal, Fabiano Cutigi Ferrari, and Simone R. S. Souza. 2019. Is It Worth Using Gamification on Software Testing Education?: An Experience Report. In Proceedings of the XVIII Brazilian Symposium on Software Quality, SBQS. https://doi.org/10.1145/3364641.3364661

[13] Ronnie Edson de Souza Santos, Cleyton Vanut Cordeiro de Magalhães, Jorge da Silva Correia-Neto, Fabio Queda Bueno da Silva, Luiz Fernando Capretz, and Rodrigo Souza. 2017. Would You Like to Motivate Software Testers? Ask Them How. In ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM. https://doi.org/10.1109/ESEM.2017.16

[14] Anca Deak, Tor Stålhane, and Guttorm Sindre. 2016. Challenges and strategies for motivating software testing personnel. Inf. Softw. Technol. 73 (2016). https: //doi.org/10.1016/j.infsof.2016.01.002

[15] Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart E. Nacke. 2011. From game design elements to gamefulness: defining "gamification". In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments. https://doi.org/10.1145/2181037.2181040

[16] Sebastian Elbaum, Suzette Person, Jon Dokulil, and Matt Jorde. 2007. Bug hunt: Making early software testing lessons engaging and affordable. In 29th International Conference on Software Engineering (ICSE’07). IEEE.

[17] Martin Fowler. 1999. Refactoring - Improving the Design of Existing Code. http: //martinfowler.com/books/refactoring.html [18] Alberto César Cavalcanti França. 2014. A theory of motivation and satisfaction of software engineers. (2014).

[19] A. César C. França, Helen Sharp, and Fabio Q. B. da Silva. 2014. Motivated software engineers are engaged and focused, while satisfied ones are happy. In 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM. https://doi.org/10.1145/2652524.2652545

[20] César França, Fabio Q. B. da Silva, and Helen Sharp. 2020. Motivation and Satisfaction of Software Engineers. IEEE Transactions on Software Engineering 46, 2 (2020). https://doi.org/10.1109/TSE.2018.2842201

[21] Gordon Fraser, Alessio Gambi, Marvin Kreis, and José Miguel Rojas. 2019. Gamifying a software testing course with code defenders. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education.

[22] Yujian Fu and Peter J Clarke. 2016. Gamification-based cyber-enabled learning environment of software testing. submitted to the 123rd American Society for Engineering Education (ASEE)-Software Engineering Constituent (2016).

[23] Tommaso Fulcini and Luca Ardito. 2022. Gamified Exploratory GUI Testing of Web Applications: a Preliminary Evaluation. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). https://doi. org/10.1109/icstw55395.2022.00045

[24] Tommaso Fulcini, Riccardo Coppola, Luca Ardito, and Marco Torchiano. 2023. A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing. Comput. Surveys (2023). https://doi.org/10.1145/3582273

[25] Vahid Garousi, Austen Rainer, Per Lauvås, and Andrea Arcuri. 2020. Softwaretesting education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570. https://doi.org/10.1016/j.jss.2020.110570

[26] D. Geer. 2005. Eclipse becomes the dominant Java IDE. Computer 38, 7 (2005). https://doi.org/10.1109/MC.2005.228

[27] Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code coverage at Google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 955–963.

[28] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011). https://doi.org/10.1109/TSE.2010.62

[29] Edward L. Jones. 2001. Integrating testing into the curriculum - arsenic in small doses. In Proceedings of the 32rd SIGCSE Technical Symposium on Computer Science Education, SIGCSE. https://doi.org/10.1145/364447.364617

[30] PK Kapur, AK Shrivastava, and Ompal Singh. 2017. When to release and stop testing of a software. Journal of the Indian Society for Probability and Statistics 18, 1 (2017).

[31] Herb Krasner. 2022. THE COST OF POOR SOFTWARE QUALITY IN THE US: A 2022 REPORT. https://www.it-cisq.org/the-cost-of-poor-quality-software-inthe-us-a-2022-report. Accessed: 30.06.2023.

[32] Daniel E Krutz, Samuel A Malachowsky, and Thomas Reichlmayr. 2014. Using a real world project in a software testing course. In Proceedings of the 45th ACM technical symposium on Computer science education. 49–54.

[33] Beáta Lőrincz, Bogdan Iudean, and Andreea Vescan. 2021. Experience report on teaching testing through gamification. In Proceedings of the 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence. https://doi.org/10.1145/3472673.3473960

[34] H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18, 1 (1947). https://doi.org/10.1214/aoms/1177730491

[35] Will Marrero and Amber Settle. 2005. Testing first: emphasizing testing in early programming courses. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE. https: //doi.org/10.1145/1067445.1067451

[36] Gabriel Goncalves Moreira, Sheila Reinehr, Andreia Malucelli, and Frederick Van Amstel. 2022. ProTesters: a board game for teaching the testing process. In Proceedings of the XXI Brazilian Symposium on Software Quality. https://doi.org/ 10.1145/3571473.3571503

[37] Reza Meimandi Parizi. 2016. On the gamification of human-centric traceability tasks in software testing and coding. In 14th IEEE International Conference on Software Engineering Research, Management and Applications, SERA, Yeong-Tae Song (Ed.). https://doi.org/10.1109/SERA.2016.7516146

[38] Goran Petrovic, Marko Ivankovic, Gordon Fraser, and René Just. 2022. Practical Mutation Testing at Scale: A view from Google. IEEE Trans. Software Eng. 48, 10 (2022), 3900–3912. https://doi.org/10.1109/TSE.2021.3107634

[39] Karen Robson, Kirk Plangger, Jan H Kietzmann, Ian McCarthy, and Leyland Pitt. 2015. Is it all a game? Understanding the principles of gamification. Business horizons 58, 4 (2015).

[40] Simon André Scherr, Frank Elberzhager, and Konstantin Holl. 2018. Acceptance testing of mobile applications: automated emotion tracking for large user groups. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems, MOBILESoft@ICSE. https://doi.org/10.1145/3197231.3197259

[41] Frank Philip Seth, Ossi Taipale, and Kari Smolander. 2014. Organizational and customer related challenges of software testing: An empirical study in 11 software companies. In 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS). IEEE.

[42] Leif Singer and Kurt Schneider. 2012. It was a bit of a race: Gamification of version control. In 2012 Second International Workshop on Games and Software Engineering: Realizing User Engagement with Game Engineering Techniques (GAS).

[43] Eliezio Soares, Gustavo Sizílio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. The effects of continuous integration on software development: a systematic literature review. Empir. Softw. Eng. 27, 3 (2022). https://doi.org/10. 1007/s10664-021-10114-1

[44] Andreas Spillner and Tilo Linz. 2019. Basiswissen Softwaretest: Aus-und Weiterbildung zum Certified Tester–Foundation Level nach ISTQB®-Standard.

[45] Klaas-Jan Stol, Mario Schaarschmidt, and Shelly Goldblit. 2022. Gamification in software engineering: the mediating role of developer engagement and job satisfaction. Empir. Softw. Eng. 27, 2 (2022). https://doi.org/10.1007/s10664-021- 10062-w

[46] Philipp Straubinger, Laura Caspari, and Gordon Fraser. 2023. Code Critters: A Block-Based Testing Game. In IEEE International Conference on Software Testing, Verification and Validation, ICST - Workshops. https://doi.org/10.1109/ ICSTW58534.2023.00077

[47] Philipp Straubinger and Gordon Fraser. 2022. Gamekins: Gamifying Software Testing in Jenkins. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). New York, NY, USA. https://doi.org/10.1145/3510454.3516862

[48] Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001). Citeseer.

[49] Pradeep Kashinath Waychal and Luiz Fernando Capretz. 2016. Why a Testing Career Is Not the First Choice of Engineers. (2016). http://arxiv.org/abs/1612. 00734

[50] Elaine J. Weyuker, Thomas J. Ostrand, JoAnne Brophy, and Rathna Prasad. 2000. Clearing a Career Path for Software Testers. IEEE Software 17, 2 (2000). https: //doi.org/10.1109/52.841696

[51] Zornitsa Yordanova. 2019. Educational Innovations and Gamification for Fostering Training and Testing in Software Implementation Projects. In Software Business - 10th International Conference, ICSOB (Lecture Notes in Business Information Processing, Vol. 370). https://doi.org/10.1007/978-3-030-33742-1_23

[52] Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald C. Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 (2020). https://doi.org/10.1007/s10664-019-09785-8

[53] Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29, 4 (1997). https://doi.org/10. 1145/267580.267590