publications

2024

  1. On the Opportunities of Large Language Models for Programming Process Data
    John Edwards, Arto Hellas, and Juho Leinonen
    arXiv preprint arXiv:2411.00414 2024
    PDF

  2. Post Primary Teachers’ Perspectives on Machine Learning and Artificial Intelligence in the Leaving Certificate Computer Science Curriculum
    Joyce Mahon, Brett A. Becker, Brian Mac Namee, and Juho Leinonen
    In Proceedings of the 24th Koli Calling International Conference on Computing Education Research 2024
    PDF

  3. Integrating Natural Language Prompting Tasks in Introductory Programming Courses
    Chris Kerslake, Paul Denny, David H Smith IV, James Prather, Juho Leinonen, Andrew Luxton-Reilly, and Stephen MacNeil
    arXiv preprint arXiv:2410.03063 2024
    PDF

  4. Koli Calling 2024: Call for Participation
    Juho Leinonen
    SIGCSE Bulletin 2024
    PDF

  5. Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students
    Stephen MacNeil, Magdalena Rogalska, Juho Leinonen, Paul Denny, Arto Hellas, and Xandria Crosland
    arXiv preprint arXiv:2410.09193 2024
    PDF

  6. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
    James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett Becker, Bailey Kimmel, Jared Wright, and Ben Briggs
    In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 2024
    PDF

  7. Evaluating Contextually Personalized Programming Exercises Created with Generative AI
    Evanfiya Logacheva, Arto Hellas, James Prather, Sami Sarsa, and Juho Leinonen
    In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 2024
    PDF

  8. Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A-Judge
    Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 2024
    PDF

  9. "Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Models
    Seth Bernstein, Paul Denny, Juho Leinonen, Lauren Kan, Arto Hellas, Matt Littlefield Sami Sarsa, and Stephen MacNeil
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 2024
    PDF

  10. Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills
    Paul Denny, David H Smith IV, Max Fowler, James Prather, Brett A Becker, and Juho Leinonen
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 2024
    PDF

  11. Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems
    Lauren E. Margulieux, James Prather, Brent N. Reeves, Brett A. Becker, Gozde Cetin Uzun, Dastyni Loksa, Juho Leinonen, and Paul Denny
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 2024
    PDF

  12. Evaluating Language Models for Generating and Judging Programming Feedback
    Charles Koutcheme, Nicola Dainese, Arto Hellas, Sami Sarsa, Juho Leinonen, Syed Ashraf, and Paul Denny
    arXiv preprint arXiv:2407.04873 2024
    PDF

  13. How Instructors Incorporate Generative AI into Teaching Computing
    James Prather, Juho Leinonen, Natalie Kiesler, Jamie Gorson Benario, Sam Lau, Stephen MacNeil, Narges Norouzi, Simone Opel, Virginia Pettit, Leo Porter, Brent N. Reeves, Jaromir Savelka, David H. Smith, Sven Strickroth, and Daniel Zingaro
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 2 2024
    PDF

  14. Analyzing Students’ Preferences for LLM-Generated Analogies
    Seth Bernstein, Paul Denny, Juho Leinonen, Matt Littlefield, Arto Hellas, and Stephen MacNeil
    In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 2 2024
    PDF

  15. Prompts First, Finally
    Brent N. Reeves, James Prather, Paul Denny, Juho Leinonen, Stephen MacNeil, Brett A. Becker, and Andrew Luxton-Reilly
    arXiv preprint arXiv:2407.04873 2024
    PDF

  16. Experiences from Integrating Large Language Model Chatbots into the Classroom
    Arto Hellas, Juho Leinonen, and Leo Leppänen
    arXiv preprint arXiv:2406.04817 2024
    PDF

  17. On the comprehensibility of functional decomposition: An empirical study
    Ewan Tempero, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, Diana Kirk, Juho Leinonen, Asma Shakil, Robert Sheehan, James Tizard, Yu-Cheng Tu, and others
    In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension 2024
    PDF

  18. Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models
    Qiming Bao, Juho Leinonen, Alex Yuxuan Peng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Alice Huang, Paul Denny, Michael Witbrock, and Jiamou Liu
    In ICLR 2024 Workshop: How Far Are We From AGI 2024
    PDF

  19. "Sometimes You Just Gotta Risk It for the Biscuit": A Portrait of Student Risk-Taking
    Juho Leinonen, and Paul Denny
    arXiv preprint arXiv:2405.01477 2024
    PDF

  20. SIGCSE Bulletin Vol. 56, No. 2
    Erik Barendsen, Lina Battestilli, Paul Denny, Mohsen Dorodchi, Heidi JC Ellis, Margaret Hamilton, Gregory W Hislop, Juho Leinonen, Violetta Lonati, Mattia Monga, and others
    SIGCSE Bulletin 2024
    PDF

  21. Koli Calling 2024: Call for Submissions
    Andreas Mühling, and Juho Leinonen
    ACM SIGCSE Bulletin 2024
    HTML

  22. Prompt Problems: A New Programming Exercise for the Generative AI Era
    Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 2024
    PDF

  23. Detecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning Models
    Muntasir Hoq, Yang Shi, Juho Leinonen, Damilola Babalola, Collin Lynch, Thomas Price, and Bita Akram
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 2024
    PDF

  24. Instructor Perceptions of AI Code Generation Tools – A Multi-Institutional Interview Study
    Judy Sheard, Paul Denny, Arto Hellas, Juho Leinonen, Lauri Malmi, and Simon
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 2024
    PDF

  25. Solving Proof Block Problems Using Large Language Models
    Seth Poulsen, Sami Sarsa, James Prather, Juho Leinonen, Brett A. Becker, Arto Hellas, Paul Denny, and Brent N. Reeves
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 2024
    PDF

  26. Using Large Language Models for Teaching Computing
    Juho Leinonen, Stephen MacNeil, Paul Denny, and Arto Hellas
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2 2024
    HTML

  27. Discussing the Changing Landscape of Generative AI in Computing Education
    Stephen MacNeil, Juho Leinonen, Paul Denny, Natalie Kiesler, Arto Hellas, James Prather, Brett A Becker, Michel Wermelinger, and Karen Reid
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2 2024
    HTML

  28. AI in Computing Education from Research to Practice
    Bita Akram, Juho Leinonen, Narges Norouzi, James Prather, and Lisa Zhang
    In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2 2024
    PDF

  29. Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
    Stephen MacNeil, Paul Denny, Andrew Tran, Juho Leinonen, Seth Bernstein, Arto Hellas, Sami Sarsa, and Joanne Kim
    In Proceedings of the 26th Australasian Computing Education Conference 2024
    PDF

  30. :trophy:
    Evaluating LLM-generated Worked Examples in an Introductory Programming Course
    Breanna Jury, Angela Lorusso, Juho Leinonen, Paul Denny, and Andrew Luxton-Reilly
    In Proceedings of the 26th Australasian Computing Education Conference 2024
    Best paper award
    PDF

  31. Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Models
    James Prather, Paul Denny, Juho Leinonen, David H Smith IV, Brent N Reeves, Stephen MacNeil, Brett A Becker, Andrew Luxton-Reilly, Thezyrie Amarouche, and Bailey Kimmel
    arXiv preprint arXiv:2401.10759 2024
    PDF

  32. Computing Education in the Era of Generative AI
    Paul Denny, James Prather, Brett A Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N Reeves, Eddie Antonio Santos, and Sami Sarsa
    Communications of the ACM 2024
    PDF

2023

  1. Benchmarking Educational Program Repair
    Charles Koutcheme, Nicola Dainese, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny
    In NeurIPS’23 Workshop on Generative AI for Education (GAIED) 2023
    PDF

  2. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
    James Prather, Paul Denny, Juho Leinonen, Brett A Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N Reeves, and Jaromir Savelka
    2023
    PDF

  3. Generative AI in Introductory Programming
    Brett A. Becker, Michelle Craig, Paul Denny, Hieke Keuning, Natalie Kiesler, Juho Leinonen, Andrew Luxton-Reilly, James Prather, and Keith Quille
    2023
    PDF

  4. Leveraging Large Language Models for Analysis of Student Course Feedback
    Zixuan Wang, Paul Denny, Juho Leinonen, and Andrew Luxton-Reilly
    In Proceedings of the 16th Annual ACM India Compute Conference 2023
    PDF

  5. Understanding Student Evaluation of Teaching in Computer Science Courses
    Zixuan Wang, Paul Denny, Juho Leinonen, and Andrew Luxton-Reilly
    In Proceedings of the 16th Annual ACM India Compute Conference 2023
    PDF

  6. The Forum Factor: Exploring the Link between Online Discourse and Student Achievement in Higher Education
    Darryn He, Hannah Steffes, Samin Yasar, Paul Denny, Andrew Luxton-Reilly, and Juho Leinonen
    In Proceedings of the 16th Annual ACM India Compute Conference 2023
    PDF

  7. Could ChatGPT Be Used for Reviewing Learnersourced Exercises?
    Nea Pirttinen, and Juho Leinonen
    In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research 2023
    PDF

  8. :trophy:
    Exploring the Interplay of Achievement Goals, Self-Efficacy, Prior Experience and Course Achievement
    Hannu Pesonen, Juho Leinonen, Lassi Haaranen, and Arto Hellas
    In The United Kingdom and Ireland Computing Education Research (UKICER) conference (UKICER 2023) 2023
    Best paper award
    PDF

  9. "It’s Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
    James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos
    ACM Transactions on Computer-Human Interaction 2023
    PDF

  10. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
    Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva
    In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 2023
    PDF

  11. Evaluating Distance Measures for Program Repair
    Charles Koutcheme, Sami Sarsa, Juho Leinonen, Lassi Haaranen, and Arto Hellas
    In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 2023
    PDF

  12. Comparing Code Explanations Created by Students and Large Language Models
    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas
    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 2023
    PDF

  13. Transformed by Transformers: Navigating the AI Coding Revolution for Computing Education: An ITiCSE Working Group Conducted by Humans
    James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michael E Caspersen, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka
    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 2023
    PDF

  14. Chat Overflow: Artificially Intelligent Models for Computing Education-renAIssance or apocAIypse?
    Paul Denny, Brett A. Becker, Juho Leinonen, and James Prather
    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 2023
    PDF

  15. Seeing Program Output Improves Novice Learning Gains
    Juho Leinonen, Arto Hellas, and John Edwards
    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 2023
    PDF

  16. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations
    Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen
    In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 2023
    PDF

  17. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators
    Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves
    arXiv preprint arXiv:2307.16364 2023
    PDF

  18. Detecting ChatGPT-Generated Code in a CS1 Course
    Muntasir Hoq, Yang Shi, Juho Leinonen, Damilola Babalola, Collin Lynch, and Bita Akram
    In AIED2023 Empowering Education with LLMs - the Next-Gen Interface and Content Generation Workshop 2023
    PDF

  19. Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources
    Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa
    arXiv preprint arXiv:2306.10509 2023
    PDF

  20. Automated Program Repair Using Generative Models for Code Infilling
    Charles Koutcheme, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny
    In International Conference on Artificial Intelligence in Education 2023
    PDF

  21. Factors Affecting Compilable State at Each Keystroke in CS1
    Steven Scott, Arto Hellas, Juho Leinonen, and John Edwards
    In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) 2023
    PDF

  22. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 2023
    PDF

  23. G is for Generalisation: Predicting Student Success from Keystrokes
    Zac Pullar-Strecker, Filipe Dwan Pereira, Paul Denny, Andrew Luxton-Reilly, and Juho Leinonen
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 2023
    PDF

  24. Using Large Language Models to Enhance Programming Error Messages
    Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 2023
    PDF

  25. Automatically Generating CS Learning Materials with Large Language Models
    Stephen MacNeil, Andrew Tran, Juho Leinonen, Paul Denny, Joanne Kim, Arto Hellas, Seth Bernstein, and Sami Sarsa
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 2023
    PDF

  26. Computing Education Postdocs and Beyond: Building a Postdoc Space for Community and Collaboration
    Francisco Enrique Vicente Castro, Joseph Wilson, Jessica Vandenberg, Juho Leinonen, and Miranda C. Parker
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 2023
    HTML

  27. The Implications of Large Language Models for CS Teachers and Students
    Stephen MacNeil, Joanne Kim, Juho Leinonen, Paul Denny, Seth Bernstein, Brett A. Becker, Michel Wermelinger, Arto Hellas, Andrew Tran, Sami Sarsa, James Prather, and Viraj Kumar
    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 2023
    PDF

  28. Lessons Learned From Four Computing Education Crowdsourcing Systems
    Nea Pirttinen, Paul Denny, Arto Hellas, and Juho Leinonen
    IEEE Access 2023
    PDF

  29. Introduction to the Special Issue on EDM in Computer Science Education (CSEDM)
    Thomas W Price, Sharon Hsiao, Bita Akram, Peter Brusilovsky, and Juho Leinonen
    Journal of Educational Data Mining 2023
    PDF

  30. Experiences from Learnersourcing SQL Exercises: Do They Cover Course Topics and Do Students Use Them?
    Nea Pirttinen, Arto Hellas, and Juho Leinonen
    In Proceedings of the 25th Australasian Computing Education Conference 2023
    PDF

  31. :trophy:
    Automated Questionnaires About Students’ JavaScript Programs: Towards Gauging Novice Programming Processes
    Teemu Lehtinen, Lassi Haaranen, and Juho Leinonen
    In Proceedings of the 25th Australasian Computing Education Conference 2023
    Best practitioner paper award
    PDF

2022

  1. Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs
    Barbara J Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S Miller, Briana B Morrison, Janice L Pearce, and Susan H Rodger
    Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education 2022
    PDF

  2. Robosourcing Educational Resources–Leveraging Large Language Models for Learnersourcing
    Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen
    arXiv preprint arXiv:2211.04715 2022
    PDF

  3. Trends From Computing Education Research Conferences: Increasing Submissions and Decreasing Acceptance Rates
    Juho Leinonen, Nea Pirttinen, and Arto Hellas
    In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research 2022
    PDF

  4. Finding Significant p in Coffee or Tea: Mildly Distasteful
    Sami Sarsa, Arto Hellas, and Juho Leinonen
    In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research 2022
    PDF

  5. Experiences With and Lessons Learned on Deadlines and Submission Behavior
    Francisco Enrique Vicente Castro, Juho Leinonen, and Arto Hellas
    In Proceedings of the 22nd Koli Calling International Conference on Computing Education Research 2022
    PDF

  6. Empirical Evaluation of Deep Learning Models for Knowledge Tracing: Of Hyperparameters and Metrics on Performance and Replicability
    Sami Sarsa, Juho Leinonen, Arto Hellas, and others
    Journal of Educational Data Mining 2022
    PDF

  7. Piloting Natural Language Generation for Personalized Progress Feedback
    Leo Leppänen, Arto Hellas, and Juho Leinonen
    In 2022 IEEE Frontiers in Education Conference (FIE) 2022
    PDF

  8. Speeding Up Automated Assessment of Programming Exercises
    Sami Sarsa, Juho Leinonen, Charles Koutcheme, and Arto Hellas
    In Proceedings of the 2022 Conference on United Kingdom & Ireland Computing Education Research 2022
    PDF

  9. :trophy:
    Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen
    In Proceedings of the 2022 ACM Conference on International Computing Education Research V. 1 2022
    Best paper award
    PDF

  10. Planning a Multi-institutional and Multi-national Study of the Effectiveness of Parsons Problems
    Barbara J Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S Miller, Briana Morrison, Janice L Pearce, and Susan H Rodger
    In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2 2022
    PDF

  11. Can Students Review Their Peers? Comparison of Peer and Instructor Reviews
    Nea Pirttinen, and Juho Leinonen
    In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 2022
    PDF

  12. Who Continues in a Series of Lifelong Learning Courses?
    Sami Sarsa, Arto Hellas, and Juho Leinonen
    In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 2022
    PDF

  13. Open IDE Action Log Dataset from a CS1 MOOC
    Juho Leinonen
    In 6th Educational Data Mining in Computer Science Education (CSEDM) Workshop 2022
    PDF

  14. Digital Education for All: Multi-University Study of Increasing Competent Student Admissions at Scale
    Nea Pirttinen, Juho Leinonen, Annemari Auvinen, Vesa Lappalainen, Katja Tynkkynen, Henrik Hedberg, Mikko-Jussi Laakso, and Kjell Lemström
    In L@S’22: Proceedings of the Ninth ACM Conference on Learning @ Scale 2022
    PDF

  15. Coverage of Course Topics in Learnersourced SQL Exercises
    Nea Pirttinen, Arto Hellas, and Juho Leinonen
    In Learnersourcing: Student-Generated Content @ Scale: Workshop as Part of the Learning @ Scale Conference 2022
    PDF

  16. Time-on-Task Metrics for Predicting Performance
    Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas
    ACM Inroads 2022
    PDF

  17. Seeking Flow From Fine-Grained Log Data
    Benjamin Ultan Cowley, Arto Hellas, Petri Ihantola, Juho Leinonen, and Michiel Spape
    In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) 2022
    PDF

  18. Pausing While Programming: Insights From Keystroke Analysis
    Raj Shrestha, Juho Leinonen, Albina Zavgorodniaia, Arto Hellas, and John Edwards
    In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) 2022
    PDF

  19. Digitalisaation vaikutus oppimiseen, oppimistilanteisiin ja oppimistuloksiin: DigiVOO-hankkeen väliraportti 2022
    Mari-Pauliina Vainikainen, Sanna Oinas, Satu Koivuhovi, Kukka-Maaria Polso, Juho Leinonen, Faruk Nazeri, Laura Nyman, Cristiana Mergianian, Natalija Gustavson, Esko Lindgren, Mikko Asikainen, Petri Ihantola, and Risto Hotulainen
    2022
    PDF

  20. :trophy:
    Time-on-Task Metrics for Predicting Performance
    Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas
    In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 2022
    Best paper award
    PDF

  21. A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
    Juho Leinonen, Paul Denny, and Jacqueline Whalley
    In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 2022
    PDF

  22. CodeProcess Charts: Visualizing the Process of Writing Code
    Raj Shrestha, Juho Leinonen, Arto Hellas, Petri Ihantola, and John Edwards
    In Australasian Computing Education Conference 2022
    PDF

  23. Methodological Considerations for Predicting At-risk Students
    Charles Koutcheme, Sami Sarsa, Arto Hellas, Lassi Haaranen, and Juho Leinonen
    In Australasian Computing Education Conference 2022
    PDF

2021

  1. Facilitating API lookup for novices learning data wrangling using thumbnail graphics
    Lovisa Sundin, Nourhan Sakr, Juho Leinonen, and Quintin Cutts
    Foundations of Data Science 2021
    HTML

  2. Visual Recipes for Slicing and Dicing Data: Teaching Data Wrangling Using Subgoal Graphics
    Lovisa Sundin, Nourhan Sakr, Juho Leinonen, Sherif Aly, and Quintin Cutts
    In 21st Koli Calling International Conference on Computing Education Research 2021
    PDF

  3. Persistence of Time Management Behavior of Students and Its Relationship with Performance in Software Projects
    Joonas Häkkinen, Petri Ihantola, Matti Luukkainen, Antti Leinonen, and Juho Leinonen
    In Proceedings of the 17th ACM Conference on International Computing Education Research 2021
    PDF

  4. Digital Education For All: Better Students Through Open Doors?
    Nea Pirttinen, Juho Leinonen, and Kjell Lemström
    In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 2021
    PDF

  5. Does the Early Bird Catch the Worm? Earliness of Students’ Work and its Relationship with Course Outcomes
    Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas
    In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 2021
    PDF

  6. Fine-Grained Versus Coarse-Grained Data for Estimating Time-on-Task in Learning Programming
    Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas
    In Educational Data Mining 2021 2021
    PDF

  7. Morning or Evening? An Examination of Circadian Rhythms of CS1 Students
    Albina Zavgorodniaia, Raj Shrestha, Juho Leinonen, Arto Hellas, and John Edwards
    In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) 2021
    PDF

  8. Exploring Personalization of Gamification in an Introductory Programming Course
    Mitchell Rogers, Wendy Yao, Andrew Luxton-Reilly, Juho Leinonen, Danielle Lottridge, and Paul Denny
    In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education 2021
    PDF

  9. Exploring the Complexity of Crowdsourced Programming Assignments
    Nea Pirttinen, and Juho Leinonen
    In CS Education Infrastructure for All III: From Ideas to Practice: Seventh SPLICE Project Workshop 2021
    PDF

  10. Analyzing Fine-Grained Material Usage Behavior
    Charles Koutcheme, Juho Leinonen, Juha Sorva, and Arto Hellas
    In CS Education Infrastructure for All III: From Ideas to Practice: Seventh SPLICE Project Workshop 2021
    PDF

  11. :trophy:
    Exploring the Effects of Contextualized Problem Descriptions on Problem Solving
    Juho Leinonen, Paul Denny, and Jacqueline Whalley
    In Australasian Computing Education Conference 2021
    Best paper award
    PDF

  12. Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
    Paul Denny, Jacqueline Whalley, and Juho Leinonen
    In Australasian Computing Education Conference 2021
    PDF

2020

  1. Choosing Code Segments to Exclude From Code Similarity Detection
    Simon, Oscar Karnalim, Judy Sheard, Ilir Dema, Amey Karkare, Juho Leinonen, Michael Liut, and Renée McCauley
    2020
    PDF

  2. Students’ Preferences Between Traditional and Video Lectures: Profiles and Study Success
    Petri Ihantola, Juho Leinonen, and Matti Rintala
    In Koli Calling’20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research 2020
    PDF

  3. Programming Versus Natural Language: On the Effect of Context on Typing in CS1
    John Edwards, Juho Leinonen, Chetan Birthare, Albina Zavgorodniaia, and Arto Hellas
    In Proceedings of the 2020 ACM Conference on International Computing Education Research 2020
    PDF

  4. Selection of Code Segments for Exclusion from Code Similarity Detection
    Simon, Oscar Karnalim, Judy Sheard, Ilir Dema, Amey Karkare, Juho Leinonen, Michael Liut, and Renee McCauley
    In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education 2020
    PDF

  5. Crowdsourcing Content Creation for SQL Practice
    Juho Leinonen, Nea Pirttinen, and Arto Hellas
    In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education 2020
    PDF

  6. A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes
    John Edwards, Juho Leinonen, and Arto Hellas
    In Proceedings of the 51st ACM Technical Symposium on Computer Science Education 2020
    PDF

2019

  1. Pass Rates in Introductory Programming and in Other STEM Disciplines
    Simon, Andrew Luxton-Reilly, Vangel V Ajanovski, Eric Fouh, Christabel Gonsalvez, Juho Leinonen, Jack Parkinson, Matthew Poole, and Neena Thota
    2019
    PDF

  2. Keystroke Data in Programming Courses
    Juho Leinonen
    2019
    PDF

  3. Experimenting With Model Solutions as a Support Mechanism
    Henrik Nygren, Juho Leinonen, Nea Pirttinen, Antti Leinonen, and Arto Hellas
    In Proceedings of the 1st UK & Ireland Computing Education Research Conference 2019
    PDF

  4. Integrating CrowdSorcerer: Lessons Learned
    Nea Pirttinen, and Juho Leinonen
    In Proceedings of SPLICE 2019 Workshop Computing Science Education Infrastructure From Tools to Data at 15th ACM International Computing Education Research Conference 2019
    PDF

  5. Non-restricted Access to Model Solutions: A Good Idea?
    Henrik Nygren, Juho Leinonen, and Arto Hellas
    In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education 2019
    PDF

  6. Pass Rates in STEM Disciplines Including Computing
    Simon, Andrew Luxton-Reilly, Vangel Ajanovski, Eric Fouh, Chris Gonsalvez, Juho Leinonen, Jack Parkinson, Matthew Poole, and Neena Thota
    In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education 2019
    PDF

  7. Admitting Students Through an Open Online Course in Programming: A Multi-Year Analysis of Study Success
    Juho Leinonen, Petri Ihantola, Antti Leinonen, Henrik Nygren, Jaakko Kurhila, Matti Luukkainen, and Arto Hellas
    In Proceedings of the 2019 ACM Conference on International Computing Education Research 2019
    PDF

  8. Does Creating Programming Assignments With Tests Lead to Improved Performance in Writing Unit Tests?
    Vilma Kangas, Nea Pirttinen, Henrik Nygren, Juho Leinonen, and Arto Hellas
    In Proceedings of the ACM Conference on Global Computing Education 2019
    PDF

  9. Exploring the Applicability of Simple Syntax Writing Practice for Learning Programming
    Antti Leinonen, Henrik Nygren, Nea Pirttinen, Arto Hellas, and Juho Leinonen
    In Proceedings of the 50th ACM Technical Symposium on Computer Science Education 2019
    PDF

2018

  1. Analysis of Students’ Peer Reviews to Crowdsourced Programming Assignments
    Nea Pirttinen, Vilma Kangas, Henrik Nygren, Juho Leinonen, and Arto Hellas
    In Proceedings of the 18th Koli Calling International Conference on Computing Education Research 2018
    PDF

  2. Taxonomizing Features and Methods for Identifying At-Risk Students in Computing Courses
    Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao
    In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education 2018
    PDF

  3. Crowdsourcing Programming Assignments With CrowdSorcerer
    Nea Pirttinen, Vilma Kangas, Irene Nikkarinen, Henrik Nygren, Juho Leinonen, and Arto Hellas
    In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education 2018
    PDF

  4. Predicting Academic Performance: A Systematic Literature Review
    Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao
    In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education 2018
    PDF

  5. A Study of Pair Programming Enjoyment and Attendance Using Study Motivation and Strategy Metrics
    Onni Aarne, Petrus Peltola, Juho Leinonen, and Arto Hellas
    In Proceedings of the 49th ACM Technical Symposium on Computer Science Education 2018
    PDF

  6. Supporting Self-Regulated Learning With Visualizations in Online Learning Environments
    Kalle Ilves, Juho Leinonen, and Arto Hellas
    In Proceedings of the 49th ACM Technical Symposium on Computer Science Education 2018
    PDF

2017

  1. Tracking Students’ Internet Browsing in a Machine Exam
    Henrik Nygren, Juho Leinonen, and Arto Hellas
    In Proceedings of the 6th Computer Science Education Research Conference 2017
    PDF

  2. Adolescent and Adult Student Attitudes Towards Progress Visualizations
    Onni Aarne, Petrus Peltola, Antti Leinonen, Juho Leinonen, and Arto Hellas
    In International Conference on Informatics in Schools: Situation, Evolution, and Perspectives 2017
    PDF

  3. :trophy:
    Thought Crimes and Profanities Whilst Programming
    Juho Leinonen, and Arto Hellas
    In Proceedings of the 17th Koli Calling International Conference on Computing Education Research 2017
    Best presentation award
    PDF

  4. Identification Based on Typing Patterns Between Programming and Free Text
    Petrus Peltola, Vilma Kangas, Nea Pirttinen, Henrik Nygren, and Juho Leinonen
    In Proceedings of the 17th Koli Calling International Conference on Computing Education Research 2017
    PDF

  5. Predicting Academic Success Based on Learning Material Usage
    Leo Leppänen, Juho Leinonen, Petri Ihantola, and Arto Hellas
    In Proceedings of the 18th Annual Conference on Information Technology Education 2017
    PDF

  6. Comparison of Time Metrics in Programming
    Juho Leinonen, Leo Leppänen, Petri Ihantola, and Arto Hellas
    In Proceedings of the 2017 ACM Conference on International Computing Education Research 2017
    PDF

  7. Student Modeling Based on Fine-Grained Programming Process Snapshots
    Juho Leinonen
    In Proceedings of the 2017 ACM Conference on International Computing Education Research 2017
    PDF

  8. Plagiarism in Take-Home Exams: Help-Seeking, Collaboration, and Systematic Cheating
    Arto Hellas, Juho Leinonen, and Petri Ihantola
    In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education 2017
    PDF

  9. Using and Collecting Fine-Grained Usage Data to Improve Online Learning Materials
    Leo Leppanen, Juho Leinonen, Petri Ihantola, and Arto Hellas
    In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering Education and Training Track (ICSE-SEET) 2017
    PDF

  10. Preventing Keystroke Based Identification in Open Data Sets
    Juho Leinonen, Petri Ihantola, and Arto Hellas
    In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale 2017
    PDF

  11. Privacy versus Information in Keystroke Latency Data
    Juho Leinonen
    2017
    PDF

  12. :trophy:
    Performance and Consistency in Learning to Program
    Alireza Ahadi, Raymond Lister, Shahil Lal, Juho Leinonen, and Arto Hellas
    In Proceedings of the Nineteenth Australasian Computing Education Conference 2017
    Best student paper award
    PDF

2016

  1. Pauses and Spacing in Learning to Program
    Leo Leppänen, Juho Leinonen, and Arto Hellas
    In Proceedings of the 16th Koli Calling International Conference on Computing Education Research 2016
    PDF

  2. Typing Patterns and Authentication in Practical Programming Exams
    Juho Leinonen, Krista Longi, Arto Klami, Alireza Ahadi, and Arto Vihavainen
    In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education 2016
    PDF

  3. Short Pauses While Studying Considered Harmful
    Leo Leppänen, Juho Leinonen, and Arto Vihavainen
    In EDULEARN 2016 2016
    PDF

  4. Automatic Inference of Programming Performance and Experience From Typing Patterns
    Juho Leinonen, Krista Longi, Arto Klami, and Arto Vihavainen
    In Proceedings of the 47th ACM Technical Symposium on Computing Science Education 2016
    PDF

2015

  1. :trophy:
    Identification of Programmers From Typing Patterns
    Krista Longi, Juho Leinonen, Henrik Nygren, Joni Salmi, Arto Klami, and Arto Vihavainen
    In Proceedings of the 15th Koli Calling Conference on Computing Education Research 2015
    Best paper award
    PDF