The purpose of this workshop is to initiate the development of best practices for formative and summative assessments using Refute Problems (Kumar, 2021; Agarwal et al., 2023). This workshop primarily targets faculty who teach introductory courses in programming, algorithms, and software engineering, but it is open to all faculty who see value in Refute Problems.
A Refute Problem presents students with an incorrect but plausible solution to a given task. To solve such a problem, students must demonstrate a mismatch between the given solution and the task. In the context of programming courses, students are given buggy code for some tasks, and they must identify an input on which the code’s behaviour does not match the desired behaviour (as per the task).
As Gen AI tools mature, they are being integrated into professional workflows e.g., software development. Software developers often spend more time on comprehending and critiquing code written by others rather than on writing code themselves (Tiarks, 2011). Therefore, it is unsurprising that experienced developers can rapidly critique AI-generated suggestions (Ziegler et al., 2022). For students, however, this form of “vibe coding” bypasses a crucial aspect of the learning process: the thoughtful consideration of both the problem and the generated solution. Agarwal et al. (2023) note that Refute Problems may help students develop the ability to critique AI-generated code. The importance of developing this ability has been highlighted by both faculty and software developers (Lau and Guo, 2023; Prather et al., 2025). The ACM/IEEE-CS/AAAI Task Force (2024) suggests in its CS2023 curriculum that “reading programs will see a sharpened focus and the idea of learning to program by writing, and reliance on learning syntax, will change towards learning to program by prompting, comprehending, verifying, editing, modifying, adapting, and testing code”.
Refute Problems can potentially be adapted to other disciplines where it is important for students to practice critiquing AI-generated artefacts. Thus, while we will largely focus on Refute Problems in the context of programming, we welcome faculty who teach other courses (including non-computing courses) where the ability of students to critique AI-generated artefacts is important.
This discussion-based workshop will be conducted in two parts.
If you are interested in participating, please register for ICCE and fill out the following form until 15 September:
Register Now