Implementing AI-Based Code Review Automation: A Case Study in Academic Software Development

Abstract

Code reviews have always been a necessary evil in software development but one that was very time consuming. Nevertheless, it is much tougher for educators in education due to evaluating codes for hundreds of students. This paper describes the design and deployment of an AI-assisted code review system in a university computer science department. The system uses the transformer and GPT-4 models for aiding the human graders in assessing programming assignments of various university courses. The system checks the code for syntax and semantic errors, design and implementation patterns, and compliance with coding standards.  In a six-month study run of involving 150 student projects, the automated system detected around 87% of code quality issues needing human inspection. It cut down the time taken by educators to review code by approximately 62%, with the same grade of feedback quality. Based on our experience, AI for code review has exciting potentials but also currently has limitations in an academic environment. It is good at spotting common mistakes as well as style issues, but not so good at picking up on context-sensitive design decisions or pedagogical issues. The study provides evidence of AI-supported code review for the academic field and suggestions for educators wishing to set up a similar system.