Innovations in Education, CDS Faculty Deploys AI Tutor
As AI becomes increasingly integrated into higher education, educators continue to face challenges with large language models (LLMs) such as ChatGPT (and other artificial intelligence chatbots), which offer instant solutions to students' homework questions and programming problems.
Nearly two years after ChatGPT's debut, the question persists: how can faculty ensure rich learning when students have access to tools that might bypass the learning process? Kevin Gold, Associate Professor of the Practice in the Faculty of Computing and Data Sciences at Boston University, delved into this question.
During the spring 2024, Gold enlisted 127 students from his DS110 “Introduction to Programming in Python and Data Science Concepts” course to test an LLM-based homework tutor. Building on the Socratic teaching method—a learning approach that promotes student engagement through guided questioning rather than direct answers—Gold developed a GPT-4 powered bot as the core of the AI tool. The bot was given the homework assignments along with solutions, and was instructed not to give answers directly. Gold appropriately named the bot the DS110 Socratic AI Tutor.
“The goal is to bring students into the interaction, work for the answers using the Socratic method, and not just give away the answer immediately,” he said.
Gold emphasized the importance of crafting an experience that students would choose over the standard ChatGPT. The new system offered three key advantages: First, it utilized the more advanced GPT-4, which was not available for free to students. Second, students were reassured by the fact that the system was provided with solutions, which alleviated concerns about unreliable or "hallucinated" responses. Third, the course-approved interaction was perceived as more legitimate and conducive to learning, unlike the direct use of ChatGPT, which many students felt was less appropriate for their educational needs.
“The DS110 AI tutor would not give away the answers but instead guided us through the questions to enable us to find the answers on our own,” said Amanda Atlas (CDS’27), one of Gold's students. “The tutor assisted with questions I needed help understanding, such as how to code and how to approach a problem.”
Throughout the semester, Gold recorded interactions and anonymously logged and coded seventeen attributes, including helpfulness and the extent of answer leakage. Surveys were also deployed to better understand students’ perceptions and use cases.
“The tutor was generally perceived as helpful to learning by students and an independent coder, and only rarely directly leaked the solution,” Gold said.
The average student rating of helpfulness was 4.0/5, comparable to the helpfulness of 4.2/5 for teaching assistant office hours. Some AI tutor tasks that the majority of students rated “Very Helpful” included clarifying what questions are asking, answering questions about Python syntax, and describing general programming approaches. Students rated as “Moderately Helpful” its ability to help debug or provide programming examples.
Gold and Educational Technologist Shuang Geng, an expert in learning analytics with BU Digital Learning & Innovation, conducted analysis and co-authored a paper on the LLM-based homework tutor. The approach received the "Best-in-Session" award at the 5th International Conference on Artificial Intelligence in Education Technology held in Spain last summer.
“I think there was a general feeling at the conference that this was addressing an important problem,” Gold said. “Students need programming homework in some form to get the practice they need to achieve mastery, and large language models are short-circuiting that. The idea that we could turn AI assistance from a bug in the process to a feature is very attractive.”
This fall, Gold and Geng are planning a fall randomized controlled trial to test if access to this tutor improves student performance on tests after access ends.
“Right now, it's difficult to figure out whether students who seek the tutor's help are different from other students in some way, before they ever use the tutor,” Gold said. “So we're essentially trying to be more rigorous in really examining the effects of this system.”
- By Maureen McCarthy