GitHub Copilot Vs Human Programmers: Are AI Tools of Any Help?

Around 60-75% of users surveyed reported satisfaction with using Copilot, feeling less frustrated while using Copilot

Generative AI models including Copilot have been at the center of debate over their efficiency and, ie., if they will put content generators and coders at peril. Cutting through the din of the noise, GitHub has attempted to quantify Copilot's influence in improving a programmer's productivity and happiness. Copilot, a pair programmer has predictive code functionalities to assist programmers with appropriate code suggestions similar to IntelliSense/IntelliCode of Microsoft, though it digs deeper beyond those offerings, thanks to Codex to let turn typed commands into actual code. "Because AI-assisted development is a relatively new field, as researchers we have little prior research to draw upon," said GitHub's Eirini Kalliamvakou in a blog post.

After early observations and interviews with users, they surveyed more than 2,000 developers to assess their experience with programming using Copilot in three dimensions: holistic measurement of productivity, the perspective of the developer, and assessing the co-pilot's effect in everyday programming scenarios. They feel it is more important to take a holistic view and a developer's perspective into account, because as per a 2021 study, developer productivity may take different meanings, from staying focused on the task at hand, and making meaningful progress to feeling good at end of the day. The survey results came out exactly to the above statement. GitHub concluded from its review that it gives developers a starting point and saves time looking for coding bits over the internet. As it allows for designating smaller, repetitive and insignificant parts of code to Copilot, developers can focus on the important part of programming.

From the survey, it was found that Copilot's benefits go beyond improving speed in coding. For example, it was found that Copilot has been immensely successful in improving coder satisfaction. Around 60-75% of users surveyed reported satisfaction with using Copilot and feeling less frustrated while using Copilot. While around 73% of the developers said Copilot helped them stay in the flow, 87% of them said they could focus on larger tasks, saving them from the trauma of doing repetitive tasks. In the paper "Evaluating the usability of code generation tools powered by large language models", they cited, "We found that, while Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks, since Copilot often provided a useful starting point and saved the effort of searching online. However, participants did face difficulties in understanding, editing, and debugging code snippets generated by Copilot, which significantly hindered their task-solving effectiveness."

Copilot isn't yet a be all end all solution for smart programming

This particular study is an addendum to the earlier study by Kalliamvakou but focuses on coding speed with Copilot and otherwise. Around 95 developers were involved in the study of which half of them found coding time reduced to half compared to those who didn't use Copilot, improving the speed to almost 55%. However, given the limited nature of the study – with only JavaScript programming involved in the experiment – it is hard to say if the results reflect reality. "It was certainly a fun experiment to do. These controlled experiments are quite time-consuming as we try to make them bigger or more comprehensive, but I'd like to explore testing for other languages in the future," she said in an interview with ZNet.