It is like a new story of the “turtle and rabbit”: A group of experienced software engineers has completed an experiment in which they do some of their work with the help of AI tools. The developers think like the fast rabbits and expected AI to accelerate their work and increase productivity. Instead, the technology slowed them down more. The AI-free turtle approach would have been faster in the context of the experiment.
The results of this experiment, published in A study This month, the software developers who are commissioned to use AI – and the authors of the study, Joel Becker and Nate Rush, technical employees of the model evaluation and threat to the non -profit technology organization organization (METR).
The researchers commissioned 16 software developers who had an average of five years of experience to do 246 tasks, each part of projects on which they were already working on. For half of the tasks, developers were allowed to use AI tools – most of the Code Editor Cursor Pro or Claude 3.5/3.7 Sonett – and for the other half, the developers held the tasks themselves.
Since the AI tools would make them more productive, the software developers predicted that the technology would shorten their final time of the tasks by an average of 24%. Instead, AI led to her task time higher than they did not use the technology.
“While I like to believe that my productivity does not suffer for my tasks when using AI, it is not unlikely that she may not have helped me as much as I expected it or maybe even hindered my efforts,” wrote Philipp Burckhardt, a participant in the study Blog post About his experience.
Why AI slows some workers
Where did the rabbits refuse from the path? In the middle of their own projects, the experienced developers probably turned to their work with numerous additional context that their AI assistants did not have, which means that according to the study, they had to retrofit their own agenda and problem-solving strategies.
“The majority of the developers who took part in the study Assets.
Other developers lost time with writing to the chatbots or waited for the AI to generate results.
The results of the study contradict high promises about the ability of AI, the economy and the workforce, including 15% Thrust to the US GDP until 2035 and finally 25% Increase in productivity.
But Rush and Becker have prevented them from making extensive claims about what the results of the study mean for the future of the AI.
On the one hand, the sample of the study was small and not generalizable, including only a special group of people to whom these AI tools were brand new. According to the authors, the study also measures the technology at a certain point in time, and do not rule out the possibility that AI tools could be developed in the future that would actually help developers to improve their workflow.
The purpose of the study was to pump the brakes for the Torrid implementation of AI in the workplace and elsewhere and to make more data known about the actual effects of AI and to make further decisions about their applications.
“Some of the decisions that we make in terms of development and use of these systems may be very high consequences,” said Rush. “If we want to do this, we don’t just take the obvious answer. Let us make high quality measurements.”
The broader effects of AI on productivity
Economists have already claimed that the research of METR coincides with broader stories about AI and productivity. While AI begins too Jump off in entry positionsaccordingly LinkedIn Anesh Raman, Chief Economic Opportunity Officer, can offer decreasing returns for qualified workers and experienced software developers.
“For those who have had five years of experience for 20 years or in this specific example, it may not be their main task we should look for, and force them to use these tools if they already work well with their existing work methods” Assets.
In a similar way, Humlum has carried out the effects of AI on productivity. He found in A Work study From May, productivity under 25,000 workers at 7,000 jobs in Denmark – a country with a similar AI recording as the USA – improved a modest 3% among the employees who used the tools.
Humlum research supports the with economist and Nobel laureate Daron Acemoglu from the claim that the markets overestimated the productivity gains from AI. Acemoglu argues that only 4.6% of the tasks within the US economy become more efficient with AI.
“In the increase in automation everything, even the processes that should not be automated, companies waste time and energy and do not receive any of the promised productivity advantages” wrote for Assets. “The hard truth is that the profits of productivity gains from every technology requires organizational adjustments, a number of additional investments and improvements in the skills of employees through training and learning at work.”
The case of the disabled productivity of the software developer indicates this need for critical thinking when AI tools are implemented, said Humlum. While previous research on AI productivity were examined self -registered data or specific and contained tasksData on the challenges of qualified workers who use the technology make the image difficult.
“In the real world, many tasks are not as easy as typing in Chatgpt,” said Humlum. “Many experts have had a lot of experience (they have) that are very advantageous, and we should not only ignore this and give up this valuable specialist knowledge that has been accumulated.”
“I would only accept this as a good memory to be very careful when these tools should be used,” he added.
