Paris— After massively experimenting with generative AI for several months, companies are expecting concrete benefits for their activities this year. Beyond the anticipated upheaval in productivity, what impact will it have on employee skills? Can generative AI help employees develop new skills? This is the subject of the second edition of the BCG study, entitled “ GenAI Doesn’t Just Increase Productivity. It Expands Capabilities ” and conducted by the BCG Henderson Institute (BCG’s think tank), Boston University and the OpenAI teams with 480 BCG consultants around the world.
The experiment was conducted on three data science exercises :
- Python programming;
- predictive modeling (Machine Learning);
- validation of statistical analyses.
The consultants in the panel, who do not have a data science background , worked on these three technical tasks. Half of them had access to GenAI, the other half did not. Their results were compared to a third control group of 44 BCG data scientists who worked without the help of GenAI.
Here are the main lessons:
Generative AI turns novices into experts in certain fields
- The most impressive results were obtained on the Python programming exercise. The “augmented” participants (those who used generative AI) obtained a score equivalent to 86% of that of the data scientists (base 100%), which is twice as high as the participants who did not use generative AI (37%). This AI-augmented group also completed their work about 10% faster than the data scientists.
- For the other two modeling and statistical analysis tasks, the “augmented” consultants also obtained higher scores than the consultants who worked without the help of GenAI (respectively 76% of the data scientists’ score versus 63%, and 88% of the data scientists ‘ score versus 68%).
Generative AI, the perfect pair to find solutions
During the predictive analysis exercise, participants augmented by generative AI were able to apply some of the most effective machine learning methods to carry out this exercise (+15 pp vs those who did not use GenAI). They thus demonstrated that AI could be a thinking partner, capable of developing their skills outside their field of expertise.
GenAI, an exoskeleton but in no way a learning tool
- In conclusion, all participants were given an assessment to measure their programming skills. Counterintuitively, participants who completed the programming task scored the same on the assessment as those who did not complete the task.
- This result suggests that generative AI would act as an exoskeleton, which would not necessarily allow the development of new skills or knowledge.
There is no doubt that knowing how to code remains a strategic advantage.
- For modeling and predictive analytics tasks, generalist consultants with coding experience performed about 10 to 15 points better than their peers who identified as novices.
- These results suggest that the engineering spirit inherent in programming—for example, the ability to break down a problem into subcomponents that can be efficiently checked and corrected—is a critical strategic advantage.
The authors of the report make several recommendations to companies in terms of recruitment, training and management.
This study raises several questions, and in particular: what impact will generative AI have on the quality of code produced by novice employees? Will the growth of augmented neo data scientists affect the careers and salaries of experienced developers?
