Paper Breakdown: AI Made Them Data Scientists, Until It Was Switched Off

Executive summary

• The paper: Wiles, Krayer, Abbadi, Awasthi, Kennedy, Mishkin, Sack and Candelon, "Generative AI and the temporary upskilling of knowledge workers," Nature Human Behaviour (published online October 26, 2026).
• The question: can generative AI allow professionals to perform well on tasks outside their expertise, and do they actually learn those skills in the process?
• The answer: yes to the first, no to the second. Consultants with ChatGPT outperformed those without it on data science tasks, but showed no additional technical knowledge when tested without the tool.
• The lesson for leaders: AI can extend what a team can do today. It does not, on its own, build what the team knows tomorrow.

Why this study matters

Organizations increasingly treat generative AI as a shortcut to new capabilities. A communications team that "now does data analysis," a campaign that "now codes its own dashboards," a ministry that "now writes its own models." The implicit promise is reskilling: give people the tool, and they become the specialist.

This paper tests that promise under rigorous conditions. It distinguishes between two things that are easy to confuse: performance with a tool and capability without it. That distinction is central for any institution that must decide whom to hire, whom to train and how much to depend on a single technology provider.

The design

A randomized controlled trial

• Participants: nearly 500 consultants at Boston Consulting Group without data science training. According to BCG's account of the experiment, 44 of the firm's data scientists, working without AI, served as a benchmark.
• Treatment: access to and training in ChatGPT (an enterprise version with GPT-4 and data analysis features). The control group worked without it.
• Tasks: three realistic data science problems: writing code to merge and clean datasets, building a predictive model, and validating statistical analyses. The researchers designed the tasks so that the AI could not solve them on its own; a human still had to direct and check the work.
• The post-test: after the tasks, participants answered technical questions without AI, to see whether anything had been learned.

Why randomization matters here

Many claims about AI productivity come from self-reports or from comparing early adopters with everyone else. Those comparisons are confounded: the people who adopt a tool first are often already more skilled. Random assignment removes that problem and allows a causal reading of the differences between groups.

The findings

1. Large performance gains. Consultants with ChatGPT scored 49, 20 and 18 percentage points higher than the control group on the three tasks.
2. Near-expert results, in part. On two of the three tasks, the AI-assisted group came close to the performance of real data scientists. Predictive modeling was the hardest: the tool often misread instructions, and users were more easily led astray.
3. No retained knowledge. On the post-test, without AI, the treated group did no better than the control group. The authors describe generative AI as working like an exoskeleton: it amplifies ability while worn, but the strength does not remain once it is removed.
4. Prior skill still matters. According to BCG's summary, participants with moderate coding experience outperformed self-described novices across all tasks, even with the same tool.

The theoretical frame

The results connect to two lines of research.

The first is work on cognitive offloading. Risko and Gilbert (2016) define it as the use of physical action or external tools to reduce the cognitive demands of a task, from writing a shopping list to relying on a smartphone's memory. Offloading improves immediate performance, but it can reduce the internal processing through which people build durable knowledge. Generative AI is offloading at an unprecedented scale: it can take over not only memory but reasoning steps.

The second is the "jagged frontier" described in an earlier experiment with BCG consultants. Dell'Acqua et al. (2023) found that AI substantially improved performance on tasks inside its capabilities but worsened it on tasks just outside them, where users trusted plausible but wrong answers. The new study's difficulty with predictive modeling, where users were "led astray," is consistent with that pattern.

Limitations worth noting

• One firm, one profession. Consultants at an elite firm are not representative of all knowledge workers.
• A short exposure. Participants were not told they would be tested, and the experiment did not involve repeated practice. As BCG's summary notes, learning might occur with repetition and intention.
• A specific tool and moment. The trial used one model at one point in time; capabilities have changed since.
• Conflicts of interest disclosed. Some authors have ties to BCG or OpenAI, and BCG supported the research. The randomized design limits, but does not eliminate, the relevance of that disclosure.

Lessons for senior leadership

1. Distinguish access from expertise. A team that performs well with AI has a capability that depends on the tool. Plan for what happens if access, pricing or performance changes.
2. Design for learning, not only output. If the goal is reskilling, build in tasks without AI, explanations of why a solution works and periodic testing. Learning requires effort that the tool removes.
3. Keep specialists in the loop. On the hardest task, assisted non-experts went wrong in ways they could not detect. Critical analyses, from polling models to financial projections, still need someone who can verify them independently.
4. Hire for judgment. Prior experience improved results even with AI. The ability to frame a problem and recognize an error is what makes the tool useful.
5. Measure capability, not just productivity. Dashboards that track speed and volume will show success. A periodic test without the tool shows whether the organization is actually becoming more capable.

The study's central image deserves a place in every AI strategy. An exoskeleton is valuable. But an organization that confuses the exoskeleton with its own muscles will discover the difference at the worst possible moment.

References

Boston Consulting Group. (2024, September 5). Generative AI instantly elevates the aptitude of knowledge workers [Press release]. https://www.bcg.com/press/5september2024-generative-ai-knowledge-workers-consultants

Dell'Acqua, F., McFowland, E., III, Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality (Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013). SSRN. https://papers.ssrn.com/abstract=4573321

Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002

Wiles, E., Krayer, L., Abbadi, M., Awasthi, U., Kennedy, R., Mishkin, P., Sack, D., & Candelon, F. (2026). Generative AI and the temporary upskilling of knowledge workers. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02566-6