AI Agent Skills Skill Transfer

Why We Keep Humans at the Center of AI Skill Transfer

AI skill transfer should involve two humans: an AI Skill Designer who guides skill acquisition and a subject matter expert (SME), with one or more AI agents observing their interaction.

By

Editorial collage of two professionals reviewing source evidence and connected skill cards

Someone recently asked me a reasonable question about RapidKT: Why should an AI Skill Designer interview an SME? Why not let a chatbot do it?

The model I advocate brings two people together. The AI Skill Designer guides skill acquisition through questions, examples, and checks of understanding. The SME demonstrates the work and explains the judgment behind it. One or more AI agents observe the interaction and the materials the two people examine, helping turn what they learn into candidate AI Agent Skills.

My experience has taught me that the right human conversation can bring out expert judgment that does not surface in a straightforward exchange of questions and answers. But experience deserves examination, not just repetition. What does the research actually say?

The research does not establish that humans must lead every interview. Studies show that AI can elicit useful information and help make expert knowledge explicit.[7][9]

What the evidence does reveal is a more important problem: being able to do something exceptionally well does not mean being able to explain it completely. And the way we listen, question, and check understanding changes what gets communicated.[1][4]

That is the foundation of my argument for skill transfer led by an AI Skill Designer and SME, with AI agents observing their interaction. A useful conversation is a starting point. The resulting AI Agent Skill still needs evidence that it can apply expert judgment.

The expert is not a database #

It is tempting to imagine skill transfer as an extraction task. Ask the right question, receive the answer, save it, and move on.

Research on expert performance shows why that picture is incomplete.

In a 2014 study published in Academic Medicine, Maura Sullivan and colleagues recorded three experts teaching an emergency airway procedure. They then conducted cognitive task analysis interviews and used additional surgeons to help construct a reference task list. Within those interviews, the proportion of steps described increased from 44% without prompting to 66% with prompting. Even after prompting, the explanations were not complete against that reference.[1]

This was a small, procedure-specific study. It is not a universal estimate of how much knowledge every expert leaves out. But it demonstrates the problem clearly: an expert can deliver an explanation while omitting material that a more structured process identifies.

An earlier study by Sullivan and colleagues found a similar pattern in colonoscopy teaching. Each of three expert surgeons described only 26% to 50% of the procedural steps identified through cognitive task analysis. Important decisions were also omitted.[2]

The distinction matters for skill transfer: expertise and the ability to articulate expertise are different capabilities.

For skill transfer, this gives the AI Skill Designer a specific job: acquire the skill by working through the SME's assumptions, distinctions, and judgment. The SME demonstrates how the work is done. The designer investigates what another person or an AI agent would need to apply that skill correctly.

A good interview investigates decisions, not just descriptions #

There is an established research tradition for this work: cognitive task analysis, a family of methods for investigating the thinking involved in skilled performance.

In their abstract, Robert Hoffman, Beth Crandall, and Nigel Shadbolt describe the Critical Decision Method as involving:

“multiple-pass event retrospection guided by probe questions.”[3]

In plain English: revisit a real event, examine it from different angles, and use targeted questions to investigate the decisions within it. Their paper describes applications to expert knowledge elicitation, system development, and instructional design.[3]

Consider a hypothetical session in which an AI Skill Designer guides skill acquisition with an experienced operations engineer acting as the SME. AI agents observe their discussion and the evidence they examine. The SME says:

“We postponed the migration because the system did not look stable.”

That sentence is an answer. It is not yet a reusable decision rule.

What did “stable” mean? Which observation raised concern? What information was available at the time? Had similar warning signs appeared before without causing trouble? What would have justified proceeding?

A useful follow-up might be: “Show me the evidence you had when you made that call, not the evidence that became available afterward.”

Another might be: “Describe a similar situation in which you made the opposite decision.”

Those questions aim to separate the expert's actual decision criteria from a broad summary of the outcome. They also create opportunities to discover where the apparent rule stops working.

Observing agents can flag unanswered questions and suggest follow-ups to the AI Skill Designer. The designer decides which questions to pursue, checks the answers against examples, and works with the SME to resolve important uncertainties.

The AI Skill Designer is responsible for deciding what still needs to be understood and acquiring enough detail to build a usable skill.

The listener is part of the skill-transfer process #

We often talk about an interview as though the speaker supplies the content and the listener merely receives it. Experimental research challenges that picture.

In “Listeners as Co-Narrators,” Janet Bavelas, Linda Coates, and Trudy Johnson studied storytelling across two experiments involving 63 pairs of participants. When listeners were distracted, they gave fewer responses connected to the specific content of the story. Their abstract reports that:

“the narrators also told their stories significantly less well, particularly at what should have been the dramatic ending.”[4]

The listener's behavior affected the speaker's performance.

This was research on personal storytelling, not a test of professional skill transfer or a comparison between humans and AI. Nevertheless, it supports a useful principle: what someone expresses is partly shaped by the interaction, not solely by what they brought into the room.[4]

For skill acquisition, my takeaway is that the AI Skill Designer must listen actively to the SME. The designer should make it possible to slow down, revisit a confusing distinction, and explain something that initially seemed too obvious to mention. Observing agents can record those clarifications and flag gaps for the designer to explore.

That is a standard for skilled interaction, not a benefit that appears automatically because the interviewer is human.

“I understand” is not evidence that a skill has transferred #

Herbert Clark and Susan Brennan's research framework on grounding in communication offers another important insight. On page 128 of their chapter, they describe how conversational partners try to:

“establish that what has been said has been understood.”[5]

Grounding is not simply acknowledging that words were spoken. It involves establishing enough mutual understanding for the purpose at hand. The authors emphasize that communication requires coordination and evidence of understanding, not just the production of sentences.[5]

For AI skill transfer, I interpret this as a requirement for the AI Skill Designer and SME to check decision rules together while the agents observe how those rules are explained and corrected.

Suppose an expert says, “We escalate only when the exception is material.” Repeating that sentence accurately does not establish what counts as material, which exceptions matter, or who has authority to decide.

The AI Skill Designer can test understanding by asking: “Let me apply your explanation to a new case. Here is what I would do and why. Where does my interpretation fail?”

That gives the SME something concrete to challenge and correct. The observing agents can use those corrections to revise the candidate skill and its test cases for the designer's review.

The AI Skill Designer and SME should challenge the agents' drafts together: a polished summary is not a substitute for checking whether the underlying rule has been understood.

Being human is not enough. AI is not disqualified #

The argument for skilled human involvement becomes weaker, not stronger, when we exaggerate what humans uniquely provide.

For example, it is not safe to assume that people always disclose more to another person. In a 2014 study by Gale Lucas and colleagues, participants interacted with a virtual interviewer they believed was controlled either by a human or by a computer. Those who believed it was computer-controlled reported less fear of self-disclosure and less concern with managing impressions. Observers also rated them as more willing to disclose.[6]

That study concerned health-screening interviews, not the capture of professional expertise. Still, it challenges the claim that a human presence necessarily makes an interview more open or productive.

The newer AI research deserves attention as well.

A 2025 preprint by Soubhik Barari and colleagues reported a web-survey experiment involving 1,800 participants. AI-based conversational probing produced more detailed and informative open-ended responses, with some cost to the respondent experience. This was evidence about improving survey data collection, not proof of equivalence to skilled expert interviewing.[7]

A 2026 conference paper on AInterviewer described a randomized pilot with 40 university students interviewed through text by either trained humans or an AI system. The researchers found no statistically significant differences in response specificity or relevance. That is promising, although a small study finding no significant difference does not establish equivalence. The task concerned students' experiences, not complex professional judgment.[8]

More directly relevant, a 2026 preprint by Mina Cho and colleagues investigated whether language models could derive usable guidance from expert math-tutoring responses. Novice participants given that guidance performed better on overall response-quality ratings than those given the expert-articulated guidance used for comparison. That advantage was driven by ratings of how caring the responses were. The differences in usefulness and how human the responses sounded were not statistically significant. The study relied substantially on model-based scoring, with an additional human-validation exercise, and examined one domain and short-term performance. It did not test live expert interviews or long-term mastery.[9]

These findings are reasons to use AI seriously, not dismiss it. They are also reasons to be precise about what has been demonstrated.

These studies do not establish that human-led interviewing is always superior. They also do not directly test the combination of an AI Skill Designer, an SME, and observing agents. For AI skill transfer, I take them as a reason to evaluate the acquisition process and the resulting skill separately.

Two humans lead skill acquisition while agents observe #

The AI Skill Designer should guide skill acquisition with the SME while one or more AI agents observe their work. This is the operating model I advocate for RapidKT. The two humans have distinct responsibilities, and both remain involved as the skill moves from acquisition to review.

The AI Skill Designer acquires the skills. The designer guides the session, probes the SME's reasoning, identifies decision criteria and boundaries, and structures what is acquired into a candidate AI Agent Skill. The designer also prepares validation cases and reviews the result with the SME.

The SME supplies the expertise. The SME demonstrates the work, explains decisions and exceptions, provides real examples, and corrects interpretations that do not reflect the practice.

Their interaction starts with real work: a decision, an incident, a difficult case, or an artifact the SME actually used. The AI Skill Designer investigates alternatives and exceptions and distinguishes what the SME observed from what they inferred. Both people work through disagreements so the resulting skill preserves the conditions that matter.

This gives the SME room to say, “That explanation sounds reasonable, but it is not why I did it.”

One or more AI agents observe the interaction. They follow the conversation and the examples and artifacts shared by the AI Skill Designer and SME. They can record decision criteria, organize source material, flag gaps, suggest follow-ups to the designer, and draft candidate skills or test cases. The SME checks whether those drafts reflect the expertise, and the designer checks whether the skill is complete enough to validate and reuse.

Self-paced contributions can prepare or extend these sessions. The SME can add an example or review a candidate skill, and the AI Skill Designer can use that input to plan the next discussion or resolve an open question. Agents can organize the material for their joint review. These contributions support the work of the two humans.

The AI Skill Designer and SME should apply the same standard throughout: do we have enough evidence to approve this skill for reuse?

The real test happens after the interview #

A long transcript is not the outcome I care about. Neither is an interview that everyone describes as pleasant.

The outcome is whether the resulting skill helps someone, or an AI agent, apply expert judgment in a situation that was not simply rehearsed during the conversation.

Research on cognitive task analysis offers an example of this distinction. In a randomized study of 26 surgical interns, a training course informed by cognitive task analysis and including hands-on simulation improved subsequent knowledge and observed procedural performance compared with traditional training. Because the intervention combined several elements, it does not isolate the effect of interviewing. But it illustrates the importance of evaluating performance, not just the quality of the instructional material.[10]

For enterprise AI skill transfer, the AI Skill Designer should test the acquired skill on new cases and ask the SME to assess whether its decisions reflect the expertise. Agents can help run those cases and surface inconsistencies for the two humans to review.

Can the resulting skill recognize when it applies? Can it handle a meaningful exception? Does it identify missing information rather than inventing a confident answer? Can reviewers trace a rule back to the evidence supporting it? Does it know when to escalate instead of acting?

A fair evaluation of this model would compare skills acquired by an AI Skill Designer and SME with observing agents against skills acquired through other arrangements. It should use representative tasks and new test cases, and account for errors, omissions, the time of both humans, and the cost of validation.

That would give us a sound basis for deciding where to automate more and where human involvement continues to earn its place.

The goal is reusable skills #

I want the interaction between an AI Skill Designer and SME, observed by one or more AI agents, to produce a reusable AI Agent Skill. The transcript is an intermediate artifact. The skill is what the two humans review, test, and approve for reuse.

I do not want to protect a human activity simply because it has traditionally been done by humans. I want to preserve the quality of the skill that comes out of it.

That is why I favor two humans at the center of AI skill transfer. The AI Skill Designer guides the process and acquires the skills. The SME brings the expertise. Observing agents help record, structure, and test what emerges from their interaction. The result must demonstrate that the acquired skill can be applied correctly.

The most important question is not “Can AI ask the questions?” It is “How will we know the skill can be applied correctly?”

References #

  1. Sullivan, M. E., Yates, K. A., Inaba, K., Lam, L., & Clark, R. E. (2014). The use of cognitive task analysis to reveal the instructional limitations of experts in the teaching of procedural skills. Academic Medicine, 89(5), 811–816.

  2. Sullivan, M. E., Ortega, A., Wasserberg, N., Kaufman, H., Nyquist, J., & Clark, R. (2008). Assessing the teaching of procedural skills: Can cognitive task analysis add to our traditional teaching methods? The American Journal of Surgery, 195(1), 20–23.

  3. Hoffman, R. R., Crandall, B., & Shadbolt, N. (1998). Use of the Critical Decision Method to elicit expert knowledge: A case study in the methodology of cognitive task analysis. Human Factors, 40(2), 254–276.

  4. Bavelas, J. B., Coates, L., & Johnson, T. (2000). Listeners as co-narrators. Journal of Personality and Social Psychology, 79(6), 941–952.

  5. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 127–149). American Psychological Association.

  6. Lucas, G. M., Gratch, J., King, A., & Morency, L.-P. (2014). It's only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior, 37, 94–100.

  7. Barari, S., Angbazo, J., Wang, N., Christian, L. M., Dean, E., Slowinski, Z., & Sepulvado, B. (2025). AI-assisted conversational interviewing: Effects on data quality and respondent experience. arXiv preprint, version 3.

  8. Gårdhus, T. P., Vitsakis, N., Frederiksen, F. L., Rogers, A., & Carlsen, H. B. (2026). AInterviewer: A platform for designing and conducting AI-led qualitative interviews. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 119–127.

  9. Cho, M., Funk, R. J., Gupta, A., & Yang, M. (2026). Large language models explain experts better than experts themselves. arXiv preprint, version 1.

  10. Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E., Theodorou, D., & Maupin, F. (2004). Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. The American Journal of Surgery, 187(1), 114–119.

Cite this article #

When quoting or referencing this article, credit Paul Greenberg and RapidKT and link to the article. Include the section title when directing readers to a particular passage.

Suggested citation

Greenberg, P. (2026, September 15). Why We Keep Humans at the Center of AI Skill Transfer. RapidKT. https://rapidkt.com/pages/blog/why_we_keep_humans_at_the_center_of_ai_skill_transfer

In-text reference: (Greenberg, 2026).

RapidKT Plain Text Website