Andela Research
We measured the impact on forty-one professional engineers’ proficiency during two weeks of structured training on agentic coding workflows. This is what moved, what did not, and what remained stubbornly hard.
Author: Cory Hymel - Head of Research, Andela
The question
AI coding tools ended their autocomplete era some time ago. They read repositories, edit files, run commands, call other tools, and check their own work. Working with one is an entirely different skill from writing code.
The skills that matter now sit around the agent rather than inside it. Someone has to design the intent, expose the right context, decide what the agent is allowed to touch, and define what finished actually means. None of that involves typing or reviewing diffs.
The research and first-hand experience is consistent that these habits arrive after the adoption of the tool. Developers using code-generation systems save effort in one place and spend it again on understanding, debugging, and validating what came back (Vaithilingam et al., 2022; Mozannar et al., 2024). A survey of developers found the recurring complaint was not capability but control; having to arm wrestle to get outputs that satisfy real requirements (Liang et al., 2024).
So...can experienced engineers pick up the operational knowledge that agentic work requires from a short, structured program? More precisely: Can they be taught to develop the specific habits of working with agents transfer within two weeks?
Why this matters
Andela's work starts from a skills-first reading of the market: what someone can currently do matters more than a paper resume. Agentic tooling makes measuring that harder and more urgent at the same time. Key reason: the relevant skill is new enough that almost nobody's résumé contains it, or has a way to accurately reflect it.
That is the practical shape of talent debt. Organizations buy the licences, announce the rollout, and then discover that the gap between having the agent and getting value from it is a learning gap that nobody budgeted for. It shows up as inconsistent adoption, quiet abandonment, and a small number of enthusiasts whose habits never spread.
Rolling out a coding agent is a procurement decision. Getting proficiency out of it is a learning decision. The two are routinely confused.
This study is a modest but concrete data point against that. Two weeks of structure moved assessed knowledge by a meaningful margin. Further, it people who started furthest behind made the most progress. Notably, it did not move mechanism selection much, which tells you where a second pass should go.
Whether you are currently funding or or neither,, this study offers the argument for funding both.
At a glance
The average score went from about percent to about percent. Nearly three in five participants improved, and nobody dropped more than a single question.
· 95% CI [, ] · p · Wilcoxon p · Cohen's dz = · n = pairedParticipants who started with six or fewer questions right gained about three and a half questions. Those who started at nine or above gained less.
+ vs. + vs. + items · n = / / · higher-baseline group finished at of 12, leaving items of headroomThe biggest jumps were all operational. Participants already had a grasp of AI supporting coding. What they didn’t have, however, was the working knowledge underneath those principles. In other words, they were not able to recognize when a failure is a sandbox limit rather than a model error, keeping repository guidance lean, or specifying what "done" looks like.
Sandbox constraints + pp · repository guidance + pp · verification criteria + ppItem level
A single average hides almost everything interesting and twelve questions move in twelve different ways.
Each row is one question with the pale opal dot showing where the cohort stood before the program, the emerald dot is where they stood after. Select any row to read the question and its answer.
Percentages are the share of the paired participants answering correctly and deltas are in percentage points.
We found that the pattern of gains is as important as the overall score change. The largest improvements concerned the environment around the agent: context hygiene, verification, execution constraints, isolated parallel work, and team configuration. Participants already performed well on the broad agent-first principle that humans should specify intent and verify outcomes while agents execute appropriate tasks. However, their remaining difficulty emerged when multiple agentic mechanisms appeared plausible. This distinction suggests that strategic understanding and operational judgment develop at different rates.
The baseline-group results also offered an interesting insight. Participants entering with the lowest assessment scores improved most, while the highest group changed little. A plausible interpretation is that the program helped establish a more consistent foundation among learners with limited initial knowledge. However, the fixed 12-item scale offered less headroom to high scorers and may have been insufficiently difficult to detect advanced learning.
TLDR; strategic understanding and operational judgement develop at different rates. If you have just taken the test, you probably recognized it: the questions about what an agent is or does are easy but the questions about which mechanisms to reach for and when were not.
Twelve rows of dots, one for each person on each question. Every column is the same participant throughout, so the figure can be read across a question or down a person. Solid emerald means someone got it right who had not before. Solid sienna means the reverse. The pale dots either side were already right or still wrong both times.
Each column is one participant, in the same order in every row. Read a column downwards to follow one person through all twelve questions. Columns are ordered by total gain, largest first, so the people the program moved most are on the left and the five who finished lower than they started are on the right.
Overall Gains
As to be expected, the individuals that started with the highest scores barely moved. However, those with low scores did gain significantly showing that short, structured learnings can improve the overall baseline of an engineering organization.
Three heavy lines on a twelve-point scale, one per group, matching its colour from the card beneath it. Behind them are the individual trajectories they average; where several people took the same path the lines stack and the colour deepens. Turn on the headroom shading to see how much of the test each group still had available to gain.
The group that started at nine or above finished barely more than a question short of a perfect score. That is all the improvement the test had left to offer them; a key learning about the course curriculum. We can also interpret this result as showing that generic, one-size-fits-all learning programs are not sufficient in improving more senior engineers.
Having a process to understand where an engineer's skills currently lie and offering tailored training is a better way to show overall improvements similar to the gains the lower starters showed in this program.
Persistent difficulty
These are the three lowest scores after the program.
All three questions have something in common...every one of them asks you to choose between adjacent plausible mechanisms. An MCP connector or a markdown file checked into the repository. The SDK, or a scheduled automation, or a packaged skill. A specialized agent role, or the default one with better prompting.
All four answers describe something a competent engineer might actually build, and three of them would half-work. The question is not whether someone understands the principle—the cohort demonstrably did—, but whether someone knows which lever a particular situation calls for.
Knowing the principle does not tell you which lever to pull.
Importantly, this finding shows that strategic framing transferred in two weeks but mechanism selection did not. It's the part of agentic engineering that looks most like ordinary senior judgement: knowing which of several defensible tools fits the shape of the problem, which is learned mostly by doing it wrong a few times.
The question now becomes: Where should this mechanism learning come from?
It's hard to answer but a place to start is with an organization's AI policies. For example, an organization could mandate that MCP is only used in certain places, or that AGENTS.md files are always structured like X. It's not a perfect answer and given that agentic engineering is so new, a lot of these mechanism best practices will become more standard over time.
The instrument
These are the actual twelve questions, unchanged. Scenario-based, four options, one right answer each. You will get the reasoning after every question, which the participants did not.
Method
Two weeks, fully online, two modules, four live hours, and a required quiz at the end of each module.
Limitations
The learning program itself was the primary goal and this study was done as an ancillary effort.
This is the single largest limitation. Scores rose, and the program is the most plausible explanation, but nothing here rules out maturation, motivation, or anything else that happened to these engineers over the same two weeks. A controlled trial would say something this design cannot.
Familiarity plausibly explains part of the speed gain and may have primed attention to particular topics during instruction. Stated fairly in the other direction: participants received no scores and no answer feedback after the pre-test, so familiarity alone does not explain newly correct answers on items they had previously missed.
It was developed by Andela's internal Assessment Team and reviewed by subject-matter experts, which is a reasonable standard for program evaluation. It is not an independently validated research scale, and it was written by people who knew what was about to be taught.
There is very little headroom for strong starters, which is why the flat higher-baseline line cannot carry much weight. It also makes every individual score coarse: one question is more than eight percentage points.
The study measures what participants could recognise and select in written scenarios. It does not measure whether they went on to ship better, safer or faster code. Those are the outcomes that matter, and this design does not reach them.
The research question and the statistical framing were developed after the data was collected. Read this as program-evaluation evidence, not as causal proof, and treat the confidence interval as descriptive rather than confirmatory.