Skip to content

This page runs its figures, its statistics and its assessment in the browser. With JavaScript disabled you will see the article structure but not the numbers. The full whitepaper is linked at the foot of the page.

Andela Research

When it comes to coding agents, access and proficiency are not the same thing.

We measured the impact on 41 professional engineers’ proficiency during two weeks of structured training on agentic coding workflows. This is what moved, what did not, and what remained stubbornly hard.

Author: Cory Hymel, Head of Research, Andela

engineers assessed before and after the program

average gain on a 12-item scenario test

2 weeksof structured instruction, delivered online

The question

What we set out to test

AI coding tools ended their autocomplete era some time ago. They read repositories, edit files, run commands, call other tools, and check their own work. Working with one is an entirely different skill from writing code.

The skills that matter now sit around the agent rather than inside it. Someone has to design the intent, expose the right context, decide what the agent is allowed to touch, and define what finished actually means. None of that involves typing or reviewing diffs.

The research and first-hand experience is consistent that these habits arrive after the adoption of the tool. Developers using code-generation systems save effort in one place and spend it again on understanding, debugging, and validating what came back (Vaithilingam et al., 2022; Mozannar et al., 2024). A survey of developers found the recurring complaint was not capability but control; having to arm wrestle to get outputs that satisfy real requirements (Liang et al., 2024).

So...can experienced engineers pick up the operational knowledge that agentic work requires from a short, structured program? More precisely: Can the specific habits of working with agents be taught within two weeks?

Why this matters

Procurement is not proficiency

Andela's work starts from a skills-first reading of the market: what someone can currently do matters more than a paper resume. Agentic tooling makes measuring that harder and more urgent at the same time. Key reason: the relevant skill is new enough that almost nobody's resume contains it, or has a way to accurately reflect it.

That is the practical shape of talent debt. Organizations buy the licenses, announce the rollout, and then discover that the gap between having the agent and getting value from it is a learning gap that nobody budgeted for. It shows up as inconsistent adoption, quiet abandonment, and a small number of enthusiasts whose habits never spread.

Rolling out a coding agent is a procurement decision. Getting proficiency out of it is a learning decision. The two are routinely confused.

This study is a modest but concrete data point against that. Two weeks of structure moved assessed knowledge by a meaningful margin. Further, the people who started furthest behind made the most progress. Notably, it did not move mechanism selection much, which tells you where a second pass should go.

Whether you are currently funding procurement, learning, or neither, this study offers the argument for funding both.

At a glance

Three findings

01

Two weeks moved the needle

The average score went from about percent to about percent. Nearly three in five participants improved, and nobody dropped more than a single question.

· 95% CI [, ] · p · Wilcoxon p · Cohen's d_z = · n = paired

02

The people who knew least gained the most

Participants who started with six or fewer questions right gained about three and a half questions. Those who started at nine or above gained less.

+ vs. + vs. + items · n = / / · higher-baseline group finished at of 12, leaving items of headroom

03

The gains were in mechanics, not ideas

The biggest jumps were all operational. Participants already had a grasp of AI supporting coding. What they didn’t have, however, was the working knowledge underneath those principles. In other words, they were not able to recognize when a failure is a sandbox limit rather than a model error, keeping repository guidance lean, or specifying what "done" looks like.

Sandbox constraints + pp · repository guidance + pp · verification criteria + pp

Item level

What advanced, and what moved backwards

A single average hides almost everything interesting and twelve questions move in twelve different ways.

Figure 1 · Percentage answering each question correctly, before and after

Each row is one question with the pale opal dot showing where the cohort stood before the program, the emerald dot is where they stood after. Select any row to read the question and its answer.

Sort

Before the program After the program Moved forward Moved backward

Percentages are the share of the paired participants answering correctly and deltas are in percentage points.

Interpretation

We found that the pattern of gains is as important as the overall score change. The largest improvements concerned the environment around the agent: context hygiene, verification, execution constraints, isolated parallel work, and team configuration. Participants already performed well on the broad agent-first principle that humans should specify intent and verify outcomes while agents execute appropriate tasks. However, their remaining difficulty emerged when multiple agentic mechanisms appeared plausible. This distinction suggests that strategic understanding and operational judgment develop at different rates.

The baseline-group results also offered an interesting insight. Participants entering with the lowest assessment scores improved most, while the highest group changed little. A plausible interpretation is that the program helped establish a more consistent foundation among learners with limited initial knowledge. However, the fixed 12-item scale offered less headroom to high scorers and may have been insufficiently difficult to detect advanced learning.

TL;DR: Strategic understanding and operational judgment develop at different rates. If you have just taken the test, you probably recognized it: the questions about what an agent is or does are easy but the questions about which mechanisms to reach for and when were not.

Figure 2 · Individual results per question

Twelve rows of dots, one for each person on each question. Every column is the same participant throughout, so the figure can be read across a question or down a person. Solid emerald means someone got it right who had not before. Solid sienna means the reverse. The pale dots either side were already right or still wrong both times.

View

Wrong → correct Correct → wrong Correct both times Wrong both times

Each column is one participant, in the same order in every row. Read a column downwards to follow one person through all twelve questions. Columns are ordered by total gain, largest first, so the people the program moved most are on the left and the five who finished lower than they started are on the right.

Overall Gains

The strongest starters barely moved

As to be expected, the individuals that started with the highest scores barely moved. However, those with low scores did gain significantly showing that short, structured learnings can improve the overall baseline of an engineering organization.

Figure 3 · Mean score before and after, by starting group

Three heavy lines on a twelve-point scale, one per group, matching its color from the card beneath it. Behind them are the individual trajectories they average; where several people took the same path the lines stack and the colour deepens. Turn on the headroom shading to see how much of the test each group still had available to gain.

The group that started at nine or above finished barely more than a question short of a perfect score. That is all the improvement the test had left to offer them; a key learning about the course curriculum. We can also interpret this result as showing that generic, one-size-fits-all learning programs are not sufficient in improving more senior engineers.

Having a process to understand where an engineer's skills currently lie and offering tailored training is a better way to show overall improvements similar to the gains the lower starters showed in this program.

Persistent difficulty

Three questions that stayed hard

These are the three lowest scores after the program.

Why these three and not others

All three questions have something in common...every one of them asks you to choose between adjacent plausible mechanisms. An MCP connector or a markdown file checked into the repository. The SDK, or a scheduled automation, or a packaged skill. A specialized agent role, or the default one with better prompting.

All four answers describe something a competent engineer might actually build, and three of them would half-work. The question is not whether someone understands the principle—the cohort demonstrably did—but whether someone knows which lever a particular situation calls for.

Knowing the principle does not tell you which lever to pull.

Importantly, this finding shows that strategic framing transferred in two weeks but mechanism selection did not. It's the part of agentic engineering that looks most like ordinary senior judgment: knowing which of several defensible tools fits the shape of the problem, which is learned mostly by doing it wrong a few times.

The question now becomes: Where should this mechanism learning come from?

It's hard to answer but a place to start is with an organization's AI policies. For example, an organization could mandate that MCP is only used in certain places, or that AGENTS.md files are always structured like X. It's not a perfect answer and given that agentic engineering is so new, a lot of these mechanism best practices will become more standard over time.

The instrument

Curious to see how you would do?
Take the assessment yourself.

These are the actual twelve questions, unchanged. Scenario-based, four options, one right answer each. You will get the reasoning after every question, which the participants did not.

Method

Inside the program

Two weeks, fully online, two modules, four live hours, and a required quiz at the end of each module.

1Two weeks, fully online Delivered to professional developers already working in engineering roles.

2Asynchronous materials Static study material available throughout, worked through at each participant's own pace.

3Four one-hour live sessions Two per module, run as concept → demonstration → practice → support, with live demos, guided exercises and troubleshooting.

4Module 1: Codex Foundations Setup, the agent loop, prompt and context design, planning, verification criteria, repository guidance, and common development workflows.

5Module 2: Advanced Workflows Configuration, permissions and sandboxing, reusable skills, Model Context Protocol connections, automations, memory, parallel work, code review, and long-horizon execution.

6A required quiz per module Ten items each, pass required, retries allowed.

7The post-program assessment The identical twelve-item instrument, taken immediately after instruction and before anyone was given access to the capstone.

8A separate capstone week No new teaching was delivered during it. Capstone and hackathon outcomes are not analyzed in the paper and are not discussed here.

Limitations

What this report does not claim

The learning program itself was the primary goal and this study was done as an ancillary effort.

No control group

This is the single largest limitation. Scores rose, and the program is the most plausible explanation, but nothing here rules out maturation, motivation, or anything else that happened to these engineers over the same two weeks. A controlled trial would say something this design cannot.

The same test, twice

Familiarity plausibly explains part of the speed gain and may have primed attention to particular topics during instruction. Stated fairly in the other direction: participants received no scores and no answer feedback after the pre-test, so familiarity alone does not explain newly correct answers on items they had previously missed.

The instrument was built for the curriculum

It was developed by Andela's internal Assessment Team and reviewed by subject-matter experts, which is a reasonable standard for program evaluation. It is not an independently validated research scale, and it was written by people who knew what was about to be taught.

Twelve questions is a short ruler

There is very little headroom for strong starters, which is why the flat higher-baseline line cannot carry much weight. It also makes every individual score coarse: one question is more than eight percentage points.

Knowledge, not performance

The study measures what participants could recognize and select in written scenarios. It does not measure whether they went on to ship better, safer or faster code. Those are the outcomes that matter, and this design does not reach them.

Exploratory by construction

The research question and the statistical framing were developed after the data was collected. Read this as program-evaluation evidence, not as causal proof, and treat the confidence interval as descriptive rather than confirmatory.