How to Assess an AI Engineer Who Uses Coding Assistants

If your engineers can use coding assistants on the job, it makes sense to let candidates use them in a relevant interview exercise. But that changes what you need to observe. A working patch tells you less when a tool can produce one before the candidate has understood the problem.

For a Seed–Series B AI startup, the question is whether this engineer can decide what to delegate, detect when the tool is wrong and take responsibility for the result. You can test that in one focused session without asking someone to build a product over the weekend.

Choose the work before choosing the tool

Start with a task the hire will actually face. An applied AI engineer might investigate why a retrieval feature returns plausible answers for the wrong account. A platform engineer might diagnose a retry that produces duplicate jobs. Neither needs a blank editor and a generic algorithm puzzle to show how they work.

Make the task small enough to explore during the interview. Supply a fictional or sanitised repository, instructions that work from a clean checkout and a way to reproduce at least one symptom. A candidate should spend the session on engineering decisions, rather than chasing credentials or fixing your setup.

If you cannot agree on the right task, revisit the remit. Our guide to choosing between a product engineer and a platform engineer can help identify what the next hire actually needs to own.

Set the same rules for everyone

Before the interview, tell candidates which tools they may use, what data can leave the environment and what you expect them to explain. Provide access to the same approved tool where possible. If you offer an alternative, assess the same competencies. Nobody should have to buy a subscription or disclose a previous employer’s code to compete.

Explain that tool use is optional and that you will assess the result and the reasoning. A candidate who fixes a narrow problem without an assistant may have made the right call. So may someone who uses one to find relevant files, then verifies its claims against the repository.

Use an exercise with a failure worth investigating

Here is an example for an applied AI hire. A fictional support assistant sometimes retrieves an article belonging to another customer. The obvious examples pass, but one tenant filter is lost when the first search returns no results.

Give the candidate a simple reproduction and ask for a safe change. They might ask an assistant to trace the fallback path or suggest tests. What matters is how they check its account of the code and what they do when it proposes a plausible fix in the wrong layer.

A strong candidate may reproduce the failure, trace the missing filter, add a regression case for the fallback and explain how they would confirm the fix in production. Ask what their tests still do not establish. That last answer often tells you more than the patch.

GitHub’s guide to reviewing AI-generated code recommends functional checks and checking that the code fits the project’s requirements and architecture. Those are useful interview signals: did the candidate establish the intended behaviour, and can they show that the change meets it?

Score the decisions you observed

• Framing: Did they identify the expected behaviour and clarify what would make a fix safe?
• Delegation: Did they give the assistant a bounded task and check its assumptions before acting?
• Debugging: Did they reproduce the issue and follow evidence when the first explanation was wrong?
• Verification: Did their tests cover the failing path, and did they check for new risks or side effects?
• Ownership: Can they explain the patch, its limitations and what they would monitor or roll back?

Write down concrete observations before comparing candidates. “Used AI well” is too vague. “Rejected a broader rewrite after tracing the bug to the fallback path and added a regression test” is evidence another interviewer can understand.

Ask for a walkthrough before you call it done

Reserve time at the end for the candidate to walk through the change as if you were reviewing a pull request. Ask what they checked themselves, what they discarded and where they remain uncertain. Then change one assumption: what if the customer filter is missing upstream? You are looking for a reasoned response, rather than an immediate second patch.

If the exercise is incomplete, distinguish setup issues, sensible prioritisation and gaps in skill. Someone who isolates a serious failure and explains the next verification step may offer stronger evidence than someone who completes more code but cannot account for what changed.

Should you score speed?

Time matters when it reflects a real constraint. It is a poor standalone measure of engineering quality. METR’s February 2026 update explains why estimating coding tool productivity is difficult as the tools and the people willing to participate change. Its experiments are not benchmarks for your interview.

Use the timebox to see how someone prioritises. Can they find the important failure, narrow the change and communicate the remaining risk? For a small team where the person writing the patch may also carry the pager, that is a better hiring signal than lines of code or an impressive prompt.

Dataworks helps Seed–Series B founders hire technical teams across the US and Europe. If you’re building an AI engineering interview and want it to reflect the work, talk to us about the role and what you need to assess.

Photo by Chris Ried on Unsplash.