The Evolution of Engineering Interviews: A Few Thoughts

Engineering work has changed. Engineering interviews should change with it.

AI is now part of the engineering toolset. The useful question is no longer whether a candidate can produce code without AI. It is whether they can use AI to move faster without giving up technical judgment, quality, or ownership.

That does not make engineering fundamentals less important. It makes them more important. AI can generate a large amount of plausible but poorly structured code very quickly. Someone still needs to recognize bad abstractions, hidden failure modes, unsafe assumptions, and unnecessary complexity.

The bar should therefore test two abilities together:

1. Can this person use AI effectively?
2. Do they understand the work well enough to judge and improve what AI produces?

My first signal: learning velocity

The first thing I want to understand is whether someone is actively keeping up with AI.

I do not mean whether they can name every new model or repeat the latest discourse. I want to know what they have actually tried recently, how those tools changed their workflow, where the tools failed, and what they stopped using.

The specific tool matters less than the learning loop. In a field changing this quickly, recent experimentation is evidence that the candidate can continue adapting after they join.

What I would evaluate now

1. Engineering fundamentals

Candidates still need to reason about data structures, interfaces, state, failure modes, performance, security, and maintainability.

The difference is that I care less about whether they recall a memorised solution and more about whether they can apply fundamentals to an unfamiliar problem. When AI produces something that looks correct but is not, can they see why?

2. Problem framing and decomposition

Real engineering work rarely arrives as a clean prompt with a single correct answer.

I would give candidates an ambiguous problem and see how they define the objective, expose assumptions, identify constraints, and break the work into testable parts. AI makes implementation cheaper, but it does not decide what should be built or how the problem should be structured.

3. AI collaboration

Candidates should be allowed to use AI during the interview.

The interviewer should observe how they brief it, provide context, challenge its output, recover from a weak direction, and decide what requires direct inspection. Good AI use is not accepting more generated code. It is creating a controlled loop between intent, generation, review, and correction.

4. AI system design and harness engineering

Building an AI product is not the same as calling a model API.

The model is one component. A production AI system also needs instructions, context, retrieval, tools, state, memory, permissions, control flow, observability, and stopping conditions. The harness connects these components and controls how the model operates inside the product.

I would test whether the candidate can reason about questions such as:

• What context should the model receive?
• Which steps require model judgment, and which should remain deterministic code?
• What tools should the model be allowed to use?
• How should state and memory be maintained?
• When should the system retry, stop, or request human approval?
• How will engineers understand what happened during a multi-step run?

A strong candidate does not simply choose a model. They design the system around it.

5. Evaluation design

Evaluation has two distinct layers.

Code and system evaluation asks whether the software works correctly. This includes unit tests, integration tests, regression tests, type checks, security checks, performance monitoring, and rollback mechanisms.

AI product evaluation asks whether the product behaves well enough for its intended users. This requires representative tasks, explicit quality criteria, repeated trials, human review where necessary, safety and adversarial cases, latency and cost measurement, and feedback from production failures.

For an agentic product, the final answer is not enough. We also need to evaluate tool selection, intermediate actions, state changes, permission handling, and whether the system knows when to stop or escalate.

The distinction is simple: the harness controls how the AI system operates; the evaluation system tells us whether it is operating well.

6. Debugging and verification

AI systems often fail without producing an obvious error. The output may be fluent, the code may run, and the result may still be wrong.

I would introduce a subtle failure and ask the candidate to investigate it. Can they inspect evidence, trace the failure across model and software boundaries, isolate the cause, and prove that the fix works? This is a stronger signal than whether they reached a correct answer on the first attempt.

7. Production judgment and ownership

The final question is whether the candidate can own the result beyond the interview environment.

What happens when the model changes? How will the team detect regressions? Which actions need permission checks? What is the fallback when the AI is uncertain? How should cost, latency, and quality be traded off?

Shipping is not the end of the task. The engineer remains accountable for the system’s behavior.

What matters less, and what matters more

What matters less is memorized syntax, speed in a blank editor, and reproducing familiar algorithm patterns without assistance.

What matters more is whether someone can frame the right problem, apply fundamentals, direct AI, design the system around the model, evaluate both code and product behavior, debug failures, and own the production outcome.

AI changes how code is produced. It does not remove the need for engineering judgment. It raises the value of the person who can turn generated output into a reliable system.