Get started

Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Machine learning and AI engineering · Mid level

How to assess a AI Engineer

This is the one role where a candidate can correctly argue that using AI is the job, which makes the standard integrity posture incoherent — Karat's 2026 survey of 400 engineering leaders found 62 percent of organisations still prohibit AI use in technical interviews while over half of candidates use it anyway. Banning the tool here tests compliance, not skill. The deeper failure is that a take-home demo is graded on whether it works once, and this role's entire value is knowing whether it works reliably. Building a plausible retrieval pipeline is now a solved, assisted task; constructing the eval set that proves the last change helped, and the failure taxonomy that says which of five chained steps broke, is not — and no part of the standard screen asks for either. The honest limit: a bounded exercise reads evaluation discipline and failure decomposition well, and predicts little about whether someone can hold quality on a system whose underlying model is swapped out from under them twice a year.

An AI engineer builds software whose central component is non-deterministic, undocumented in its failure modes, and replaced by its vendor on a schedule nobody controls. That single fact reorganises the whole skill set. In conventional engineering, correctness is asserted by tests written once and trusted thereafter. Here, the equivalent of a test suite has to be constructed by the engineer out of examples, graded by a mixture of exact checks, heuristics and model-graded rubrics, maintained as the underlying model changes, and trusted only as far as its own measured agreement with human judgment. Teams that do not build this end up shipping on impressions and cannot tell an improvement from a regression.

So the first competency to hire for is evaluation discipline, and it separates candidates sharply. Ask what changed the last time they modified a prompt and how they knew it helped. The strong answer describes a fixed set of cases, a scoring method with known limitations, and a number that moved. The weak answer describes trying it and it seeming better. Both candidates have shipped similar demos.

The second is failure decomposition. Any system that chains retrieval, a model call, a tool invocation and a formatting step will fail in ways that are attributable only if it was built to be. A strong engineer instruments each hop, can say whether last week's bad output came from the retriever surfacing the wrong document or the model ignoring the right one, and designs the boundary so those two are distinguishable. Without that, debugging degrades into rewriting the prompt and hoping — which is where a large share of production generative AI work currently sits.

The third is economic and adversarial realism. Latency and cost per request are product constraints in this role in a way they rarely are elsewhere: an architecture that is correct and takes eleven seconds is not shippable, and one that costs forty cents a call may not survive contact with the business model. In parallel, any system that reads untrusted content and can call a tool has a security surface most engineers have never had to reason about, and the candidate who has never considered that a retrieved document might contain instructions is not ready to own one.

The assessment implication is unusual and worth stating to buyers directly: this is a role where the candidate should be permitted to use AI assistance during the exercise, because withholding it tests something nobody is hiring for. What makes the exercise discriminating is not restriction but observation and follow-up — give them an existing system that is quietly failing on a subset of inputs, watch how they go about establishing that, and then ask them to defend their diagnosis out loud. Generation is free; attribution of failure is not, and it is the whole job.

What the job actually needs

How people fail in this seat

What most employers do instead

CV screen for framework names, an unmonitored take-home building a small retrieval or agent demo, and an interview about model architectures borrowed from machine learning hiring.

This is the one role where a candidate can correctly argue that using AI is the job, which makes the standard integrity posture incoherent — Karat's 2026 survey of 400 engineering leaders found 62 percent of organisations still prohibit AI use in technical interviews while over half of candidates use it anyway. Banning the tool here tests compliance, not skill. The deeper failure is that a take-home demo is graded on whether it works once, and this role's entire value is knowing whether it works reliably. Building a plausible retrieval pipeline is now a solved, assisted task; constructing the eval set that proves the last change helped, and the failure taxonomy that says which of five chained steps broke, is not — and no part of the standard screen asks for either. The honest limit: a bounded exercise reads evaluation discipline and failure decomposition well, and predicts little about whether someone can hold quality on a system whose underlying model is swapped out from under them twice a year.

The assessment

About 80 minutes end to end.

The systems it runs in

A Python service the candidate did not write, calling a hosted model over an API, retrieving from a pgvector store, and invoking a pricing lookup tool. The two hundred interactions are supplied as traces rather than as a log file, and the trace schema is the design. Each one carries the user message, the retrieved chunk ids, the exact span assembled into the prompt, the tool call and its result, and the thumbs-down where there is one — so the distinction the whole assessment turns on is available to anybody who opens one level deeper than the retrieval result. The right document was returned and the answering row is not inside the span that was passed: both facts are in the same trace, one field apart, and the sandbox records whether the candidate ever expanded the second one. There is no evaluation harness in the fixture. Building one is the second segment, and the absence is deliberate, because a supplied harness would hand over the measurement that is being assessed.

Genuinely fragmented, and the file will not manufacture a default it cannot support. Anthropic's, OpenAI's or Google's API, or open weights served locally, in the model slot; LangChain, LlamaIndex, LangGraph, a typed framework or the vendor SDK called directly in the orchestration slot; Pinecone, Weaviate, Qdrant or Elasticsearch in place of pgvector; Langfuse, LangSmith, Braintrust, Ragas or promptfoo where a buyer already has tracing and evaluation tooling, in which case the fixture ships against theirs. This is the newest title in the corpus and the least standardised, and a rubric that assumed a framework would be measuring which employer a candidate happened to work for. What cannot be swapped away is that the traces record the assembled span separately from the retrieved document, because a fixture that only logs which document was returned cannot distinguish the two candidates this design exists to separate.

Working speed is not scored. Not scored, and this is the fixture where the fast path is most convincingly productive. Editing the prompt is available in the first two minutes, is the first thing an assistant proposes, and works — a firmer instruction converts confabulated prices into refusals, so previously failing cases start passing and the candidate finishes with visible improvement and no idea what the failure rate is. Establishing a baseline before touching anything produces nothing to demonstrate for a third of the session. A speed criterion would rank the prompt edit first, which is exactly the ranking the scoring section already refuses by annotating the count of newly passing cases as not a positive signal. The two timings that are reported — time to first quantified measurement and time to first prompt edit — are a pair whose order is the finding, and the order is destroyed by turning either into a rate.

What the candidate actually does

TaskWhat happens
It is wrong sometimes and nobody knows how often
coding_sandbox · 40 min		A working system the candidate did not build — a support assistant that retrieves from a document set, calls a model, invokes a pricing lookup tool, and formats an answer. There is no evaluation harness. There are two hundred logged interactions, held as traces rather than as flat log lines: the user message, the retrieved chunk ids, the exact span assembled into the prompt, the tool call, the model's output and, on about a third of them, a thumbs-down. The ticket says customers are complaining that answers are sometimes wrong and asks the candidate to fix it. The dominant failure class is a chunking boundary: the document set contains pricing tables, the chunker splits them mid-table, so the retriever surfaces the correct document and the span handed to the model does not contain the row that answers the question. On the surface this presents as the model making things up. AI assistance is permitted and expected in this role, and is logged.
The fork. Rewriting the prompt is the first thing anybody suggests, the first thing an assistant suggests, and it works — several of the visibly bad cases start passing, because a firmer instruction not to invent prices makes the model refuse instead of confabulate. The candidate can end the session with a demonstrably improved prompt, a set of previously failing cases now passing, and no idea what the failure rate is or what caused it. The class is untouched: the answer still is not in the retrieved span, so the system now declines to answer a third of pricing questions instead of answering them wrongly, and nobody finds out for a month.
The eval set you would keep
written_artifact · 10 min		The cases the candidate would keep as a permanent regression set, how each is scored — exact match, a heuristic, a model-graded rubric — and, for each scoring method, what it is known to get wrong. Two or three sentences per method, not an essay.
The fork. A set of cases with exact-match answers is quick to write and easy to score and covers only the questions with short factual answers, which is the part of the traffic that was never failing. Including the cases that are hard to grade means committing to a grader whose own accuracy has to be described.
Three things about the system that are not about quality
judgment_scenario · 10 min		Three short cases. A document in the retrievable set contains text addressed to the assistant, instructing it to disclose an internal discount schedule. The pricing tool's credentials can write as well as read, and the task only ever needs to read. And a stated architecture, with a stated cost per call and a stated volume, against a stated revenue per customer. For each, the candidate says what they would do and in what order.
The fork. All three are easy to acknowledge in principle and easy to defer in practice, and the brief gives the candidate a quality problem to work on instead. The third case has arithmetic in it that a candidate can do or can wave at, and doing it changes the answer.
Defend the diagnosis
live_call · 20 min		A spoken conversation over the candidate's own work. How many of the two hundred were actually bad, and how do you know that number? For the last change you made, what got better and what is your evidence? The vendor swaps the underlying model next month — which of the cases in your regression set still mean anything, and which were measuring the old model's habits? If the retrieval had been fine and the model had been ignoring a correct span, what in your setup would have told you the difference?
The fork. A candidate who improved the prompt without measuring can describe the improvement fluently and cannot answer the first question, because the number was never established. The model-swap question separates a regression set built from real traffic from one built to pass.

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Establishes the failure rate before changing anythingweight 0.25Produces a baseline before touching the system — how many of the two hundred are wrong, by a stated criterion, with the ambiguous ones counted somewhe…
1 Opens the prompt within the first few minutes and edits it. The thumbs- down cases are read as examples rather than counted, and no rate is ever produced.
3 Reads through the logs systematically and forms a qualitative view of what is going wrong before making changes, without producing a number or a repeatable way of producing one.
5 Produces a baseline before touching the system — how many of the two hundred are wrong, by a stated criterion, with the ambiguous ones counted somewhere explicit — and can re-run it after a change.
Attributes the failure to a hopweight 0.25Inspects the span actually handed to the model rather than the document that was retrieved, identifies that the answering row is outside it, demonstra…
1 Concludes the model is hallucinating and fixes the prompt. Retrieval is never inspected.
3 Checks retrieval and confirms the right document is being returned, concluding that retrieval is working and the fault is downstream — stopping one level above the chunk boundary.
5 Inspects the span actually handed to the model rather than the document that was retrieved, identifies that the answering row is outside it, demonstrates it on specific logged cases, and separates that class from whatever else is in the remaining failures.
The evaluation set and its grader are honestweight 0.2Cases drawn to represent the traffic including the hard-to-grade ones, a stated scoring method for each, and a plain statement of each method's known …
1 No eval set, or a handful of cases chosen because they are easy to check, with no scoring method described.
3 A reasonable set of cases with a scoring method, described without any account of what that method gets wrong.
5 Cases drawn to represent the traffic including the hard-to-grade ones, a stated scoring method for each, and a plain statement of each method's known failure — what an exact match misses, where the heuristic is brittle, how far a model-graded rubric has been checked against human judgment and how far it has not.
Adversarial and economic realismweight 0.15Treats retrieved text as untrusted data rather than instruction and says what structural control follows, scopes the tool credential to what the task …
1 Treats retrieved content as trusted input. Sees no problem with a read-only task holding write credentials. Does not engage with the cost arithmetic.
3 Identifies the injected instruction as a problem and proposes a prompt-level defence, without addressing tool permissions; acknowledges cost without calculating anything.
5 Treats retrieved text as untrusted data rather than instruction and says what structural control follows, scopes the tool credential to what the task needs, and does the arithmetic on cost per call against revenue per customer and states the conclusion it forces.
Distinguishes what was verified from what was inferredweight 0.15Marks the boundary unprompted — this class I demonstrated on these cases, this one I suspect and did not have time to confirm, this number is a baseli…
1 Presents the diagnosis with uniform confidence. Cannot say which parts were demonstrated on specific cases and which were reasoned about.
3 Broadly accurate about their own work, with one or two inferences described at the same confidence as things they checked.
5 Marks the boundary unprompted — this class I demonstrated on these cases, this one I suspect and did not have time to confirm, this number is a baseline and this one is an estimate.

How it is scored

Weighted mean of five criteria, 1-5 against the anchors, each reported with the log excerpt, written line or transcript quotation behind it. Two raw numbers are reported beside the score and outside the weighted mean, per the sequence pattern: time from session start to the first quantified measurement of the failure rate, and time from session start to the first edit of the prompt. The sign and size of the gap is the most informative artefact this session produces, and it is the same observation the data engineering design makes about profiling before writing, applied to a different pair of actions. Also reported and annotated as not a positive signal: how many of the previously failing cases pass at the end of the session. In this fixture that number can be raised substantially by a prompt change that converts wrong answers into refusals, and a buyer reading it as the result would rank that submission first.

Integrity

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

Who decides

Recommended. The evaluation criterion in particular rewards a candidate for admitting what their grader gets wrong, which reads as weakness to an inattentive scorer and is the strongest signal in the artefact. A reviewer confirms or overrides each criterion with a written reason, and should read the baseline and the grader description before looking at how many cases pass.

What this does not measure

This is the newest title in the corpus and the least standardised, and a buyer should be told plainly that the assessment matches one common reading of it — building and maintaining a system around a hosted model — and not the others. It measures nothing about fine-tuning, training infrastructure, or model research; if the seat means that in your organisation, this is the wrong exercise and the machine learning engineer design is closer. It does not measure holding quality on a system whose underlying model is replaced by its vendor twice a year, which is the defining long-run difficulty of the role; the call asks a question about it, which reads intent and not track record. It does not measure product judgment about what should be built, nor latency engineering under real load. On integrity, this role is the one place where the usual posture would be incoherent: candidates can correctly argue that using AI is the job, so assistance is permitted and expected here, and the fixture is built so that assistance produces the plausible wrong remedy rather than the diagnosis. Fairness notes: the call is spoken and often in a second language, scored on content and never on fluency, accent, register or assertiveness; no criterion is named clarity or communication. The exercise assumes familiarity with a retrieval-and-tools architecture, which is recent enough that exposure varies with employer rather than with ability — reviewers should weigh the attribution reasoning over idiomatic familiarity with any particular framework, and the anchors are written accordingly. Offer captions, extra time, or the conversation in written form on request.

Almost everything that makes this role hard is invisible in the artefact it produces. Building a plausible retrieval-and-tools system is now an assisted, largely solved task; a demo that works once can be produced by most candidates and by most assistants. The value of the seat is entirely in knowing whether it works reliably, and that knowledge is made of two things the standard screen never asks for: an evaluation set that can return bad news, and a system built so that failures are attributable to a hop rather than to the system in general. So this design gives the candidate no building to do. It gives them a working system that is quietly wrong, and asks them to establish that.

The fork is the most reliably tempting one in this batch, because the wrong remedy genuinely works. The visible symptom is a model inventing prices. The obvious response is a firmer instruction, it is what an assistant proposes immediately, and it makes several of the failing cases pass — the model stops confabulating and starts declining. A candidate can finish the session with a demonstrably better prompt, a set of cases that flipped from bad to good, and a system that now refuses a third of pricing questions instead of answering them wrongly. Nobody finds out for a month. The actual cause is a chunk boundary that splits the pricing tables, so the retriever returns the right document and the span handed to the model does not contain the row. The distinction between "the right document was retrieved" and "the answer was inside the span that was passed" is one level of inspection deeper than most people go, and it is the whole task.

That is why the two raw timings are reported. Time to first measurement of the failure rate and time to first prompt edit are put in front of the buyer as a pair, outside the score, because the order of those two actions is the most predictive thing the session produces and it is invisible in any submitted artefact. It is the same observation this hub's data engineering design makes about profiling a source before writing against it, applied to a different pair of actions and for the same reason: two candidates can end the session with similar-looking work and be entirely different hires.

Integrity works differently here than anywhere else in the corpus, and it should be stated to buyers rather than buried. This is the one role where a candidate can correctly argue that using AI is the job, which makes a ban both unenforceable and slightly absurd. So assistance is permitted, expected and logged, and no inference about its use is drawn beyond what the log records. The design does not need one. The fixture is built so that assistance produces the plausible wrong answer confidently, and the control is a conversation about a number that either was measured or was not. "How many of the two hundred were actually bad, and how do you know?" cannot be answered by anything except having done it.

The model-swap question at the end of the call is the mid-band instrument: not a harder question, but a changed requirement that invalidates an assumption. A regression set assembled to make the current system look good is full of cases that encode this month's model's habits, and its author cannot say which of them would survive a vendor swap. A set drawn from real traffic and graded against what the customer needed can answer it immediately. That difference takes ninety seconds to observe and is the closest thing this design has to a direct sample of the long-run job.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

  1. Lightcast, The Stanford AI Index Report 2025, US job-postings analysis for calendar year 2024, https://lightcast.io/resources/research/stanford-ai-index-2025
  2. Lightcast, The Stanford AI Index Report 2026, US job-postings analysis for calendar year 2025, https://lightcast.io/resources/research/stanford-ai-index-2026
  3. Karat, Engineering Interview Trends 2026, survey of 400 engineering leaders in the US, India and China, published January 2026, https://karat.com/engineering-interview-trends-2026/

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample reportOr talk to us about this role