Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Machine learning and AI engineering · Senior level

How to assess a MLOps Engineer

The whiteboard architecture question is answered from a reference diagram, and reference diagrams are exactly what a model produces best; a candidate can sketch a defensible platform without ever having operated one. Worse, the question is oriented toward building the platform, and this seat's actual daily work is detecting that something already built has gone quietly wrong. A generic DevOps interview covers the half of the job that transfers from infrastructure work and misses the half that does not: a healthy service serving degraded predictions has green dashboards, passing health checks and no alerts, because nothing in conventional observability is watching the distribution of what goes in or comes out. Nothing in the standard screen asks a candidate to notice that.

MLOps is the most assessable role in this family, which is worth saying because the family as a whole is rated a medium fit. The reason is that its failures are concrete, bounded and reproducible in a sandbox in a way that a research trajectory is not. A model is in production. Its predictions have degraded. The service is healthy, the latency is fine, no alert has fired, and the cause is one of a small number of knowable things: the upstream feature pipeline changed a default, the training and serving code compute a feature differently, the input distribution has moved away from what the model was trained on, a categorical encoding met an unseen value, or the wrong artefact version is loaded. Working through that list methodically is the job, and it is directly observable.

The distinctive competency is monitoring the right layer. Infrastructure engineers coming into this seat instrument the service — uptime, error rate, p99 — and are surprised that a model can be silently useless while every one of those is green. The MLOps question is what the model is receiving and emitting: the distribution of each input feature against its training baseline, the distribution of scores, the rate of unseen categories, the fraction of requests falling back to a default. A candidate who reaches for prediction-distribution monitoring unprompted has done this work; a candidate who reaches for CPU utilisation has done adjacent work.

Training-serving skew is the second, and it is a code-reading problem more than a design one. The feature computed in a batch job over a full history and the feature computed in a request handler from a partial context are written by different people at different times, and the moment they diverge the model is being asked questions in a dialect it was not trained on. Finding that divergence by reading two implementations side by side is a bounded, timed, entirely fair exercise, and it is close to the highest-value thing this seat does.

Reproducibility is the third. Asked to reproduce the model currently serving traffic, a strong engineer can name what would have to be pinned — data snapshot, feature code version, library versions, random seed, environment — and can say honestly which of those their current setup does not capture. This is a governance requirement in regulated settings and a debugging requirement everywhere.

The limit still applies, though it binds less tightly here than for the research end of the family. A simulation can establish that someone diagnoses a degraded model competently, monitors the right signals, and reasons carefully about rollout and rollback. It cannot establish that they will make good platform choices whose consequences land two years out, or that they can persuade a sceptical engineering organisation to adopt a shared workflow — which is a large part of what senior MLOps hires are actually for. Assess the diagnosis, and interview separately for the adoption problem rather than pretending one covers the other.

What the job actually needs

reproducibility of training and serving
detecting training-serving skew
monitoring for data drift rather than only uptime
safe rollout and rollback of a model
making an experiment someone else ran repeatable

How people fail in this seat

monitors the service and not the predictions
allows feature computation to differ between training and serving
cannot reproduce the model currently in production
builds a rollback path that has never been exercised
treats a model artefact as a deployment rather than as a dependency with a lifecycle

What most employers do instead

CV screen for platform keywords (Kubernetes, MLflow, Kubeflow, Terraform, SageMaker), a generic DevOps interview, and sometimes a system-design whiteboard on drawing an ML platform architecture.

The assessment

About 75 minutes end to end.

The systems it runs in

A running system rather than a model, which is the line between this design and the machine learning engineer's one in the same hub. A ranking model served by a request handler behind Kubernetes, registered in the MLflow Model Registry with its version and lineage, its features materialised through a Feast feature store, trained by a batch job that lives in a separate repository, deployed through a pipeline whose run history is readable, and monitored by Grafana dashboards over Prometheus that are all green. The green dashboards are the fixture, not the scenery: the monitoring the buyer already owns is present, correctly configured, and structurally unable to see this incident, which is why the written segment can refuse the answer "add observability" — the observability is already there. The skew is found by opening the training job's feature code and the request handler's feature code side by side across two repositories, and the file-open order across both is the single most diagnostic record the session produces. The decoy is a routine library version bump sitting in the deploy history an hour before the degradation window, which is a coincidence and looks exactly like a cause.

Kubernetes
MLflow Model Registry
Feast
Prometheus
Grafana
a deploy history with library version bumps
separate training and serving repositories

Amazon SageMaker or Vertex AI where a buyer runs a managed platform end to end; KServe, Seldon or BentoML in place of the plain serving handler; Tecton or a warehouse-native feature store in place of Feast; Argo CD, GitHub Actions or GitLab CI as the deploy surface; Evidently, Arize or WhyLabs where drift monitoring already exists, in which case the fixture is rebuilt with those dashboards green too, because a drift monitor watching the wrong feature is a sharper version of the same problem. MLflow is the default because it is the common open-source registry rather than because it is universal; Weights and Biases or a cloud provider's registry answers the reproducibility criterion identically. What cannot be swapped away is the two-repository split, since a fixture where both implementations of the feature sit in one file is not this job.

Working speed is not scored. Not scored, and this file is the clearest case in the hub. Rolling back the library version bump is the fastest action available, is what an incident channel rewards at minute four, is what an assistant proposes when handed a deploy history and a time window, and changes nothing — the model keeps serving in a dialect it was not trained on and the candidate has an action to report. Reading two feature implementations side by side is slow and is the entire competency. The fourth judgment case makes the opposite point in the same breath: a genuinely low-risk hotfix should just go out, and an MLOps hire who wraps ceremony around it is the reason teams route around the platform, so neither haste nor caution is scored on its own. Time to first mitigating action and time to first correct causal statement are reported raw and outside the mean, and the gap between them — not either number — is the artefact.

What the candidate actually does

Task	What happens
Every dashboard is green coding_sandbox · 35 min	A recommendation model in production. The business has noticed that its quality fell off roughly ten days ago. Uptime is fine, latency is fine, the error rate is flat, no alert has fired and the health checks pass. The candidate has the serving repository, the training repository, a feature store, a week of request and response logs, the deploy history and the dashboards. The cause is training-serving skew: the training job computes a recency feature over the full order history in a batch, and the request handler computes the same feature from a thirty-day context window, so every long-dormant customer arrives at the model looking like a thirty-day-dormant one. It is found by reading two implementations side by side. A decoy sits in the deploy history — a routine library version bump that landed about an hour before the degradation window opens, which is coincidence. The fork. Rolling back the version bump is fast, safe-sounding, defensible, and exactly what an incident channel wants to see somebody doing at minute four. It is also what an assistant proposes when given a deploy history and a degradation window, because the correlation is right there and the skew is not in any single file. The rollback changes nothing, the candidate has an action to report, and the model keeps serving in a dialect it was not trained on.
What would have caught this written_artifact · 12 min	The monitoring the candidate would add so this is detected next time. Specific signals with a stated comparison baseline and a stated threshold behaviour, not a list of tools and not the word observability. The fork. A list of everything worth monitoring is easy to write, reads as thorough, and produces an alert estate nobody acts on. The useful answer names the two or three signals that would have fired in this specific case and says what they are compared against, which requires having understood the cause rather than the category.
Reproduce what is serving, and what you would refuse live_call · 20 min	A spoken conversation over the candidate's own diagnosis and this system. Reproduce the model currently serving traffic — what has to be pinned, and which of those does this repository not capture? Walk me through your rollout for the fix: what is live between step one and step three, and how do you find out you are wrong? It is Friday afternoon and the business wants the corrected model out today — what do you do, and what would you refuse under any timeline? The band instrument here is consequence and refusal rather than depth. The fork. A confident account of reproducibility is available to anyone who has read about it, and the discriminating move is naming what this particular setup does not pin — which requires having looked, and which makes the candidate's own answer weaker.
Four rollouts judgment_scenario · 8 min	Four short cases, three lines each. A retrained model with better offline metrics and no online evidence. A rollback path that exists and has never been exercised. A model whose new version changes the score distribution without changing the ranking, feeding a downstream threshold. A hotfix that is genuinely low risk and should just go out. For each the candidate says how they would put it live. The fork. Three of the four warrant caution, which sets the trap in the fourth: a candidate who has inferred that the expected answer is always a staged rollout will wrap ceremony around a change that does not need it, and an MLOps hire who does that to every deploy becomes the reason teams route around the platform.

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Monitors and inspects the right layerweight 0.25Goes to the inputs and outputs first — the distribution of each feature against what training saw, the distribution of scores, the rate of defaulted o…

1 Investigates the service. CPU, memory, latency percentiles, error rates, pod restarts. Concludes the system is healthy and the problem must be upstream of engineering.

3 Looks at the request and response logs and notices the predictions have changed character, without comparing any input feature against its training baseline.

5 Goes to the inputs and outputs first — the distribution of each feature against what training saw, the distribution of scores, the rate of defaulted or unseen values — and uses the comparison to localise the problem to a specific feature before reading any code.

Finds the skew by reading both implementationsweight 0.25Reads the two implementations side by side, states the exact divergence and which population it distorts — long-dormant customers clamped to the windo…

1 Never opens both. Diagnoses from the deploy history, the logs, or the model artefact alone.

3 Opens both repositories and identifies that the feature is computed in two places, without establishing how the two differ or which customers it affects.

5 Reads the two implementations side by side, states the exact divergence and which population it distorts — long-dormant customers clamped to the window — and demonstrates it against a specific request in the logs.

Sequences mitigation and diagnosis deliberatelyweight 0.15Says explicitly which actions are mitigation and which are diagnosis, can justify the order chosen, and either explains why the version bump is a coin…

1 Rolls back the library version, reports it as the fix, and stops. Alternatively, makes no attempt to limit the damage while investigating for the full session.

3 Investigates carefully without considering whether anything should be done for users in the meantime, or acts first and then reasons, without saying which they were doing.

5 Says explicitly which actions are mitigation and which are diagnosis, can justify the order chosen, and either explains why the version bump is a coincidence or tests it cheaply and sets it aside with a reason.

Reproducibility honestyweight 0.2Names what is pinned here and what is not, having looked, and says what that means in practice — whether the artefact currently serving traffic could …

1 Says the model can be reproduced from the training repository. Does not distinguish code from data, environment or seed.

3 Lists what would need pinning — data snapshot, feature code version, library versions, seed, environment — as a general account, without checking which of them this repository actually captures.

5 Names what is pinned here and what is not, having looked, and says what that means in practice — whether the artefact currently serving traffic could be rebuilt at all, and what evidence would be needed to be sure it was the same model.

Rollout, rollback and refusalweight 0.15States what fraction of traffic sees what at each step, names the signal that halts the rollout and who watches it, says whether the rollback path has…

1 Ships the corrected model directly because it is a fix, or applies the same heavy ceremony to all four judgment cases including the one that does not need it.

3 Proposes a staged rollout in general terms without saying what is live at each stage, what signal would stop it, or what happens to the downstream threshold when a score distribution moves.

5 States what fraction of traffic sees what at each step, names the signal that halts the rollout and who watches it, says whether the rollback path has ever actually been exercised, distinguishes the case that needs none of this, and names something they would refuse regardless of the timeline.

How it is scored

Weighted mean of five criteria, 1-5 against the anchors, each reported with the code excerpt, log line, written line or transcript quotation behind it. Two raw timings are reported beside the score and outside the weighted mean, per the sequence pattern, and this fixture is the pattern's canonical case: time to first mitigating action, and time to first correct causal statement. The gap between them is the most informative artefact the session produces. It separates the candidate who limited the damage and then found the cause from the candidate who guessed correctly and never acted, and both from the candidate who rolled back everything within four minutes and learned nothing — three profiles that can otherwise finish with similar-looking incident notes. Also reported and annotated as not a positive signal: whether the candidate performed any action that made the metric recover during the session.

Integrity

Monitored sandbox with file-open order across both repositories, which is the single most diagnostic record this session produces
Full command and query log in execution order, with timestamps against the incident timeline
Edit timeline and paste-versus-typed provenance
AI assistance permitted and logged; the fixture is built so that assistance proposes the rollback confidently, because the skew is not visible in any one file
Same-day spoken conversation on the candidate's own diagnosis, with both implementations on screen
No automated cheating verdict is produced. What is recorded is which files were opened and when, and what was said afterwards about a divergence that either was read or was not.

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

File-open order across both repositories with timestamps
Full command and query log against the incident timeline
Any comparison or check the candidate ran, with its output
The monitoring proposal as submitted
The four rollout judgments as written
Call recording and transcript
Time to first mitigation and time to first correct causal statement, reported raw

Who decides

Recommended. The sequencing criterion depends on what a particular organisation expects during a quality degradation that is not an outage, and reviewers will reasonably differ on how much mitigation should precede diagnosis when nothing is down. A reviewer confirms or overrides each criterion with a written reason and should read the file-open order before the score, because it is the record that distinguishes a diagnosis from a good guess.

What this does not measure

MLOps is the most assessable seat in a family rated medium fit, and the reason is that its failures are concrete, bounded and reproducible in a sandbox in a way a research trajectory is not — so the family's caveat binds less tightly here, but it still binds. This design establishes that somebody diagnoses a degraded model competently, monitors the right signals and reasons carefully about rollout. It does not establish that they will make good platform choices whose consequences land two years out, and it does not touch the adoption problem, which is most of what a senior MLOps hire is actually for: persuading a sceptical engineering organisation to use a shared workflow is a political and pedagogical job that no session observes. Interview separately for it rather than reading it into this score. The design also does not measure cost and capacity engineering, GPU scheduling, or building a platform from nothing; it measures operating one that exists. Fairness notes: the fixture is Python and a conventional serving stack, which advantages candidates from that idiom for reasons unrelated to judgment, so timings are reported as context and never scored. The call is spoken and frequently in a second language, and every criterion is scored on the content of the reasoning, never on fluency, accent, register or assertiveness; no criterion here is named clarity or communication. The refusal criterion carries a cultural and seniority risk, since declining a Friday release request is not equally safe for everyone, and reviewers should credit the risk being surfaced in any form rather than looking for a confident refusal. Offer captions, extra time, or the conversation in written form on request.

The characteristic MLOps incident is one where nothing is broken. The service is up, latency is fine, no alert has fired, every health check passes, and the model has been quietly useless for ten days. Conventional observability cannot see it, because nothing in conventional observability watches the distribution of what goes into a model or comes out of it. That is the entire seat, and it is why a generic DevOps interview covers the half of this job that transfers from infrastructure work and misses the half that does not.

So the fixture is an incident with green dashboards, and the first scored behaviour is where the candidate looks. An infrastructure engineer moving into this seat instruments the service and is surprised there is nothing there. The MLOps question is what the model is receiving and emitting: each input feature against its training baseline, the distribution of scores, the rate of unseen categories or defaulted values. A candidate who reaches for prediction and input distributions unprompted has done this work. A candidate who reaches for CPU utilisation has done adjacent work, and the difference shows up in the first four minutes of the file-open log.

The cause is training-serving skew, and it is deliberately a code-reading problem rather than a design one. The recency feature is computed over full history in a batch job and from a thirty-day context in the request handler, so every long-dormant customer arrives at the model looking recently active. It is not subtle once you have both files open next to each other, and it is invisible in any one of them — which is exactly the property that makes it the right fixture. Nothing about the serving code looks wrong. Nothing about the training code looks wrong. Finding the divergence by reading two implementations side by side is a bounded, timed, entirely fair exercise, and it is close to the highest-value thing this seat does.

The decoy is doing real work in this design. A routine library version bump landed about an hour before the degradation window opens, and it is a coincidence. Rolling it back is fast, feels responsible, is what an incident channel rewards at minute four, and is what an assistant proposes when handed a deploy history and a time window — because the correlation is explicit in the data and the skew is not in any file. The rollback changes nothing and gives the candidate an action to report. This is why the two timings are reported as a pair and outside the score: time to first mitigating action and time to first correct causal statement. The candidate who mitigated at minute three and named the skew at minute twenty-two behaved differently from the one who named it at minute twenty-two having done nothing, and differently again from the one who rolled back at minute four and never found anything — and all three can write similar-sounding incident notes.

The call takes the senior band's instrument, which is consequence and refusal rather than difficulty. The reproducibility question is the sharpest of the three, because a confident general account of what should be pinned is available to anyone who has read about it, and the discriminating answer is naming what this particular repository does not capture. That requires having looked, and it makes the candidate's own position weaker, which is the shape of every honest answer in this design. The fourth judgment case is the corresponding restraint check: it is a genuinely low-risk hotfix that should simply go out, placed after three cases that warrant caution, because an MLOps hire who wraps ceremony around every change is the reason engineering teams start routing around the platform they were hired to run.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

Lightcast, The Stanford AI Index Report 2026, US job-postings analysis for calendar year 2025, https://lightcast.io/resources/research/stanford-ai-index-2026
US Bureau of Labor Statistics, Occupational Outlook Handbook, Computer and Information Research Scientists, 2025, https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample report Or talk to us about this role