Get started

Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Machine learning and AI engineering · Senior level

How to assess a Machine Learning Engineer

The notebook take-home supplies a target variable, a metric and a split, which are the three decisions this role is paid to make, and then asks the candidate to do the part that is now automated. The theory interview rewards fluent recall of material that is cheaply retrievable and does not distinguish a candidate who has shipped something that degraded in production from one who has read extensively about models that did. Both formats are unmonitored generation tasks, which is the exact shape current models complete to a high standard. And there is a limit worth stating plainly rather than selling around: a timed simulation reads diagnostic judgment on a bounded, seeded problem very well, and it does not predict whether someone can carry an ambiguous eighteen-month research programme. For that, references from people who watched the work and a portfolio of decisions the candidate can be interrogated on remain necessary. A simulation should be used to remove the large population who cannot diagnose, not to certify the small population who can research.

The commercially useful machine learning engineer is defined by what they do before and after the model, not by the model. Before: constructing an evaluation set that can actually return bad news. After: knowing, from telemetry rather than from complaint, when the thing has stopped working. The middle part — selecting an architecture, tuning it, getting it to converge — is the part that has become cheap, and it is also the only part the standard hiring process observes.

Evaluation design is where this role is won. An honest eval set has to be representative of production traffic rather than of the training distribution, split in a way that respects time and entity boundaries rather than randomly, and instrumented so that the aggregate metric does not hide a failure concentrated in a segment that matters. Nearly every catastrophic production model traces to a violation of one of those: a random split on data where the same customer appears in both halves, a feature populated by the downstream process being predicted, a holdout drawn from a period before a behaviour change. These are not exotic mistakes. They are the ordinary ones, they validate beautifully, and they are invisible to a hiring process that supplies the split.

The second differentiator is metric scepticism. A strong candidate can tell you what their chosen metric fails to capture — that AUC says nothing about calibration when the downstream system uses the score as a probability, that accuracy is meaningless at the base rate in question, that the offline lift was measured on a population the model will never see because a rules engine filters traffic upstream. A weak candidate reports the metric.

The third is the willingness to conclude that a model is the wrong instrument. A large share of machine learning projects in ordinary companies should have been a heuristic, a lookup, or a change to a business process, and the engineer who says so early saves more money than the engineer who delivers a good model late. That conversation is spoken, it happens under mild social pressure from a sponsor who has already told the board about the AI initiative, and it is directly observable in a simulation.

What a simulation cannot do here is worth repeating to any buyer, because pretending otherwise damages the argument for the parts it does well. A forty-minute diagnostic exercise tells you whether someone recognises leakage, reads a learning curve honestly, and can defend their own reasoning when challenged. It tells you nothing about whether they can hold a research direction for a year and a half through negative results, which is the thing senior machine learning hiring is often actually for. Use the simulation as the screen and keep the deep-dive on the candidate's own past work as the decision; the two answer different questions and neither substitutes for the other.

What the job actually needs

How people fail in this seat

What most employers do instead

CV and publication screen, a Kaggle-style notebook take-home, a theory interview on bias, variance and model architectures, and a general software-engineering coding round borrowed from another family.

The notebook take-home supplies a target variable, a metric and a split, which are the three decisions this role is paid to make, and then asks the candidate to do the part that is now automated. The theory interview rewards fluent recall of material that is cheaply retrievable and does not distinguish a candidate who has shipped something that degraded in production from one who has read extensively about models that did. Both formats are unmonitored generation tasks, which is the exact shape current models complete to a high standard. And there is a limit worth stating plainly rather than selling around: a timed simulation reads diagnostic judgment on a bounded, seeded problem very well, and it does not predict whether someone can carry an ambiguous eighteen-month research programme. For that, references from people who watched the work and a portfolio of decisions the candidate can be interrogated on remain necessary. A simulation should be used to remove the large population who cannot diagnose, not to certify the small population who can research.

The assessment

About 85 minutes end to end.

The systems it runs in

Somebody else's repository, opened as a pull request the candidate reviews rather than a codebase they own. Inside it: a scikit-learn gradient-boosted ranking model for fraud review, an `evaluate.py` harness that runs unmodified and prints AUC 0.94, an MLflow run recording that figure with its parameters and its artefact, and a written description of the production system including the rules engine that filters the obvious cases before the model is ever called. That last item is a tooling decision and the design depends on it. Three of the four defects are in the repository — the train_test_split that is random over rows where GroupShuffleSplit on merchant was needed, the feature computed from a table the review process itself writes, and the ranking metric handed to a consumer who sets a threshold on the raw score. The fourth exists only in the gap between the harness and the paragraph, which is why the fixture has to contain a system description at all and why reviewing the code alone cannot reach it. The second segment is posted as pull-request comments in the order the candidate would post them, because ordering and blocking-versus-optional are what the proportion criterion reads.

XGBoost or LightGBM in place of the scikit-learn estimator, and a PyTorch build where a buyer's models are neural — PyTorch dominates new work and TensorFlow retains a large installed base, so this file names neither as the default and the choice changes no anchor, because the defects are in the evaluation rather than in the model. Weights and Biases or Neptune in place of MLflow; a GitLab merge request in place of a GitHub pull request. What cannot be swapped away is the artefact under review: this design is about a model and the evidence offered for it, which is precisely what separates it from the MLOps design in this hub, where the artefact is the system that keeps a model serving after everyone has agreed it was good.

Working speed is not scored. Not scored. The harness runs in the first minute and prints 0.94, and confirming it with a suggestion of hyperparameter tuning is a complete, collegial, defensible forty minutes that finds nothing. Every scored behaviour here is slower than that by construction: re-running the evaluation grouped by merchant, re-scoring on the filtered population, establishing when a column is populated, working out roughly how much of the 0.94 each defect accounts for. A speed criterion would rank the candidate who accepted the number first. The design also has a top answer that sometimes contains no number at all — "I could not construct a clean evaluation in the time I had, so I do not know what this model does" is correct here, and it is the slowest possible finish.

What the candidate actually does

TaskWhat happens
Audit somebody else's validated model
coding_sandbox · 40 min		The model is already built. A colleague — described in the brief as experienced, well-regarded and about to go on leave — has produced a ranking model for prioritising fraud review, an evaluation harness that runs, and a short write-up reporting AUC 0.94 and recommending release. The candidate is asked to look at it before it ships on Thursday. Four things are wrong and none are flagged. The split is random over rows, and the same merchant appears in both halves. One feature is computed from a table the review process itself writes, so it is populated after the outcome it helps predict. The evaluation set is drawn from all transactions, while in production a rules engine filters roughly the obvious cases out before the model ever sees them, so the measured lift is on a population the model will never be asked about. And the downstream queue consumes the score as a probability to set a review threshold, which AUC says nothing about. A strong candidate will not find all four.
The fork. The harness runs. It prints 0.94. The write-up is confident, competent and pleasant to read, and the colleague is senior and liked. Confirming the result and suggesting a round of hyperparameter tuning is a complete, defensible, collegial forty minutes of work that leaves the candidate looking constructive. An assistant asked to review this code finds style issues and does not find the rules-engine population problem, because that problem is not in the code — it is in the difference between the code and how the system is described in a paragraph of the brief.
The review you would send
written_artifact · 15 min		The comments the candidate would actually post on the pull request, in the order they would post them, before the colleague goes on leave. Not a report to management — a review a peer has to read and act on in two days.
The fork. Listing every defect at equal weight is thorough, is easy, and leaves a person with two days and no priority. Listing only the most exciting one is fast and leaves the others in the release. The senior behaviour is selection — and wording the blocking comment so that somebody who is not wrong about anything else can act on it.
It ships Thursday
live_call · 20 min		A live spoken call with an AI engineering director. The release is in the board deck, the colleague is on leave from tomorrow, and the fraud review team has already been told their queue will be prioritised by the model from next week. The director is reasonable, will accept a well-argued no, and will also accept a qualified yes and hold the candidate to it. The band instrument here is consequence and refusal: what do you refuse to ship, what do you let go, and what happens to the review team if you are wrong.
The fork. A qualified yes — ship it, monitor it, fix the split next sprint — is available, sounds mature, and is what a director in a hurry wants to hear. It is also sometimes the right answer, which is what makes this a fork rather than a trap: the discriminating thing is whether the candidate can say which defect they are willing to carry into production and why that one, rather than whether they say yes or no.
Four splits
judgment_scenario · 10 min		Four short descriptions of how an evaluation set was constructed, three lines each. A random split on data where the same customer recurs. A time-based split whose holdout period sits after a product change. A stratified split that balances the class but not the entity. One that is honest. For each the candidate says whether the reported number can be believed and what it would over- or under-state.
The fork. Three of the four are contaminated, which sets the trap in the honest one: a candidate who has inferred within the exercise that the expected answer is always "this is leaking" will condemn it too. Over-calling is scored as a failure in the same way under-calling is.

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Finds the contamination in the evaluationweight 0.25Finds the split contamination and at least one of the two harder defects, demonstrates the effect rather than asserting it — re-runs grouped by mercha…
1 Runs the harness, accepts the number, reviews the modelling code for style and structure. None of the four defects identified.
3 Finds the split problem, which is the one visible in the code, and misses the population mismatch, which is visible only by comparing the harness to the description of the production system.
5 Finds the split contamination and at least one of the two harder defects, demonstrates the effect rather than asserting it — re-runs grouped by merchant, or re-scores on the filtered population — and can say roughly how much of the 0.94 each accounts for.
Does not pass on a number they have reason to distrust flooredweight 0.2States plainly that the reported figure does not describe production performance, gives whatever corrected estimate they were able to produce with its…
1 Repeats 0.94 to the director, in the review or on the call, as the figure to expect in production — either having already identified a defect that undermines it, or without having checked anything that would support it. Reported on the face of the report as a finding rather than averaged into the score.
3 Qualifies the number verbally, without saying what it would be under an honest evaluation or that the honest figure is unknown.
5 States plainly that the reported figure does not describe production performance, gives whatever corrected estimate they were able to produce with its own caveats, and where they could not produce one says so rather than substituting a guess.
Metric scepticism about the decision being madeweight 0.2Identifies that a ranking metric says nothing about calibration when the consumer sets a threshold on the score, says what should be measured instead,…
1 Treats AUC as the question. Does not ask what the downstream queue does with the score.
3 Notes that AUC is not the only metric worth having and suggests adding others, without connecting the choice to how the review threshold is set.
5 Identifies that a ranking metric says nothing about calibration when the consumer sets a threshold on the score, says what should be measured instead, and names the capacity constraint the review team is actually operating under.
Proportion in the reviewweight 0.2Marks explicitly what blocks Thursday and what can follow, drops or labels the cosmetic comments, and words the blocking comment so a respected collea…
1 Approves. Alternatively, posts everything found at equal weight, including style points, with no indication of what blocks the release.
3 Identifies the serious defects and reports them alongside minor ones without separating blocking from optional, leaving the colleague to rank them.
5 Marks explicitly what blocks Thursday and what can follow, drops or labels the cosmetic comments, and words the blocking comment so a respected colleague with two days can act on it rather than defend against it.
Refusal, and consequence to the people downstreamweight 0.15States which defect they are prepared to carry into production and why that one, names what the review team would see if the model is worse than repor…
1 Agrees to the release with no conditions, or refuses without offering the director anything to do. Never mentions the review team.
3 Argues for a delay on technical grounds, correctly, without translating it into what the fraud review team would experience.
5 States which defect they are prepared to carry into production and why that one, names what the review team would see if the model is worse than reported, proposes what would have to be true to ship on Thursday anyway, and says what they would refuse under any timeline.

How it is scored

Weighted mean of five criteria, 1-5 against the anchors, each reported with the code excerpt, review comment or transcript line behind it. The second criterion is floored — a score of 1 is reported on the face of the report and never averaged away — because passing a validation figure onward as a production expectation, having seen or not looked for the reasons it is not one, is a claim about evidence rather than a lapse of technique, and a candidate who audits well and then does it must not surface as strong. Reported beside the score and outside the mean: how many of the four defects were found and which, as a plain list rather than a count folded into a criterion, since a candidate who finds two hard ones is stronger than one who finds three easy ones. Also reported and annotated as not a positive signal: whether the candidate recommended release.

Integrity

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

Who decides

Required rather than recommended. The proportion criterion is a judgment about which comments were worth making in the time available, and it can only be made by somebody who would have to live with the resulting review culture. The refusal criterion has a legitimately arguable answer — some organisations should ship this on Thursday with monitoring — and a reviewer disagreeing with the anchor is useful information about the hiring team rather than a defect in the design. A reviewer confirms or overrides each criterion with a written reason.

What this does not measure

The family this role sits in is rated a medium fit for a reason that has to be said plainly rather than sold around, and it applies most sharply here. This design reads diagnostic judgment on a bounded, seeded problem very well. It does not predict whether somebody can hold a research direction for eighteen months through negative results, which is what a large share of senior machine learning hiring is actually for. Nothing in eighty-five minutes reaches that, and no simulation should be sold as if it does — use this to remove the substantial population who cannot audit an evaluation, and keep references from people who watched the work and a deep-dive on the candidate's own past decisions as the deciding instrument. This is also why the corpus contains no applied-scientist or research-engineer page: those seats are defined almost entirely by the thing a session cannot observe, and writing an assessment for them would have been a claim we cannot support. More narrowly, this design does not measure training at scale, distributed systems work, model architecture research, or the engineering quality of production serving code. Fairness notes: the sandbox is Python and the harness is a conventional one, which advantages candidates from that idiom for reasons unrelated to judgment, so timings are reported as context and never scored. The call is spoken and frequently in a second language, and every criterion is scored on the content of the argument, never on fluency, accent, register or assertiveness; no criterion here is named clarity or communication. The refusal criterion carries a cultural and seniority risk, because blocking a respected colleague's release in front of a director is not equally safe for everyone — reviewers should credit the objection being made in any form, including only in the written review, rather than looking for a confrontational style. Offer captions, extra time, or the call in written form on request.

The band change from mid to senior in machine learning is not a change in difficulty, and the standard mistake is to administer a harder version of the mid-level exercise — a bigger dataset, a nastier target, less time. A senior machine learning engineer is not paid to produce models faster. They are paid to be the person who can look at a result somebody else is pleased with and work out whether it is real. So this design does not ask the candidate to build anything. It gives them a finished model, a working harness, a confident write-up and a release date, and asks what they think.

Four things are wrong with it and they are graded in ascending order of what they reveal. The random split with a recurring merchant on both sides is in the code and a careful mid-level engineer finds it. The feature written by the review process is one layer further out and requires asking when a column is populated rather than what it contains. The population mismatch is the one that separates the field: in production a rules engine removes the obvious cases before the model sees anything, so the measured lift describes a population the model will never be asked about — and that defect is not in the code at all. It exists only in the difference between what the harness does and what a paragraph of the brief says the system does, which is why an assistant asked to review this code does not find it and why a candidate who only reads code does not either. The fourth, using a ranking metric where the consumer sets a threshold on the score, is a question about the decision rather than about the model.

Nobody is expected to find all four, and the rubric is written so that finding two hard ones beats finding three easy ones. The defects found are reported as a plain list beside the score rather than as a count, precisely so that a buyer can see which ones, because which ones is the information.

The floored criterion is not about competence. A candidate can miss defects honestly; that is what the first criterion is for. What is floored is passing the 0.94 onward to a director as the number to expect in production — either having already found a reason it is not, or without having checked anything that would make it one. That is a claim about evidence, and this corpus treats a false claim about evidence the way its regulated designs treat a withheld material term: reported on the face of the report, never averaged away by strong scores elsewhere. It is worth saying that the honest top answer here sometimes contains no number at all. A candidate who says "I could not construct a clean evaluation in the time I had, so I do not know what this model does, and that is the finding" is right, and a mechanical ranking would place a confident corrected figure above them. Human review is required here for exactly that reason.

The call is the senior instrument, and it deliberately does not have a correct answer in the form buyers expect. Shipping on Thursday with monitoring and a committed fix is a legitimate decision for some organisations; blocking is legitimate for others. What is scored is neither. It is whether the candidate can say which specific defect they are willing to carry into production and why that one rather than another, and whether they can describe what the fraud review team would experience if the model is worse than advertised — the consequence to other people, which is the thing the band is actually hired for and the thing no algorithm interview has ever observed.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

  1. US Bureau of Labor Statistics, Occupational Outlook Handbook, Computer and Information Research Scientists, 2025, https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm
  2. Lightcast, The Stanford AI Index Report 2026, US job-postings analysis for calendar year 2025, https://lightcast.io/resources/research/stanford-ai-index-2026
  3. Lightcast, The Stanford AI Index Report 2025, US job-postings analysis for calendar year 2024, https://lightcast.io/resources/research/stanford-ai-index-2025

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample reportOr talk to us about this role