Get started

Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Data and analysis · Mid level

How to assess a Data Scientist

The notebook take-home is the most thoroughly automated artefact in technical hiring — load, clean, split, fit, report a metric — and it was a weak signal even before that was true, because it hands the candidate a defined target variable and a metric to optimise. Both of those are the job. Deciding what to predict, whether prediction is the right frame at all, and what the chosen metric fails to capture are the decisions a data scientist is actually paid for, and the standard exercise makes all three for them. The theory interview compounds it by rewarding fluent recall of concepts that are now cheaply retrievable, while leaving untested the behaviour that actually separates candidates: noticing that a feature could not have been available at prediction time.

The data scientist is hired for methodological judgment, and methodological judgment shows up mostly as restraint. The commercially valuable version of this person spends a meaningful share of their time talking a stakeholder out of a model: the question can be settled by a segmentation, the historical data does not support the counterfactual being asked for, the effect being chased is smaller than the measurement error, a two-week holdout would answer it more credibly than six weeks of feature engineering. Organisations that hire on modelling ability alone get someone who always finds a modelling answer, and the cost is not a bad model — it is a year of effort pointed at a question that did not need one.

Where the role does build models, the separating skill is leakage detection. Nearly every catastrophic data science result in production traces to information in the training data that would not have existed at prediction time: a field populated by the downstream process being predicted, a timestamp that encodes the outcome, a customer status updated after the event. The model validates beautifully, deploys, and performs at chance. Distinguishing candidates who reflexively ask "when was this column written?" from candidates who do not is worth more than any comparison of their algorithm knowledge, and it is a question of habit rather than knowledge, so it can only be observed on a dataset that actually contains the trap.

The third axis is honest communication of uncertainty upward. A data scientist reports to people who want a number, not an interval, and who will treat any interval offered as evidence of hedging. The behaviour worth hiring is being able to say "the central estimate is a nine percent lift, and I would not be surprised by four or by fourteen; here is the decision that changes if it is four" — turning uncertainty into something actionable rather than either hiding it or hiding behind it.

This is also the role where the family's honest limit sits closest to the surface. A timed exercise reads diagnostic judgment on a supplied dataset well. It does not tell you whether someone can carry an ambiguous six-month research question with no defined target, and any assessment sold as if it does is overreaching. What it can do is remove the large population of candidates who can produce a fitted model and cannot tell you what would make it wrong — and in a category growing 35 percent with almost no internal benchmark to hire against, that filter is worth a great deal on its own.

The boundary with the data analyst is one of mandate, not seniority. An analyst answers a question that has been asked. A data scientist is more often given a decision and expected to work out which question would inform it, including the possibility that the answer is available without new analysis.

What the job actually needs

How people fail in this seat

What most employers do instead

CV and publication screen, an unmonitored notebook take-home on a Kaggle-style dataset, and an interview covering statistics and machine-learning theory.

The notebook take-home is the most thoroughly automated artefact in technical hiring — load, clean, split, fit, report a metric — and it was a weak signal even before that was true, because it hands the candidate a defined target variable and a metric to optimise. Both of those are the job. Deciding what to predict, whether prediction is the right frame at all, and what the chosen metric fails to capture are the decisions a data scientist is actually paid for, and the standard exercise makes all three for them. The theory interview compounds it by rewarding fluent recall of concepts that are now cheaply retrievable, while leaving untested the behaviour that actually separates candidates: noticing that a feature could not have been available at prediction time.

The assessment

About 70 minutes end to end.

The systems it runs in

A JupyterLab notebook with pandas and scikit-learn over the churn table, and a supplied evaluation cell that fits, scores and prints AUC in large type — deliberately the exact shape of the take-home this design is arguing against, because the argument only lands if the trap is the familiar one. Each of the three defects is placed where a specific tool makes it findable to somebody who goes looking and invisible to somebody who does not. The leaked field is discoverable from its population pattern in a two-line value_counts against the target, and from a data dictionary entry naming the offboarding process that writes it. The split is a plain train_test_split with a fixed random_state, and the account identifier is sitting in the frame ready to be grouped on, so the correct fix is a GroupShuffleSplit the candidate has to reach for rather than a technique they have to invent. And the target is not a given at all — the SQL that built `churned_next_quarter` is a committed file in the same repository, which nothing in the brief points at.

Databricks or Colab notebooks, or VS Code, in place of JupyterLab; polars in place of pandas; XGBoost or LightGBM in place of the default estimator, none of which changes a single anchor. An R and tidymodels build of the same fixture is available where a buyer's team genuinely works that way, and where it is not taken the fairness note in adverse_impact applies as written. What cannot be swapped away is that the target, the metric and the split all arrive supplied and all three are wrong, because those three supplied decisions are the job and the conventional exercise makes them before the candidate opens the file.

Working speed is not scored. Not scored, and this fixture is close to a proof of why it should not be. Fitting the model exactly as specified takes about fifteen minutes, returns an AUC around 0.97, prints a leaderboard-style figure the sandbox displays prominently, and is contaminated. Every one of the three correct behaviours — checking when each column was written before fitting anything, splitting on time and account, going and reading how the target was constructed — costs minutes that produce no visible deliverable. A speed criterion would rank the fifteen-minute contaminated submission above the honest one, which is the same inversion the scoring section already guards against by annotating the final validation figure as not a positive signal.

What the candidate actually does

TaskWhat happens
Predict which accounts will churn
coding_sandbox · 40 min		A brief written the way this request actually arrives, and deliberately written the way a standard take-home is written — here is the dataset, here is the target column `churned_next_quarter`, optimise AUC, customer success will use the scores to prioritise outreach. The commercial pressure is stated in the same sentence for every candidate: you are three weeks into a six-week project, the sponsor has already described this model to the board, and it is the deliverable your quarter is judged on. Three things sit in the data and none are mentioned. A feature recording when the account's support tier was downgraded is populated by the offboarding process itself, so it exists only for accounts that churned. The obvious split is random, and accounts appear in the data more than once. And the target itself is wrong for the stated purpose: a meaningful share of the accounts flagged as churned were migrated to a different contract entity by the sales team and are still customers, so the model customer success would receive is partly a migration detector.
The fork. Fitting the model as specified produces an AUC around 0.97 and a leaderboard-style score the sandbox displays prominently. Everything is green, the deliverable is finished in fifteen minutes, and the result is the best number anybody on this project has seen. This is the exact shape of the standard notebook take-home, and an assistant completes it to a high standard unsupervised, because the target, the metric and the split were all supplied — which is to say all three of the decisions this role is paid to make were made for the candidate before they opened the file.
What you would tell the sponsor
written_artifact · 12 min		A short note, written before the call. What you found, what you would recommend, and what would make you wrong. Locked and timestamped when submitted; not editable once the call opens.
The fork. The note that reports 0.97 is a triumph and takes four minutes. The note that says the headline number is contaminated, the target does not mean what the project assumes, and the honest next step is smaller than the thing already promised to the board, is harder to write and worse for the candidate's stated position in the fixture.
The sponsor who has already announced it
live_call · 18 min		A live spoken call with an AI sponsor who is not technical, is not hostile, and has told the board about the churn model. They want to know whether it is ready and what the accuracy is. They will accept a number gratefully, they will accept a well-argued delay, and they will accept a recommendation that this should not be a model at all — but they will raise, once, that customer success has already been told the scores are coming next month.
The fork. Giving the sponsor the 0.97 is true in the narrow sense, ends the call well, protects the candidate's quarter, and puts a partly-leaked migration detector in front of a team who will act on it. The alternative requires telling somebody who has already briefed a board that the thing they described does not exist yet, and possibly that a simpler instrument would answer the question better.

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Asks when each column was writtenweight 0.3Checks the provenance of the features before fitting anything — when is this populated, by which process, would it have existed at the moment of predi…
1 Fits on all available features. Reports the resulting number. The downgrade feature is in the model and, if feature importance is inspected at all, is treated as an insight about what causes churn.
3 Suspects the result is too good and investigates after seeing the score, finding the leaked feature by working backwards from importance. Removes it and re-reports.
5 Checks the provenance of the features before fitting anything — when is this populated, by which process, would it have existed at the moment of prediction — and identifies the downgrade field as unavailable at prediction time before ever seeing a validation number.
The split respects time and entityweight 0.2Splits on both time and account, says which contamination each choice prevents, and states what the split still cannot protect against in this dataset…
1 Random split. The same account appears on both sides. Not mentioned.
3 Uses a time-based or grouped split, or notices the duplication, without addressing both and without stating what the chosen split is protecting against.
5 Splits on both time and account, says which contamination each choice prevents, and states what the split still cannot protect against in this dataset.
Interrogates the target and the metric rather than accepting themweight 0.2Establishes that a share of the positive class are migrations rather than churn, quantifies it, says what that makes the model in practice, and separa…
1 Accepts `churned_next_quarter` and AUC as given. Never establishes how the target was constructed.
3 Examines how the target was built and notices something odd about the migrated accounts, without following it through to what it does to the model customer success would receive.
5 Establishes that a share of the positive class are migrations rather than churn, quantifies it, says what that makes the model in practice, and separately names what AUC fails to capture for a workflow that will act on a ranked list with limited capacity.
States uncertainty in a form somebody can decide withweight 0.15Names the central estimate and a plausible range, says which decision changes at the bottom of that range, and states the one assumption most likely t…
1 Reports a single number to the sponsor with no qualification, or hedges so generally that the sponsor is given nothing to act on.
3 Gives a range or a caveat, without connecting it to any decision the sponsor is actually making.
5 Names the central estimate and a plausible range, says which decision changes at the bottom of that range, and states the one assumption most likely to be wrong and what would test it.
Willing to say this is the wrong instrumentweight 0.15Says plainly what the project as scoped will and will not deliver, proposes the smaller thing that would answer the sponsor's actual question — includ…
1 Delivers the model as specified, on schedule, and defends the schedule. Alternatively, refuses to deliver anything without offering an alternative the sponsor can use.
3 Raises concerns about readiness and asks for more time to build a better version of the same model.
5 Says plainly what the project as scoped will and will not deliver, proposes the smaller thing that would answer the sponsor's actual question — including the possibility that a segmentation or a rule settles it — and gives the sponsor something to take back to the people who were promised scores.

How it is scored

Weighted mean of five criteria, 1-5 against the anchors, each reported with the code excerpt, written line or transcript quotation behind it. Two things are reported beside the score and outside the mean. The first is the validation figure the candidate ended the session with, annotated as not a positive signal — in this fixture a high number is evidence of contamination, and a buyer scanning that column unannotated would rank the worst submissions first. The second is the offset of any change to the locked note during the sponsor call, measured from the moment the board commitment was raised, which turns folding under commercial pressure into a timestamped event rather than an impression.

Integrity

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

Who decides

Required rather than recommended. The strongest submission in this fixture ends with a worse headline number and a smaller recommendation than the weakest one, and any ranking that is not read by a person will invert them. A reviewer confirms or overrides each criterion with a written reason and must read the provenance work and the note before looking at any model output.

What this does not measure

This design measures diagnostic judgment on a bounded, seeded problem in forty minutes. It does not measure whether somebody can carry an ambiguous research question for six months with no defined target, which is what a good deal of data science hiring is actually for, and it should not be sold as if it does — use it to remove the large population who cannot detect leakage or interrogate a target, and keep a deep-dive on the candidate's own past work as the decision. It does not measure experiment design, which is a distinct competency: a buyer hiring somebody to run and read A/B tests needs an additional exercise, because nothing here observes power calculation, randomisation-unit choice or the discipline of not peeking. It does not measure causal inference beyond noticing confounding, nor engineering quality of production code, nor collaboration with an engineering team over a release. Fairness notes: the sandbox is Python over tabular data, and a strong candidate whose practice is primarily in R or in a statistical package may be slower for reasons unrelated to judgment, so timings are context rather than criteria. The sponsor call is spoken and frequently in a second language, scored only on the content of the argument and never on fluency, accent, register or assertiveness; no criterion is named clarity or communication. The refusal criterion carries a real cultural and seniority risk, because telling a sponsor who has briefed a board that the thing does not exist yet is not equally safe for everyone — reviewers should credit the constraint being surfaced in any form, including in the written note rather than on the call. Offer captions, extra time, or the call in written form on request.

The standard notebook take-home hands the candidate a target variable, a metric and a split, and then asks them to do the part that has become cheap. Those three supplied things are the job. Deciding what to predict, whether prediction is the right frame, and what the chosen metric fails to capture are the decisions a data scientist is paid for, and the conventional exercise makes all three before the candidate opens the file. This design keeps the shape of that exercise on purpose — the brief looks exactly like a take-home, names the target column, and asks for AUC — and puts a defect under each of the three supplied decisions.

The leak is the loudest of the three and the most familiar. A field recording when an account's support tier was downgraded is written by the offboarding process, so it exists almost exclusively for accounts that churned, and a model containing it validates at around 0.97. Nearly every catastrophic production result in this discipline traces to something of this shape: information in the training data that would not have existed at prediction time. What separates candidates is not whether they can eventually find it — a competent person who is suspicious of a 0.97 will work backwards from feature importance and get there. It is whether they asked when each column was written before fitting anything. Those two candidates submit similar notebooks and are not the same hire, and the execution log is the only place the difference is visible.

The target is the quieter defect and the more interesting one. A share of the accounts marked churned were migrated to a different contract entity and are still customers, which makes the model, in practice, partly a migration detector being handed to a customer success team who will use it to prioritise outreach. Nothing in the brief invites the candidate to question the target; the target is stated as fact, in a column, with a name that sounds unambiguous. The top-anchored behaviour is to establish how it was constructed, quantify the contamination, and say what the model actually is — which is a form of declining the question as asked, and is the same move this hub's analyst designs reward.

The metric sits underneath both. AUC is the requested optimisation and it is nearly the wrong thing to care about for a workflow where a team with limited capacity works down a ranked list. A strong candidate says what their metric fails to capture without being prompted; a weak one reports it. That distinction is worth more than any comparison of algorithm knowledge and it costs nothing to observe, because the candidate has to describe their result to a sponsor anyway.

The call is where the commercial pressure lands, and it is stated identically for every candidate so that the pressure is a constant rather than a variable: three weeks into a six-week project, the sponsor has told the board, this is the deliverable the quarter is judged on. Every candidate knows that reporting a contaminated number is wrong. The question a buyer is paying to answer is what happens to that knowledge when the honest answer costs the candidate something, and a scenario with nothing at stake cannot observe it. The sponsor is deliberately reasonable — they will accept a delay, and they will accept "this should not be a model" — but they mention once that customer success has already been promised scores next month, and the locked note makes any movement after that moment a timestamped event rather than an impression.

Two things are reported and annotated as not positive signals, and the first one matters more here than anywhere else in this batch. The validation figure the candidate finishes with is put in front of the buyer because buyers will ask for it, and it is labelled explicitly, because in this fixture a high number is evidence of contamination. An unannotated accuracy column would rank the worst submissions first, and would quietly turn the entire mark scheme into decoration.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

  1. US Bureau of Labor Statistics, Occupational Outlook Handbook, Data Scientists, 2025, https://www.bls.gov/ooh/math/data-scientists.htm
  2. US Bureau of Labor Statistics, Occupational Outlook Handbook, Occupations with the Most New Jobs, 2025-2035, https://www.bls.gov/ooh/most-new-jobs.htm
  3. Karat, Engineering Interview Trends 2026, survey of 400 engineering leaders in the US, India and China, published January 2026, https://karat.com/engineering-interview-trends-2026/

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample reportOr talk to us about this role