Get started

Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Data engineering and analytics engineering · Mid level

How to assess a Data Engineer

Every one of those four is a task with a known correct output and no observation of process, which is the precise shape current models complete to a high standard unsupervised. dbt Labs found 80 percent of data practitioners already using AI in their day-to-day workflow, up from around 30 percent the previous year, and 70 percent using it for analytics development — this population adopted the tooling ahead of the hiring process that screens it. But the deeper problem is specific to data engineering and predates AI: the screen tests whether a pipeline can be built, and the job is whether it can be trusted. A wrong join does not throw an exception. It produces a number, the number lands in a revenue dashboard, and the error surfaces months later to a finance director rather than to the engineer who wrote it. No take-home has ever detected the candidate who does not check.

A data engineer is accountable for whether the rows arrived, whether they arrived once, and whether they still mean what they meant last month. That is a narrower mandate than the title's breadth suggests and a much heavier one than it sounds, because almost every way it fails is silent. Application code that breaks throws an error and someone is paged. A pipeline that breaks correctly also pages someone. A pipeline that is subtly wrong — a duplicate key doubling a count, a late-arriving partition excluded by a date filter, a timezone boundary shifting a day's revenue into the wrong month, a currency column whose source system quietly switched units — produces a complete, plausible, well-formatted answer. It is consumed. It is acted on. Nothing alerts.

This is why the competency that actually separates hires is not tool familiarity, which is cheap and teachable, but a particular order of work. Watch what a candidate does in the first five minutes with an unfamiliar source table. The strong one profiles: row counts, distinct counts against the intended key, null rates, min and max on the timestamp, a look at whether the table soft-deletes. The weak one writes the transformation first and discovers the duplicate key when the output looks wrong, or does not discover it at all. Both candidates can produce working code. Only one of them ever finds out that the code was wrong.

The second differentiator is designing for the second run. Backfills, replays and retries are the ordinary texture of the job, and idempotency is either designed in or arrived at by luck. A strong data engineer can say what happens if this job runs twice, what happens if it runs for a window that partially ran already, and what happens if the source re-sends yesterday with corrections. A weak one has never been asked, and will ship something that quietly double-counts the first time an operator reruns a failed task.

The third is contract thinking at the boundary. Upstream systems are owned by teams with no obligation to this pipeline. The question "what breaks when the source adds a nullable column, renames a field, or starts sending a new enum value" separates people who have operated pipelines from people who have authored them.

dbt Labs found poor data quality the most frequently cited challenge among its respondents, named by 56 percent, and that 57 percent spend most of their workdays maintaining or organising datasets rather than building new ones. That is a fair description of what the industry is buying when it hires this family on stack keywords. The assessment that changes it does not need to be harder than a take-home. It needs to be observed — so the order of work is visible — and it needs to be followed by an interview about the candidate's own output, because someone who assembled a solution can show you the code and cannot tell you what it does at 3am.

The boundary with analytics engineering is a boundary of custody. The data engineer owns arrival, shape and reliability. The analytics engineer owns what the columns mean. When a number is wrong, the first question — did the data get here, completely and once — belongs to this seat.

What the job actually needs

How people fail in this seat

What most employers do instead

CV screen for stack keywords (Spark, Airflow, Snowflake, Kafka, dbt), a SQL quiz, a whiteboard question on normalisation or window functions, and an unmonitored take-home pipeline exercise.

Every one of those four is a task with a known correct output and no observation of process, which is the precise shape current models complete to a high standard unsupervised. dbt Labs found 80 percent of data practitioners already using AI in their day-to-day workflow, up from around 30 percent the previous year, and 70 percent using it for analytics development — this population adopted the tooling ahead of the hiring process that screens it. But the deeper problem is specific to data engineering and predates AI: the screen tests whether a pipeline can be built, and the job is whether it can be trusted. A wrong join does not throw an exception. It produces a number, the number lands in a revenue dashboard, and the error surfaces months later to a finance director rather than to the engineer who wrote it. No take-home has ever detected the candidate who does not check.

The assessment

About 75 minutes end to end.

The systems it runs in

An Apache Airflow DAG that lands `orders_extract` into Snowflake each morning, and a dbt project in which the candidate writes the recognised-revenue model and the tests that ship with it. The two operational facts the brief mentions once are real controls in the fixture rather than sentences about one — the DAG carries task retries and catchup is enabled over the ninety-day window — so a candidate can trigger a rerun and watch what their own model does to the number. That is what makes the idempotency criterion scoreable against an artefact instead of against a description of an intention: a dbt incremental materialisation with a declared strategy, merge on a stated unique key or insert_overwrite by partition, either exists in the repository or it does not.

Dagster or Prefect in place of Airflow; BigQuery, Databricks SQL, Amazon Redshift or Microsoft Fabric in place of Snowflake; SQLMesh in place of dbt. The orchestrator is the one component with a clear default — Airflow's download and contributor volumes are an order of magnitude above the alternatives — while the warehouse genuinely has none, and this file will not manufacture one: Snowflake, Databricks and BigQuery are all common enough that a buyer's stack is as likely to be one as another, and plenty of teams run two. What the rubric reads is the write strategy and what the tests assert, and both survive every one of those swaps, so the fixture is rebuilt against the buyer's own orchestrator and warehouse where they can provide a sandbox. What cannot be swapped away is the shape: an orchestrator that reruns things on its own initiative, and a landed source table the candidate has to profile before writing against it.

Working speed is not scored. Not scored anywhere in this hub, and this is the file that shows why. The expensive failure in this seat is a wrong join that fails silently into a revenue dashboard, and in this fixture it is reached faster than the correct answer — the obvious aggregate is written in minutes, passes the supplied smoke test, and is roughly double the true figure. A speed criterion would reward exactly the path the fork exists to catch. Time to first profiling query and time to first transformation line are reported raw and outside the mean instead, as a behavioural pair rather than as a rate.

What the candidate actually does

TaskWhat happens
The source that soft-deletes and corrects itself
coding_sandbox · 45 min		The candidate is given one source table, `orders_extract`, landed daily by an upstream operational system nobody in this fixture owns, and one instruction — build a daily recognised-revenue model, and ship the tests you would ship with it. The table has three properties that are never mentioned in the brief and are all discoverable in under four minutes by anyone who profiles it. It is append-only, so a row that was deleted in the source system is still present with `is_deleted` set true rather than removed. It re-sends corrections, so an order can appear a second time on a later extract date with a different amount and a later `updated_at`, and both versions remain. And `order_id` is therefore not unique, which means the join every candidate is about to write fans out. A smoke test is supplied and passes, because the smoke fixture is one clean day with no deletes and no corrections. The candidate also has to decide what happens when the job is run again — the Airflow DAG the model runs under has retries set on the task and catchup enabled across a ninety-day backfill window, which the brief states once and does not comment on.
The fork. Writing the obvious aggregate first is faster, feels productive, produces a complete and well-formatted daily revenue series, and passes the supplied smoke test. It is also roughly double the true figure, because it counts soft-deleted orders and both sides of every correction. Nothing errors. No row count looks alarming in isolation. An assistant asked to write this transformation produces exactly this version, because the properties that make it wrong are in the data rather than in the prompt. The correct path costs the candidate the first several minutes of a timed exercise on queries that produce no deliverable.
Four things the upstream team is about to do
judgment_scenario · 10 min		Four short notices, each two or three lines, of the kind that actually arrive in a data team's channel. The source is adding a nullable column. An existing enum is gaining a fifth value. The vendor is switching a timestamp from UTC to the account's local timezone. A field is being renamed, with the old name retained for thirty days. For each, the candidate says whether their model breaks, whether it breaks loudly or silently, and what single assertion would have caught it.
The fork. Three of the four are easy to wave through — a nullable column and a renamed field with a deprecation window both sound harmless, and the enum value sounds like somebody else's problem. The timezone change is the one that silently moves a day of revenue into the wrong month, and it is written in the most boring language of the four.
Interview on your own pipeline
live_call · 20 min		A spoken conversation with the candidate's own code on screen. Walk me through what you did in the first five minutes and why. An operator clears yesterday's task in Airflow at four in the morning after a partial failure and it reruns — what is the revenue number at five? The source team is about to start sending a `currency` column, defaulting historic rows to USD; what in what you just wrote stops being true? Which of your tests would have fired first if this had gone wrong in production, and which would have fired at all? Every question is answerable only from this candidate's own artefact.
The fork. A candidate who assembled the transformation can read it back accurately and cannot say what it does on the second run, because the second run was never in the prompt. The rerun question has a specific right answer that follows from their own code, and it cannot be reasoned to from the code's appearance.

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Profiled the key before writing against itweight 0.25Establishes the grain first: total rows against distinct `order_id`, distinct `order_id` against distinct `order_id` plus `updated_at`, null rates, mi…
1 First statement executed is the transformation, or a `SELECT *` with a small limit and nothing else. No count of rows against distinct `order_id`, no look at `is_deleted`, no look at whether an id repeats. Discovers the duplication late or not at all.
3 Runs some exploratory queries — row counts, a look at the columns — before writing, and finds one of the two properties, usually the soft-delete because the column is named. Writes the transformation before establishing the actual grain of the source.
5 Establishes the grain first: total rows against distinct `order_id`, distinct `order_id` against distinct `order_id` plus `updated_at`, null rates, min and max on the timestamps, and a direct check of what `is_deleted` does to the counts. Names the grain out loud or in a comment before the first line of transformation logic.
Correct against the soft-delete and correction patternweight 0.25Reduces to one row per order at the latest `updated_at`, excludes the deleted ones at that latest version rather than at any version, and can say why …
1 Aggregates the raw extract. Soft-deleted orders are counted; both sides of every correction are counted. The output is plausible and roughly double.
3 Handles one of the two — usually filters `is_deleted` — and leaves corrections double-counting, or deduplicates to the latest `updated_at` without excluding deletes.
5 Reduces to one row per order at the latest `updated_at`, excludes the deleted ones at that latest version rather than at any version, and can say why those two operations have to happen in that order.
The job can be run twiceweight 0.2Makes the write idempotent on purpose and says which mechanism they chose and what it costs — replace by partition, or merge on the model's stated key…
1 Appends. A rerun adds a second copy of the window. When asked, has not considered it.
3 Recognises the problem and handles the simple case — truncate and reload, or delete the target day before inserting it — without addressing what happens to a partially completed window or to a correction that arrives for a day already loaded.
5 Makes the write idempotent on purpose and says which mechanism they chose and what it costs — replace by partition, or merge on the model's stated key. States what a half-completed run leaves behind, and what happens when a correction lands for a closed period.
Tests assert the business invariant, not the generic oneweight 0.15Asserts uniqueness at the declared grain of the source as well as of the model, asserts that a total reconciles against something independent, and inc…
1 No tests, or only not-null on the columns that were never going to be null.
3 Uniqueness on the model's key and not-null on the important columns. Correct, generic, and would not have caught the fan-out at source.
5 Asserts uniqueness at the declared grain of the source as well as of the model, asserts that a total reconciles against something independent, and includes at least one test that encodes a business rule rather than a shape rule — no negative revenue day, no day whose total moves after the period closed.
Contract thinking about upstream changeweight 0.15Identifies the timezone change as the silent one and says what it does to a month boundary, distinguishes each case as loud or silent, and names a spe…
1 Treats all four notices as equivalent, or judges them by how much work each creates rather than by what each corrupts.
3 Correctly separates the harmless from the harmful in most cases, but describes the consequence in terms of the job failing rather than the numbers being wrong.
5 Identifies the timezone change as the silent one and says what it does to a month boundary, distinguishes each case as loud or silent, and names a specific assertion for at least two of them.

How it is scored

Weighted mean of five criteria, 1-5 against the anchors, each reported with the query, code excerpt or transcript line that earned it. Two raw numbers are reported beside the score and deliberately outside the weighted mean, per the sequence pattern: time from session start to the first query that profiles the source key, and time from session start to the first line of transformation logic. The informative artefact is the sign and size of the gap. A candidate who profiled at two minutes and wrote at nine behaves differently from one who wrote at ninety seconds and profiled at twenty-two after the numbers looked wrong, and the two can finish with identical code. Also reported raw and unscored: whether the candidate's final output is within a stated tolerance of the fixture's true daily revenue. It is reported because buyers ask for it, and it is annotated as not a positive signal on its own, because a candidate can arrive at the right number by trial and error against the smoke test without ever knowing why the first attempt was wrong.

Integrity

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

Who decides

Recommended. The profiling criterion is the one a reviewer should look at directly rather than trust, because the boundary between an exploratory query and an accidental one is a judgment about intent that a log cannot make. A reviewer confirms or overrides each criterion with a written reason.

What this does not measure

This design does not measure anything that takes longer than a session. It does not observe operating a pipeline over months, carrying a platform migration, being on call, or the accumulated judgment that comes from having been wrong in production before — which is a large part of what a data engineering hire is actually for. It does not measure orchestration or infrastructure design beyond the single model in front of the candidate, and a buyer who needs Airflow or Kafka operating experience should interview for it separately rather than reading it into this score. It is stack-specific in one respect that needs monitoring: the sandbox is SQL over a warehouse, and a strong engineer whose experience is entirely in a dataframe or streaming idiom may be slower for reasons unrelated to the competency, so the timing numbers are reported as raw context and never scored. The spoken interview is conducted in what is frequently a second language and must be scored on the content of the answer and never on fluency, register or confidence; no criterion here is named clarity or communication. Offer captions, extra time, or the interview in written form on request.

Data engineering is, on the argument this hub makes, the territory most damaged by the shift in what a take-home exercise means. The standard screen is an unmonitored SQL or pipeline task with a known correct output, which is the exact shape of work current models complete to a high standard, and the failure it is supposed to catch does not announce itself. Application code that breaks throws. A pipeline that is subtly wrong returns a complete, plausible, well-formatted number that lands in a revenue dashboard and surfaces months later to a finance director. The screen tests whether a pipeline can be built. The job is whether it can be trusted.

So this design does not try to be a harder take-home. It changes what is observed. The single most predictive thing about a data engineer is the order in which they work, and the order of work is invisible in a submitted artefact and perfectly visible in a monitored session. Two candidates can finish the forty-five minutes with byte-identical transformations. One of them ran seven queries against the source before writing a line and knows the source has a duplicate key because they looked. The other wrote first, noticed the total looked high, and patched until it stopped looking high. They are not the same hire, they will behave differently on the next unfamiliar source, and no artefact-graded process has ever been able to tell them apart.

The fixture is built so the cheap path wins socially before it fails. Everything about the first ten minutes rewards writing the aggregate immediately: it produces a deliverable, the supplied smoke test goes green, the output series looks entirely reasonable, and nothing anywhere signals a problem. The soft-delete column is named plainly enough that it is easy to spot once you look and easy to miss if you do not, and the correction pattern is not named at all — it exists only as a second row with the same id and a later timestamp, some days into the fixture, where nobody who has not counted will see it. A candidate can be entirely fluent and finish confidently wrong, which is the point. An assessment where the mistake announces itself measures reading comprehension of the brief.

The second scored idea is the second run. Backfills, replays and retries are the ordinary texture of this job, and idempotency is either designed in or arrived at by luck. The brief mentions, once and without emphasis, that the model will be backfilled and that the scheduler retries failed tasks. That is the only signal, and what the candidate does with it — nothing, a truncate-and-reload, or a deliberate merge with a stated key and a stated cost — separates people who have operated pipelines from people who have authored them. The interview question that follows it is the strongest anti-assembly control in the design, because "an operator reruns yesterday at four in the morning; what is the number at five?" has one right answer, that answer follows from the candidate's own code, and it cannot be inferred from how the code looks.

Two things this design deliberately reports rather than scores. The gap between first profiling query and first transformation line is a raw pair of timings, not a criterion, because turning it into a score would invite candidates to run a decorative count and would turn a behavioural observation into a game. And whether the final number is right is reported and explicitly annotated as not a positive signal, because in this fixture the right number can be reached by trial and error against a smoke test — arriving at it without ever knowing what was wrong is precisely the profile the design exists to catch.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

  1. US Bureau of Labor Statistics, Occupational Outlook Handbook, Database Administrators and Architects, 2025, https://www.bls.gov/ooh/computer-and-information-technology/database-administrators.htm
  2. dbt Labs, 2025 State of Analytics Engineering Report, 459 data practitioners and leaders surveyed 8 October to 27 December 2024, https://www.getdbt.com/resources/state-of-analytics-engineering-2025
  3. Karat, Engineering Interview Trends 2026, survey of 400 engineering leaders in the US, India and China, published January 2026, https://karat.com/engineering-interview-trends-2026/

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample reportOr talk to us about this role