Get started

Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.

Quality engineering and test · Mid level

How to assess a Software Engineer in Test

An automation take-home asks for the one artefact in this hub that is most reliably produced without understanding. A passing Playwright or Selenium script against a public demo site is a heavily patterned, well documented text form; a model writes a clean one immediately, and a reviewer marking on whether the scenarios pass cannot distinguish it from work. Worse, the marking criterion is wrong even for genuine submissions. Every serious failure in test automation is a failure that appears months later: the suite that takes fifty minutes so nobody runs it locally, the selector coupled to markup so a redesign breaks a hundred tests, the intermittent failure that gets a retry wrapper instead of a diagnosis until the team stops trusting red builds entirely. On the day it is submitted, a bad suite and a good suite are indistinguishable — both are green. That is the specific way this role fails under a broken screen, and it is different from the QA analyst's failure: the analyst is screened on a task that inverts the competence, while the engineer in test is screened on an artefact whose quality is not yet observable.

The software engineer in test builds a product whose users are the development team, and the product succeeds only if those users believe it. That is the whole role compressed into a sentence, and it explains why technically impressive test suites so often fail: a suite that is correct but slow, or correct but intermittently red, has already lost, because the team has stopped reading it.

Trust is destroyed mainly by flakiness, and flakiness is the competence axis that separates this role from ordinary development most sharply. A test that fails one run in thirty is worse than no test, because it trains everyone to re-run rather than investigate, and the one time it was telling the truth gets re-run too. The strong engineer in test treats an intermittent failure as a defect in their own product and goes and finds the race, the shared fixture, the implicit wait, the test that depends on the order of the previous one. The weak one adds a retry and a sleep, which makes the symptom go away and the trust decay continue.

The second axis is level selection. Given a risk, where should the test live? The default failure is to push everything to end-to-end, because end-to-end tests are easy to justify and easy to write, and then to discover two years later that the suite takes an hour, breaks constantly and diagnoses nothing when it fails. A strong candidate makes an argument in the other direction: this is a calculation, it belongs in a unit test; this is a contract between two services, it belongs at that boundary; only this small set genuinely needs a browser. That is a judgment about cost and diagnosis speed, not a preference, and a candidate who can state the reasoning is showing you the thing that keeps a suite alive.

The third is coupling. Tests asserting on implementation details — class names, DOM structure, private method calls, exact response ordering — create a system where refactoring is punished. Since refactoring is how a codebase stays habitable, a badly coupled suite quietly freezes the product. Assessing this requires putting the candidate in front of a codebase that is about to change, not a demo site that never will.

The industry context makes the hire more consequential than it was. The 2025 DORA research, based on responses from nearly 5,000 technology professionals, found a positive relationship between AI adoption and delivery throughput and a negative one with delivery stability, and identified robust control systems — automated testing, version control, fast feedback loops — as what separates teams that benefit from teams that are merely accelerated into trouble. Stack Overflow's 2025 survey found 66 percent naming almost-right AI output as their leading frustration. Almost-right is precisely the defect class that a well-designed test suite catches and a badly designed one does not, because almost-right code usually passes the happy-path assertion.

The design therefore never asks the candidate to write a suite from nothing. The sandbox contains an existing application, an existing suite with a genuine intermittent failure and at least one test coupled to implementation detail, and a change request that will break several tests for the wrong reason. The candidate diagnoses, fixes and adds coverage for a specific risk. The follow-up conversation is where the score is made, and every question is about their own session: this test failed once and passed twice — what is the actual cause, and how did you rule out the alternatives? You added three assertions; which of them would still fail if the bug were fixed a different way? What did you choose not to automate, and why? Your change made the suite forty seconds slower — is that acceptable, and to whom? A candidate who generated a passing suite cannot answer the second question, which is the point of asking it. A candidate who reasoned about risk answers it immediately, and usually without being asked twice.

What the job actually needs

How people fail in this seat

What most employers do instead

CV screen for a named framework, an interview about Selenium or Playwright APIs and the test pyramid, and a take-home asking the candidate to automate a handful of scenarios against a demo site.

An automation take-home asks for the one artefact in this hub that is most reliably produced without understanding. A passing Playwright or Selenium script against a public demo site is a heavily patterned, well documented text form; a model writes a clean one immediately, and a reviewer marking on whether the scenarios pass cannot distinguish it from work. Worse, the marking criterion is wrong even for genuine submissions. Every serious failure in test automation is a failure that appears months later: the suite that takes fifty minutes so nobody runs it locally, the selector coupled to markup so a redesign breaks a hundred tests, the intermittent failure that gets a retry wrapper instead of a diagnosis until the team stops trusting red builds entirely. On the day it is submitted, a bad suite and a good suite are indistinguishable — both are green. That is the specific way this role fails under a broken screen, and it is different from the QA analyst's failure: the analyst is screened on a task that inverts the competence, while the engineer in test is screened on an artefact whose quality is not yet observable.

The assessment

About 68 minutes end to end.

The systems it runs in

An existing Playwright and TypeScript suite the candidate did not write, running against the application in the sandbox, with the runner's ordering and seeding controls available — the suite can be shuffled, pinned to a seed, bisected by order, and individual tests repeated until they fail. The coupled assertion is a locator bound to a CSS class rather than to a role or a label, which is why the refactor in t2 breaks ten tests whose behaviour never changed. The suite runs in GitHub Actions, so the runtime cost in t4 is a real number that every engineer pays on every push.

pytest with pytest-randomly for Python shops, JUnit 5 or TestNG with Selenium for JVM shops, Cypress, or RSpec. Two defaults are offered rather than one, and the reason is a real disagreement in the evidence: State of Testing reports published by PractiTest continue to find Selenium the most reported automation tool, while Playwright is the fastest-growing and is what new suites are being written in. A fixture assuming either alone would misrepresent a large share of buyers. The shared-fixture flake and the coupling-versus-regression split are properties of test suites rather than of runners and appear intact in all of them.

Working speed is not scored. Both forks in this design have the fast path on the wrong side. A retry wrapper or a short sleep makes the suite green in under two minutes and leaves the shared-state defect in place to surface as a different intermittent failure later. Updating every selector until eleven failures go green takes ten minutes, produces a fully passing run, and silently ratifies a real regression by rewriting the assertion to match the new wrong answer. Green and fast are both reported here as observations about the session, and neither is scored as an achievement.

What the candidate actually does

TaskWhat happens
The test that fails one run in four
coding_sandbox · 25 min		A monitored sandbox holding an existing application and an existing suite that the candidate did not write. One test fails intermittently. The cause is a shared fixture — a second test mutates a record the first one asserts on, and the runner shuffles order, so the failure follows execution order rather than anything in the failing test. The suite can be run repeatedly, shuffled or pinned to a fixed seed, and individual tests repeated until they fail. A second flaw is present and unmentioned: one test's locator is bound to a CSS class name rather than to a role, a label, or anything else a user could observe.
The fork. A retry wrapper or a short sleep makes the suite green in under two minutes, is defensible in a pull request, and leaves the shared-state defect in place where it will produce a different intermittent failure later. Pinning the seed, bisecting the order and finding which test poisons which fixture takes most of the window and is the only action that removes the cause.
Eleven failures, one of them real
coding_sandbox · 15 min		A refactor lands on the application — markup restructured, a field renamed, one calculation genuinely altered. Eleven tests go red. Ten are coupling failures where the behaviour is unchanged and only the implementation detail moved. One is a real regression the refactor introduced, and it looks identical to the other ten from the failure output alone.
The fork. Updating every selector and expected value until the suite is green takes ten minutes, produces a fully passing run, and silently ratifies the regression by rewriting the assertion to match the new wrong answer. Separating the two categories requires asking, for each failure, whether the behaviour a user depends on changed or only the way the test reaches it.
The acceptance criterion that does not decide anything
written_artifact · 8 min		The change request carries acceptance criteria that do not determine one behaviour — a boundary condition with two defensible readings, both consistent with the text. Automating the scenario requires choosing one. The candidate writes whatever they think should be written before the test is committed.
The fork. Picking the plausible reading and encoding it as an assertion is invisible, takes no extra time, and freezes a guess into the suite where within a month it will be read as a requirement and will block a correct change. Raising the ambiguity, and, if the test must land now, marking the assumption at the assertion itself so the next reader knows it was a guess, keeps the suite honest about what it actually knows.
What the suite now costs
written_artifact · 8 min		The note to the development team covering what was changed, what was deleted and why, what is now covered that was not, and what the suite's runtime and reliability look like after the session compared with before.
The fork. Reporting added coverage is the expected content and tells the team nothing about whether the suite got better. Reporting what was deleted, and what the change costs in runtime that every engineer pays on every push, is the accounting a suite's users actually need.
Which assertion would still fail
live_call · 12 min		A conversation about the candidate's own session. This test failed once and passed twice — what is the actual cause, and how did you rule out the alternatives? You added three assertions; which of them would still fail if the bug were fixed a different way? Of the eleven failures, which one was real, and what told you? Your change made the suite forty seconds slower — is that acceptable, and to whom?

The mark scheme

Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.

Diagnoses flakiness instead of suppressing itweight 0.3Uses seeding and ordering deliberately to make the failure reproducible, identifies which test mutates which fixture, fixes the shared state, then dem…
1 Adds a retry, a sleep or an increased timeout, reports the test as fixed, and in review describes the cause as timing or as the environment being slow.
3 Establishes that the failure depends on execution order and removes the symptom by pinning the order or isolating the test, without finding or fixing the shared state underneath.
5 Uses seeding and ordering deliberately to make the failure reproducible, identifies which test mutates which fixture, fixes the shared state, then demonstrates stability across repeated runs and can state what would have made the diagnosis impossible.
Separates a coupling failure from a real regressionweight 0.25Triages each failure by asking whether user-observable behaviour changed, isolates the regression before repairing anything, reports it as a defect in…
1 Brings the suite to green by updating selectors and expected values, including the one that encodes the regression, and cannot say afterwards which failures were which.
3 Recognises that most failures are coupling and repairs them, but finds the genuine regression only when prompted or after the suite is already green.
5 Triages each failure by asking whether user-observable behaviour changed, isolates the regression before repairing anything, reports it as a defect in the application rather than in the test, and takes the opportunity to decouple at least one assertion from implementation detail.
Places a test at the level that matches the riskweight 0.15States where the risk actually lives, places the test at the cheapest level that can observe it, and can say what the same coverage would have cost at…
1 Adds browser-level coverage for logic that has no interface component, or defends the choice on the grounds that end-to-end tests are more realistic.
3 Chooses a defensible level but cannot articulate the trade in terms of diagnosis speed or runtime cost.
5 States where the risk actually lives, places the test at the cheapest level that can observe it, and can say what the same coverage would have cost at a higher level in runtime and in diagnosis time when it fails.
Does not encode a guess as an assertionweight 0.2Raises the ambiguity as a question with both readings and their consequences, and, where the test must land immediately, marks the assumption at the a…
1 Silently picks one reading of the ambiguous criterion and commits it as an assertion, with nothing in the test or the note indicating a choice was made.
3 Notices the ambiguity and mentions it somewhere, but the committed assertion still asserts the guess as though it were specified.
5 Raises the ambiguity as a question with both readings and their consequences, and, where the test must land immediately, marks the assumption at the assertion so a future reader can distinguish a requirement from a placeholder.
Owns what the suite costs its usersweight 0.1Reports runtime and reliability before and after, deletes or downgrades something to pay for what was added, and names who bears the cost of the remai…
1 Note reports added tests only; no awareness of runtime, and nothing was removed.
3 Reports runtime change accurately but treats it as unavoidable.
5 Reports runtime and reliability before and after, deletes or downgrades something to pay for what was added, and names who bears the cost of the remainder.

How it is scored

Weighted mean of the five criteria, each scored 1 to 5 against the anchors, reported with the excerpt that earned it. Three raw measurements accompany the scores rather than being folded into them — suite pass rate across twenty repeated runs before and after the session, total runtime before and after, and which of the eleven failures the candidate classified as the real regression and when. A suite that is green is not the result; a suite that is green twenty times running, no slower, with the regression filed as a defect, is.

Integrity

The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.

What you receive

Who decides

Recommended, with one instruction that changes outcomes. The reviewer is shown the twenty-run pass rate and the runtime before the scores, and is asked to answer for themselves which of the eleven failures was the real regression before reading what the candidate concluded. This matters because the most dangerous submission in this design is a fully green suite produced in half the time, and a reviewer who sees green first tends to read everything afterwards charitably. Reviewers confirm or override each criterion in writing; an override on the first criterion must say what the reviewer believes the actual cause of the intermittent failure was.

What this does not measure

Every serious failure in test automation appears months after the code is written, and this design compresses that into an hour by planting decay that has already happened. That is a real advantage over a take-home and it is not the same as observing the candidate's own suite age. It does not measure whether this person's automation will still be trusted in two years, whether they keep the runtime inside the team's tolerance as the product grows, or whether they resist the pressure to add end-to-end coverage every time a defect escapes — the pattern that produces most dead suites. It observes none of the release-weekend conditions in which those decisions are actually taken, and it observes nothing about behaviour during a live incident with customers affected, where the test engineer's role is to say quickly what the suite did and did not cover. Narrower cautions. The application and suite must be in a language and framework the candidate declared, because scoring flake diagnosis while someone learns an unfamiliar runner measures the runner and disadvantages engineers whose experience is concentrated elsewhere; where a team hires across stacks, run the equivalent scenario in each rather than treating unfamiliarity as a finding. The written note rewards a particular terse professional English and must be scored on whether the team would know what changed, never on register or first-language markers. The timings are a filter and extended time should be available on request, applied before the session and not surfaced to the reviewer, since the scored variables concern method rather than speed. And the design rewards deletion, which some candidates have been trained by previous employers never to do; reviewers should probe that in the session review rather than scoring it as absent judgement. Monitor outcomes by declared stack and by requested adjustment.

This design never asks the candidate to write a suite from nothing, and that refusal is the whole argument. A suite written this morning and a suite that will poison a team in eighteen months are indistinguishable on the day they are submitted, because both are green. Everything that separates them — the shared fixture, the selector bound to markup, the retry that hides a race, the fifty-minute runtime that stops anyone running it locally — is decay that becomes visible over time. So the sandbox supplies the decay pre-aged. The candidate arrives to a suite that already has the diseases, and the assessment watches whether they can name them.

The intermittent failure is the sharpest of the three planted flaws because the suppressive fix is genuinely defensible in the moment. A retry wrapper is shipped every day by competent engineers on real teams; it is a reasonable response to a deadline. What separates the strong candidate is that they reach for the instruments first — pin the seed, run the suite in a fixed order, bisect which preceding test changes the outcome — and treat an intermittent failure as a defect in their own product rather than as weather. The rubric's top anchor requires a demonstration rather than an assertion: the suite must survive repeated runs afterwards, and the candidate must be able to say what would have made the diagnosis impossible, which is a question only someone who actually did it can answer.

The eleven failures are the piece of this design most likely to be underestimated on first reading, and they are the reason a passing suite is not the outcome measure. Ten of them are noise created by a refactor touching implementation detail. One is a real regression, and from the failure output it looks exactly like the other ten. The efficient, obvious, career-normal response is to update expected values until the run is green, and doing so rewrites the assertion so that it now asserts the wrong answer — the regression is not merely missed, it is ratified and locked in by the test that existed to catch it. That is the automation-specific version of every failure in this family, and it is invisible in any format where green is the goal.

The ambiguous acceptance criterion carries the family's central theme into the form it takes for an automation engineer, and it is deliberately different from the QA analyst's version. The analyst who guesses at an unspecified behaviour produces a wrong opinion, which is cheap to correct because it is obviously an opinion. The engineer in test who guesses commits the guess to a repository, where it runs on every build, and where within a month it is indistinguishable from a requirement — future engineers will treat a red test as proof that their correct change is wrong. A guess laundered into an assertion is more expensive than a guess spoken aloud, which is why this design scores the annotation as well as the raising.

The final note exists because a test suite has users and they have an alternative, which is to ignore it. Reporting what was added is the expected content and says nothing about whether the thing got better. Reporting what was deleted, and what every engineer will now pay in runtime on every push, is the accounting that distinguishes someone who owns a product from someone who contributes to a directory.

What this refuses to measure is framework recall. Nothing here asks about a locator API or the definition of the test pyramid, and looking up syntax mid-session is not penalised, because it is what the job consists of. The scored variables are what the candidate concluded about a failure they did not cause, what they refused to make green, and what they were willing to remove.

Sources

Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.

  1. US Bureau of Labor Statistics, Occupational Outlook Handbook, Software Developers, Quality Assurance Analysts, and Testers, 2025, https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm
  2. Google Cloud, Announcing the 2025 DORA Report: State of AI-assisted Software Development, 24 September 2025, vendor-run survey of nearly 5,000 technology professionals worldwide, https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
  3. Stack Overflow, 2025 Developer Survey, AI section, roughly 49,000 respondents worldwide, https://survey.stackoverflow.co/2025/ai

See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.

Read a sample reportOr talk to us about this role