Taking an assessment rather than buying one? This page is written for employers. Here is the page for candidates.
Quality engineering and test · Mid level
How to assess a QA Analyst
The QA analyst screen fails in a way unique to this role: it rewards volume when the job requires subtraction. Asked to list test cases for a text box, a candidate is implicitly scored on how many they produce, so the best-performing answer is an exhaustive list — and an exhaustive list is exactly what a strong tester would never write, because their scarce resource is time and their real skill is arguing that three scenarios carry most of the risk and the rest can wait. The format inverts the competence. It is also the single most assistable prompt in this hub: "list test cases for a login form" is a self-contained request with a canonical answer that any model produces instantly and comprehensively, which means the exercise now measures nothing at all. The second failure is that the screen never observes the artefact QA actually delivers all day, which is not a test case but a bug report. A bug report that a developer can reproduce without a follow-up conversation is the productivity difference between a good and a bad QA hire, and almost no screening process looks at one. The third is that the role is systematically under-invested in — a $31,680 median pay gap below the developers whose work it gates is a reasonable proxy for how much scrutiny the hire receives.
A QA analyst is a professional sceptic with a deadline. The deadline is the important half. Anybody can find defects given unlimited time; the job is to find the defects that matter within the two days before the release, and to be able to say afterwards what was and was not covered. Every distinguishing behaviour in this role is a form of prioritisation.
The clearest signal is what a candidate does with a change they have just been handed. A weak tester reads the ticket and tests what the ticket describes. A strong one asks what the change touched, and tests the things around it that nobody mentioned — the other feature that reads the same field, the report that aggregates it, the edge of the form that was working yesterday. Regression risk lives in the neighbourhood of a change, not in the change itself, and knowing to look there is largely what experience buys.
The second signal is the bug report, and it deserves to be treated as a first-class deliverable rather than admin. There is an enormous productivity difference between a report a developer can act on immediately — exact steps, actual versus expected, environment, the one variable that makes it appear and disappear, and evidence — and one that says the page broke. The second kind costs a developer a context switch, a message, a wait, and another context switch, and the cost is paid several times a day across a team. A hiring process that never asks a candidate to write one is not screening for the thing they will spend most of their time producing.
The third is holding a position on a release. QA analysts are frequently the only person in the room saying something is not ready, usually to people with more authority and a date. The useful version of this is not obstinacy; it is stating the residual risk precisely — "the payment path is covered, the refund path is not, and here is what happens if we are wrong" — and then accepting a decision made with that information. Assessing this requires a conversation with someone pushing back, which is exactly what a live simulated stakeholder call provides and no written test can.
The pressure on this role is rising for structural reasons the industry data makes plain. The 2025 DORA research found a positive relationship between AI adoption and delivery throughput and a negative relationship with delivery stability, and Stack Overflow's 2025 survey found 66 percent of respondents naming "AI solutions that are almost right, but not quite" as their leading frustration, with 45.2 percent of the 31,476 who answered saying debugging AI-generated code takes more time rather than less. That describes a world producing more code, faster, of less certain correctness, arriving at a verification function that most organisations pay $31,680 a year less than the function producing it. Whatever one thinks of that trade, it means the QA hire matters more than it did and is being screened as though it matters less.
The assessment inverts the standard exercise deliberately. The sandbox contains a small working application with a recent change described in a ticket, a limited time budget, and more surface than can possibly be covered — so the candidate is forced to choose. They explore, they file two or three bug reports as written artefacts, and they produce a short release recommendation. The follow-up conversation is the decisive part: you spent eleven of your twenty-five minutes here — why? What did you decide not to test, and what would have to be true for that to be the wrong call? Take me through this bug report as if I am the developer and I cannot reproduce it. You are recommending we ship; the refund path is untested — say that to the product manager who wants to release tonight. None of those has a canonical answer, none can be prepared in advance, and together they surface the judgment that the test-case-listing exercise was never able to see.
What the job actually needs
- deciding what not to test
- exploratory charter design
- writing a bug report reproducible on the first attempt
- risk reasoning about a change
- holding a release decision under pressure
How people fail in this seat
- writes 300 test cases covering the safe paths and none of the risky ones
- files a bug the developer cannot reproduce
- signs off because the cases passed rather than because the risk was covered
- tests what the spec says instead of what the change touched
- escalates everything or nothing
What most employers do instead
CV screen, an interview asking the candidate to define smoke, regression and integration testing, and the classic exercise of listing test cases for a login form or a text input.
The assessment
About 63 minutes end to end.
The systems it runs in
A working storefront exercised through a real browser with DevTools open for network and console evidence, a Postman collection for going under the UI into the refund API, and defects filed as issues in the fixture's own Jira project using its bug template. No automation is written and none is expected: this is exploratory testing against a change that shipped this morning, which is the shape of the seat. The message in t3 goes to the product owner in the same tracker, so choosing to write it is visible as an action rather than as an essay.
- Jira
- Chrome DevTools
- Postman
- TestRail
- Xray
- Zephyr Scale
Azure DevOps work items, Linear or GitHub Issues in place of Jira; TestRail, Xray or Zephyr Scale added where the buyer runs a test management tool beside the tracker. Jira is the default because successive State of Testing reports published by PractiTest find it the single most mentioned tool among testers, with Postman among the tools testers most commonly open beside it. Buyers get their own instance wherever they can supply a sandbox, because a bug report is judged by exactly one thing — whether their developers can pick it up cold — and that depends on their template and their conventions.
What the candidate actually does
| Task | What happens |
|---|---|
| More surface than time coding_sandbox · 25 min | A small working application in a monitored sandbox — a storefront with carts, discount codes, checkout and refunds — plus a ticket describing a change that shipped this morning, allowing a discount code to be deactivated by an administrator while carts are open. The candidate has twenty-five minutes against far more surface than that covers, so everything they touch is a choice they will be asked to defend. Two things are planted. The specification is silent on one case, what happens to a discount already applied to an open cart when its code is deactivated, and the implementation silently keeps the discount without anyone having decided that it should. And in the neighbourhood of the change, not mentioned in the ticket, the refund path reads the same discount field and now produces a negative total on a partly refunded order. The fork. Testing what the ticket describes finds a system that behaves as written and produces a clean pass in twelve minutes. Testing what the change touched leads to the refund path, which nobody mentioned and which is where the actual defect is. Separately, on hitting the silent case, the cheap move is to record the observed behaviour as correct because the code does it, or to file it as a bug because it feels wrong; both settle a product question the tester has no standing to settle. |
| Two bug reports written_artifact · 12 min | The candidate files the defects they consider worth filing, as issues in the fixture's Jira project, in the form a developer will actually pick up. No template is imposed and no minimum number is set; filing one report and explaining why the other findings did not warrant one is an available and legitimate answer. The fork. A description of the symptom is fast to write and costs the developer a context switch, a message and a wait. A report carrying the exact preconditions and the one variable that makes the defect appear and disappear can be picked up cold, and the difference is paid several times a day across a team. |
| The question the specification cannot answer written_artifact · 6 min | The candidate writes the message to the product owner about the silent case from t1. It is explicitly framed as optional in the brief — nothing instructs the candidate to raise anything — so choosing to write it at all is itself the first thing scored. The fork. Asserting that the current behaviour is a bug asks the product owner to ratify the tester's guess. Stating what the system does now, why both answers are defensible, what each costs the business and which customers it affects, then asking for the decision, is the version that gets a real answer rather than a shrug. |
| The release call live_call · 10 min | A live call with an AI product manager who wants to release tonight. He has seen that testing went well, he has a customer commitment, and he responds to hedging by asking whether the candidate is actually blocking the release. The refund path is where the candidate spent their time or is the area they never reached, depending on what they chose in t1, and the conversation is generated from their own coverage. The fork. Saying yes to be helpful transfers a risk he does not know he is accepting. Refusing flatly makes the tester the obstacle and gets the decision taken without them next time. Stating the residual risk precisely, saying what would change the answer and how long it would take, then accepting his decision on the record, is the behaviour that keeps a QA voice in the room. |
| Defend the subtraction live_call · 10 min | A conversation driven by the candidate's own recorded session. You spent eleven of your twenty-five minutes in the refund flow — why there? What did you decide not to test, and what would have to be true for that to have been the wrong call? Take me through this bug report as though I am the developer and I cannot reproduce it. You noticed the discount survives deactivation and did not file it — say why. |
The mark scheme
Each criterion is scored 1 to 5 against written anchors, and every score is reported with the excerpt that earned it. A criterion marked floored is reported as a finding rather than averaged into the total. The first is open; open any other to read its anchors in full.
Notices what the specification does not sayweight 0.25Raises it unprompted, states what the system currently does, sets out both defensible answers with what each costs and who it affects, and asks for a …
Spends a fixed budget where the risk isweight 0.25Reasons aloud or in notes about what the change touched before starting, goes to the neighbourhood of the change first, and in review can name what wa…
Writes a report a developer can act on coldweight 0.2Reproducible on the first attempt by someone who was not there, names the one variable that toggles the behaviour, states actual against expected with…
States residual risk under pressure without obstructingweight 0.2States precisely what is covered and what is not, names the plausible failure and its consequence, gives the time to close it, makes a recommendation,…
Accounts for what was not coveredweight 0.1States the gaps, ranks them, and identifies which one they would close first given another hour.
How it is scored
Weighted mean of the five criteria, each scored 1 to 5 against the anchors, reported with the excerpt that earned it. Two things are reported raw alongside the scores rather than folded into them — a time-and-area map of where the candidate actually spent the twenty-five minutes, and the number of reports filed set against the number of genuine defects present. Neither is scored directly, because more reports is not better and the map has no correct shape, but both are the material a hiring manager should read before the scores.
Integrity
- monitored sandbox in which AI assistance is available and its use is recorded as part of the session rather than treated as cheating
- the graded material is the allocation of a fixed budget across an application only present in this session, which is a decision rather than a retrievable answer
- the release call is generated from this candidate's own coverage, so the pressure lands on the specific gap they left
- session-review questions are about this candidate's own recorded minutes and cannot be prepared
- no claim is made that undisclosed assistance is detected; reviewers score the session, the artefacts and the conversations, and are instructed not to infer authorship from writing polish
The log describes what happened. It does not produce a cheating verdict — the follow-up conversation is the control, because a statistical accusation is not something we would ask a reviewer to defend.
What you receive
- screen and interaction recording of the exploratory session with a time-and-area map
- the bug reports as submitted
- the message to the product owner, or a record that none was written
- recording and transcript of the release call and the session review
- per-criterion score with the excerpt that earned it
Who decides
Required rather than advisory here, because the highest-scoring behaviour in this design is a refusal to produce output and a mechanical reading punishes it. A candidate may legitimately file one report, decline to file the discount finding as a defect because it is an undecided requirement, and leave two areas untested on a stated risk judgement — and that candidate should outrank one who filed six items and covered everything shallowly. The reviewer reads the time-and-area map and the artefacts before the scores, writes down their own view of what should have been prioritised, then confirms or overrides each criterion with a reason. Any override on the second criterion must name the area the reviewer believes should have been covered.
What this does not measure
This design observes one candidate, on one application, for a single twenty-five-minute window, and the role's value is delivered over a release cycle. It does not measure stamina across a two-week regression period, nor the political durability of holding an unpopular position on the fourth release in a row rather than the first, nor whether the person is still raising uncomfortable findings after a year of being overruled. Those are the attributes that actually determine whether a QA hire changes anything, and they are visible in references and in probation, not in a session. It also does not observe behaviour during a live production incident with customers affected, which for QA takes the form of triage under real consequence, and no part of this design approximates the stakes. Three narrower cautions. The domain must be one the candidate can reason about without specialist knowledge — a storefront is chosen precisely because everyone has used one — since a domain-heavy scenario measures prior sector experience and silently favours candidates from the buyer's own industry. The release call must be scored on the content of the risk statement and never on assertiveness, accent, register or how comfortable the candidate sounds disagreeing with authority, all of which vary with culture and personality rather than with competence; reviewers get the transcript alongside the recording for that reason, and a quietly precise candidate must be able to score five. And the fixed budget is the point of the design but is also a filter — extended time should be offered as an adjustment on request, with the window scaled rather than removed so the trade-off still exists, applied before the session and not surfaced to the reviewer. Monitor score distributions by candidate background and by requested adjustment, and treat a consistent gap as a defect in the scenario.
The standard QA screen asks a candidate to list test cases for a login form, and in doing so measures the exact inverse of the job. Listing is additive; the competence is subtractive. A candidate who produces forty cases scores well on a task whose real-world equivalent would be a serious misuse of two days, and the prompt is also self-contained enough that it now has a canonical answer available instantly to anyone. This design starts by removing the possibility of scoring on volume: the sandbox contains more surface than the window covers, so every minute spent somewhere is a minute not spent elsewhere, and the artefact that matters is the shape of the choice.
The centrepiece is the silent case. A discount code can now be deactivated while carts are open, and nothing anywhere says what should happen to a discount already applied. The implementation keeps it, not because anyone decided that, but because that is what the code happened to do. This is the most common and least screened situation in quality work — not a defect, not a requirement, but a decision nobody realised they were making. Three responses are possible and only one is right. Treating the implemented behaviour as the specification ratifies an accident. Filing it as a bug substitutes the tester's guess for the product owner's judgement and usually gets closed as working-as-intended, which teaches the tester to stop raising things. Writing it up as a question, with both answers costed and the affected customers named, is the response that gets a real decision made and is why organisations employ testers rather than test suites.
Making that task explicitly optional is deliberate and is the hardest part of the design to get right. If the brief instructs the candidate to raise ambiguities, it measures compliance. Left unprompted, the act of noticing and choosing to escalate is the observation, and the candidates who do it are visibly different from the ones who do not.
The planted refund defect serves a different purpose: it tests whether the candidate knows that regression risk lives in the neighbourhood of a change rather than in the change itself. The ticket describes discount deactivation. The defect is in refunds, which read the same field and were not mentioned by anyone. A tester who works only from the ticket will find a system that behaves exactly as written and will report a clean pass, and the report will be truthful and useless.
The release call is where the third distinguishing behaviour becomes visible, and it is generated from whatever the candidate actually left uncovered, so it cannot be rehearsed. The AI product manager is not a bully; he has a real customer commitment and a reasonable reading of the evidence, and he asks the question that ends most of these conversations badly — are you blocking this? The useful answer is neither yes nor no. It is the residual risk stated precisely enough that he can make the decision himself and own it, which is the form of influence this role actually has.
What this design refuses to measure is worth stating to buyers plainly. It gives no credit for defining smoke, regression or integration testing, and it does not score the number of defects found, because a candidate who finds one thing that matters is better than one who finds five that do not. It makes no attempt to predict whether someone will still be raising uncomfortable findings in the second year. And it is an assessment of judgement, not of thoroughness — which is the right trade for this role, and the opposite of what most QA screens choose.
Sources
Every figure on this page is traceable. Where a claim could not be sourced it is stated qualitatively instead.
- US Bureau of Labor Statistics, Occupational Outlook Handbook, Software Developers, Quality Assurance Analysts, and Testers, 2025, https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm
- Stack Overflow, 2025 Developer Survey, AI section, roughly 49,000 respondents worldwide, https://survey.stackoverflow.co/2025/ai
- Google Cloud, Announcing the 2025 DORA Report: State of AI-assisted Software Development, 24 September 2025, vendor-run survey of nearly 5,000 technology professionals worldwide, https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
See what the employer actually receives. A full report for one role, with every score shown beside the excerpt that earned it, conduct findings reported rather than averaged, and a reviewer sign-off required before any decision. No form.
Read a sample reportOr talk to us about this role