On this page
A complete working course for engineers who write tests, build test frameworks, or lead test teams. Sixteen modules, about sixty minutes. The modules are independent; the contents list takes you to any of them.
The test pyramid
Many fast unit tests at the base, a few slow end-to-end tests at the top. Everything in this course hangs off it. The shape is the rule: each step up costs more to write, run, and debug, so a test belongs at the lowest level that can actually prove the behaviour.
| Tier (top → bottom) | What lives there | Cost |
|---|---|---|
| Acceptance | Customer validates the product meets their need, against agreed criteria, in their environment. | few · slow · costly |
| System / E2E | The whole product, black box, against requirements, including load, failover, security. E2E flows are one technique inside it. | |
| Integration | Two or more real components talking: service to DB, service to broker, service to service. testcontainers, WireMock. The level most often missing. | |
| Component / API | One deployable service with real infrastructure, driven only through its public interface. Postman, REST Assured, Schemathesis, Pact. | |
| Unit | One function or module, dependencies mocked. Milliseconds each, thousands of them. pytest, JUnit, Jest, GoogleTest. | many · fast · cheap |
1. Why we test
Testing exists because software is built from assumptions, and assumptions are wrong at a predictable rate. The job of testing is to find where they are wrong before a user does, and to give the people shipping the software an honest picture of the risk they are taking.
Testing is the process of evaluating a system by exercising it, to find defects and to measure whether it meets its requirements. It shows the presence of bugs, never their absence.
Verification and validation
Verification asks: did we build the thing right? Does the implementation match the specification. Unit, integration, and system tests are mostly verification. Validation asks: did we build the right thing? Does the product meet the user's actual need. Acceptance testing and usability testing are validation. A product can pass every verification test and still fail validation; that is what a well-tested feature nobody wanted looks like.
The cost curve
A defect found while writing the code costs minutes. Found in code review, an hour. Found by an integration test, half a day. Found in system test, days, because the failure no longer names the component. Found by a customer, weeks plus the trust. Every practice in this course is a way to push defect discovery left along that curve.
Seven principles worth internalising
- Testing shows presence of defects, not absence.
- Exhaustive testing is impossible; risk decides where to spend effort.
- Early testing is cheaper testing.
- Defects cluster: a few modules hold most of the bugs.
- Repeating the same tests stops finding new bugs (the pesticide paradox). Suites need refresh.
- Testing is context dependent: a payment system and a game are tested differently.
- A bug-free product can still be the wrong product.
2. What a test framework is
A test framework is the reusable machinery that lets you write, organise, run, and report tests without rebuilding the plumbing each time. Tests are the content; the framework is the structure they plug into.
Every framework, from pytest on a single library to a 40-person hardware lab, provides the same six things.
Discovery and structure
How tests are found, grouped, tagged, selected. Conventions like test_*.py, markers, suites. e.g. pytest collection, JUnit runners, Robot suites
Fixtures and lifecycle
Setup and teardown at the right scope: per test, per module, per session. Spinning up a database, a container, a topology. e.g. pytest fixtures, JUnit @BeforeEach, testcontainers
Assertions and oracles
How a test decides pass or fail. Plain equality for units; eventually() and statistical windows for distributed systems. e.g. assert, Hamcrest, AssertJ, custom async matchers
Drivers and abstraction
How tests talk to the thing under test: browser, HTTP client, device CLI, protocol stack. The layer that lets one test run against a simulator or real hardware. e.g. Selenium WebDriver, Playwright, requests, Netmiko
Execution and orchestration
Parallelism, retries, ordering, timeouts, environment provisioning, CI hooks. e.g. pytest-xdist, GitHub Actions, Jenkins, Tekton
Reporting and artifacts
Results, history, flakiness trends, and attached evidence: logs, screenshots, pcaps, core dumps. e.g. Allure, ReportPortal, JUnit XML, TestRail
Two terms get mixed up with it. A test runner is only the execution piece (pytest is both a runner and a framework). A test harness is the framework plus the environment, stubs, simulators, and data needed to run a suite end to end. In distributed systems the harness is where most of the engineering goes.
Designing one: the order that works
- Driver abstraction first. Tests express intent ("register a user", "cut the link"), never raw protocol. Each driver has a simulated and a real backend behind the same interface. Everything else can be retrofitted; this cannot.
- Topology as code. One declarative spec for the system under test drives local, CI, and lab provisioning. Tests declare what they need; the harness provides or claims it.
- Deterministic identity. Every run gets an ID; every entity created carries it. Reruns are idempotent and teardown is one predicate.
- Async oracles.
eventually(predicate, timeout),always(predicate, duration),observed(events, matcher, within). Forbidsleep(). - Fault injection as an API. If chaos is a separate manual exercise it never runs.
- Evidence on failure by default. A red result with no artifacts is a red result nobody will debug.
3. Levels: the pyramid
Levels answer how much of the system is under test. Each level up integrates more, runs slower, and fails for more possible reasons.
Unit
One function, class, or module, in isolation, dependencies mocked or stubbed. Milliseconds each, thousands of them, run on every save. Owned by developers.
On coverage. 80% line coverage is the figure most teams quote; 60% is a floor. Coverage measures lines executed, not behaviour verified: a test can hit 100% of lines and assert nothing. Treat it as a detector for untested modules, never as a quality score. Branch coverage is more honest than line coverage; mutation testing (module 9) is more honest than both.
Tools: pytest, unittest (Python); JUnit 5, TestNG (Java); Jest, Vitest (JS/TS); Go testing; GoogleTest, Catch2 (C/C++). Mocking: unittest.mock, Mockito, gMock. Coverage: coverage.py, JaCoCo, gcov/lcov.
Integration
Two or more real components talking: service to database, service to message broker, two services over their real API. Catches the bugs unit tests structurally cannot: wrong schema, wrong serialisation, wrong assumption about ordering. This is the level most often missing from a beginner's mental model, and the one that catches the most bugs per test in distributed systems.
Tools: testcontainers (real Postgres/Redis/Kafka in Docker), WireMock/MockServer for external HTTP, Pact for contract tests, docker-compose for small topologies.
Component and API
One deployable service, real infrastructure, exercised only through its public interface. API testing lives here: is the endpoint correct, does it validate input, reject bad auth, return the documented schema and status codes, behave under retries. Contract testing is the specialised form that checks producer and consumer agree on the interface without running both.
Tools: Postman/Newman, REST Assured, requests + pytest, Karate, Schemathesis (fuzz from OpenAPI), Pact, Dredd.
System
The complete integrated product, deployed in a production-like configuration, tested as a black box against its requirements. Includes non-functional requirements: performance, failover, security, capacity. Done by the vendor before release. Simulators replace things outside the product, never things inside it.
End-to-end (E2E)
A complete user or business flow through the real system: log in, do the thing, see the result. E2E is a technique used inside system testing, not a synonym for it. System testing also includes load, failover, and security runs that are not flows. Keep E2E suites small: they are slow, flaky, and expensive to debug, so they cover critical paths only.
Tools: Selenium, Playwright, Cypress (web); Appium (mobile); Robot Framework (keyword-driven); custom harnesses over device APIs.
Acceptance (UAT)
The customer or product owner validates the system meets their need, usually in their environment or a staging copy of it, against pre-agreed criteria that are often contractual. System test proves the product meets its spec; acceptance proves the spec was the right spec. Variants: business acceptance, operational acceptance (can ops run it), alpha and beta.
4. Types by purpose
Types answer what property are we checking. A type is not a level: functional testing happens at every level from unit to acceptance. The two axes are independent, and conflating them is the single most common confusion in test vocabulary.
Functional
Functional
Does it do what the requirements say. Input in, expected output out. A unit test of a parser is functional; an E2E checkout flow is functional; a load test is not. Any level. Same tools as the level it runs at.
Regression
Re-running existing tests to confirm a change broke nothing. A reason for running tests, not a separate kind. The automated suite in CI is the regression suite. e.g. CI on every merge, nightly full runs
Smoke
Fast, shallow pass over the most important paths, run first. Answers one question: is this build worth testing further? Minutes long; a failure blocks everything downstream. e.g. 10 to 30 tagged tests, post-deploy
Sanity
Narrow, deep check that a specific fix or feature works before the full regression. Smoke is wide and shallow; sanity is narrow and focused. Define both in the test plan so nobody argues. e.g. targeted subset by ticket
Exploratory
Unscripted, skilled manual testing: design and execute in the same session, following the product's behaviour. Finds the bugs no script anticipated. The highest-value form of manual testing. e.g. session-based test management, charters
Installation, upgrade, recovery
Fresh install, upgrade from N-1 and N-2, downgrade, rollback, restore from backup, restart mid-operation. Routinely skipped, routinely where release-day incidents come from. e.g. versioned environment snapshots, scripted upgrade paths
Non-functional
Performance: load, stress, soak, spike
Load: expected traffic, are latency and error budgets met. Stress: push past capacity, where does it break and does it recover. Soak: normal load for days, catches leaks and drift. Spike: sudden surge. e.g. k6, JMeter, Locust, Gatling
Security
Vulnerability scanning, dependency audit, dynamic scans, penetration testing, auth and authz checks, secrets detection. Shift-left versions run in CI; pentests run pre-release by specialists. e.g. OWASP ZAP, Burp, Trivy, Semgrep, Snyk
Resilience and chaos
Inject failures deliberately: kill a pod, partition the network, add latency, skew a clock, fill a disk. Verify the system degrades the way the design says. Most production bugs in distributed systems are recovery bugs. e.g. Chaos Mesh, Litmus, Gremlin, tc netem, Toxiproxy
Compatibility, usability, accessibility
Browsers, OS versions, devices, screen readers, locales. Usability is qualitative and human-run; accessibility has automatable parts. e.g. BrowserStack, Sauce Labs, axe-core, Lighthouse
Conformance and interoperability
Follows the standard; works with other implementations. Central in telecom, payments, healthcare, and anything protocol-driven. Keep separate from functional suites: different owners, cadence, and meaning on failure. e.g. standards test suites, certification labs, plugfests
5. How tests get executed
Manual
A human executes the test, from a written script (scripted manual) or by exploring (exploratory). Scripted manual testing is what you should automate. Exploratory is what you cannot, and it is where skilled testers earn their keep: new features, UX judgement, and "what happens if I do this stupid thing". Manual is not lesser; repetitive manual is wasteful.
Automation
Tests executed by code. "Automating manual tests" is the common origin story but too narrow: load tests, chaos tests, property-based tests, and soak runs were never manual and never could be. Automation is the default for anything that runs more than a few times. The trap is automating a flaky suite and calling it done: automation only pays when the tests are reliable enough that a red result means something.
Black, white, and grey box
Black box: test through the external interface, no knowledge of internals (system, acceptance). White box: full knowledge of code and structure (unit, coverage-driven). Grey box: partial knowledge, such as using logs and metrics from inside the system to assert on externally triggered behaviour. Good system testers work grey box.
Static and dynamic
Everything above is dynamic: the code runs. Static testing examines code without running it: linting, type checking, static analysis, code review, requirement and design review. The cheapest bugs you will ever find.
Tools: mypy, pyright, ESLint, golangci-lint, clang-tidy, SonarQube, Semgrep, CodeQL.
Practices: TDD, BDD, shift-left
TDD: write the failing test first, then the code, then refactor. A design discipline more than a testing one. BDD: specify behaviour in structured natural language (Given/When/Then) that executes as tests; useful when product and QA co-own the spec, noisy when only engineers read it. Shift-left: move testing earlier, into design and development, so defects are found where they are cheap. The pyramid is shift-left made into a shape.
Tools: Cucumber, Behave, SpecFlow, pytest-bdd.
6. Test design techniques
Test design is how you get from a requirement to a finite set of cases that catch the defects worth catching. Without technique you get either ten thousand redundant cases or five that miss the boundaries. These are the ones that carry the load.
Equivalence partitioning and boundary value analysis
Divide the input space into classes the system should treat identically (partitions), test one value per class, then test the edges of each class, because off-by-one errors live at boundaries. Take an age field that accepts 18 to 65:
| Value | Partition | Expected |
|---|---|---|
| 17 | just below the lower boundary | rejected |
| 18 | lower boundary | accepted |
| 19 | just above the lower boundary | accepted |
| 40 | mid-range | accepted |
| 64 | just below the upper boundary | accepted |
| 65 | upper boundary | accepted |
| 66 | just above the upper boundary | rejected |
Minimum cases from this one field: 17, 18, 19, 64, 65, 66, plus one mid-range and the invalid types (empty, negative, non-numeric).
Decision tables
When behaviour depends on combinations of conditions, tabulate every combination and its expected action. Forces you to ask what happens for the combinations nobody specified.
| Logged in | Has subscription | Trial expired | Expected |
|---|---|---|---|
| No | – | – | Redirect to login |
| Yes | Yes | – | Full access |
| Yes | No | No | Trial access with banner |
| Yes | No | Yes | Paywall |
State transition testing
Model the object as states and transitions; test every valid transition, then every invalid one (an event arriving in a state that should ignore or reject it). Use it for anything with a lifecycle: orders, sessions, connections, protocol state machines. The invalid transitions are where the bugs are.
Pairwise (combinatorial)
When you have many configuration parameters, testing every combination is impossible. Most defects involve one or two parameters interacting, so a set that covers every pair of values catches most of them at a fraction of the cost. Six parameters with three values each is 729 combinations exhaustively and around 15 pairwise.
Tools: PICT, AllPairs, pairwise generators in most languages.
Use-case and scenario testing
Walk the real user journeys end to end, including the unhappy paths: the cancelled order, the timeout mid-payment, the retry. This is where E2E cases come from, and where you find that the individually correct components do not add up to a correct flow.
Error guessing and checklists
Experience-based: empty inputs, maximum lengths, Unicode, concurrent edits, clock changes, leap days, double-clicks, back button, network loss mid-request. Keep a team checklist and grow it from every escaped defect.
Risk-based prioritisation
You cannot run everything. Rank features by likelihood of failure (complexity, churn, history) times impact of failure (revenue, safety, reputation) and spend depth where the product is high on both. The risk matrix is the argument you make when someone asks why a test was not run.
7. Strategy and planning
The test life cycle
Requirement analysis → Test planning → Case design → Environment setup → Execution → Defect cycle → Closure and report
The stages overlap in any real team. The point of naming them is that each produces an artifact somebody else depends on, and skipping one is where the surprises come from.
Strategy versus plan
A test strategy is the organisation-level or product-level statement of approach: which levels and types apply, what is automated, what environments exist, what the entry and exit criteria are, who owns what. It changes rarely. A test plan is the release-level or feature-level instance: scope, schedule, estimates, resources, risks, and the specific criteria for this release. It changes every cycle.
What a plan has to contain
- Scope: what is in and, explicitly, what is out.
- Approach: levels, types, techniques, and the manual/automated split per area.
- Entry criteria: what must be true before testing starts (build passes smoke, environment ready, test data loaded).
- Exit criteria: what must be true to stop (pass rate, no open severity-1, coverage of priority-1 requirements).
- Environments and data: what exists, who owns it, how it is reset.
- Estimates and schedule: with the assumptions written down.
- Risks and mitigations: including the risk that testing is cut short.
- Roles: who designs, who executes, who triages, who signs off.
Traceability
Every requirement maps to the cases that cover it; every case maps back to a requirement or a risk. The traceability matrix is how you answer "is feature X tested" in ten seconds and how you notice a requirement with zero cases before the release, not after.
Estimation
Count cases by type, apply a per-case execution time by level (a unit case is seconds, an E2E case with setup is minutes, a manual exploratory session is an hour), add environment setup, add a defect-retest allowance of 20 to 30 percent, add a contingency. Write the assumptions next to the number. Estimates without assumptions are guesses with a decimal point.
Gating
Pre-merge: static, unit, integration, plus API tests on the touched service. Under 10 minutes or people stop running it. Post-merge and nightly: full system suite plus chaos. Weekly or pre-release: soak, load, security scans, compatibility. Anything above integration that goes flaky gets quarantined with an owner and a deadline, not a retry decorator.
8. Environments and data
Most large test efforts fail here rather than in test design. An environment nobody can reproduce, data that drifts, a shared staging server three teams fight over: these produce more false failures than any amount of bad assertions.
Environment tiers
| Tier | What it is | Who uses it | Lifetime |
|---|---|---|---|
| Local | Developer machine, containers for dependencies | Unit and integration | Minutes |
| Ephemeral CI | Fresh environment per pipeline run | Integration, API, smoke | One run |
| Shared staging | Production-like, long-lived | System, E2E, exploratory | Weeks, reset on schedule |
| Performance | Production-sized, isolated | Load, soak, stress | Reserved per run |
| Pre-production / UAT | Mirror of production with production data shape | Acceptance, release rehearsal | Per release |
Rules that keep environments trustworthy
- Provision from code. If it cannot be rebuilt from a repo, it will drift.
- Isolate per run where possible: a namespace, a tenant, a database schema per pipeline.
- Reserve shared and scarce environments through a queue, not a chat message.
- Pin versions of every dependency the environment includes, including test tools and simulators.
- Reset on a schedule and after every destructive test.
Test data
Synthetic data generated per run beats copied production data: it is deterministic, it has no privacy exposure, and it can be shaped to the boundaries you are testing. Where production shape matters (performance, migrations), use anonymised subsets with referential integrity preserved. Every entity a test creates carries the run ID so cleanup is one query and reruns never collide.
Tools: factory_boy, Faker, Bogus, dbt seeds, testcontainers, Kubernetes namespaces, Terraform or Pulumi for environments.
9. Specialised techniques
Not new levels or types, but methods that find more with less.
Property-based testing
Instead of fixed examples, state a property (decode(encode(x)) == x for all x) and let the tool generate thousands of inputs, then shrink failures to the smallest reproducer. Excellent for parsers, codecs, and data pipelines. e.g. Hypothesis, QuickCheck, fast-check, jqwik
Mutation testing
Deliberately break the code (flip a condition, drop a line) and check whether the tests notice. Surviving mutants show where coverage is fake. The honest answer to "is 80% coverage good". e.g. mutmut, Stryker, PIT
Fuzzing
Feed malformed and random input at an interface to find crashes and undefined behaviour. Standard for anything that parses untrusted input. e.g. AFL++, libFuzzer, boofuzz, Schemathesis for APIs
Contract testing
Consumer writes what it expects from the provider; provider verifies it satisfies every consumer's contract. Replaces most cross-service integration tests in microservice estates. e.g. Pact, Spring Cloud Contract
Snapshot and visual regression
Store the rendered output and diff on every run. Cheap for UI and serialised structures, noisy if used for everything. e.g. Jest snapshots, Percy, Chromatic, Playwright screenshots
Test impact analysis
Map tests to the code they exercise and run only the affected subset on a change. Keeps pre-merge under the ten-minute line as the suite grows. e.g. pytest-testmon, Bazel test selection, Launchable
10. Defects and triage
Lifecycle
New → Triaged → Assigned → Fixed → Ready for retest → Verified → Closed
Side exits: Rejected (not a bug, or by design), Duplicate, Deferred (real, not this release), Cannot reproduce. A defect that bounces between Fixed and Reopened more than once is a signal the root cause was never found.
Severity versus priority
Severity is technical impact: how badly does it break the product. Priority is business urgency: how soon must it be fixed. They are independent. A typo in the legal footer is low severity, high priority. A crash in a feature no customer has enabled is high severity, low priority.
| High priority | Medium priority | Low priority | |
|---|---|---|---|
| Critical severity | Stop the release | Fix before release | Fix, may ship with workaround |
| Major severity | Fix before release | Fix this release | Next release |
| Minor severity | Fix this release | Backlog | Backlog |
What a good defect report contains
- Title that states the failure, not the feature: "Checkout returns 500 when coupon expired mid-session", not "Checkout bug".
- Environment and build identifiers.
- Minimal reproduction steps, with the data used.
- Expected versus actual, with the requirement referenced.
- Evidence attached: logs, screenshots or recordings, packet captures, core dumps, API responses.
- Frequency: always, intermittent with a rate, once.
- Impact assessment: which users, which flows, workaround if any.
The test is whether a developer can act on it without re-running the test. If they have to ask a question first, the report is incomplete.
Triage
A short, regular meeting of test, development, and product that assigns severity and priority, confirms ownership, and decides deferrals. Its output is a defect list where every item has an owner and a target. Its failure mode is becoming a status meeting; keep it to decisions.
Root cause analysis
For every escaped defect and every severity-1, ask why it was not caught earlier and which level should have caught it. The answer becomes a new test at that level, or a change to the strategy. Without this loop the same class of bug escapes every release.
11. Metrics and reporting
Metrics exist to support decisions, mostly one decision: is this release ready. Anything that does not feed a decision is decoration.
| Metric | What it tells you | What it does not |
|---|---|---|
| Requirement coverage | Which requirements have cases, and which have passed | Whether the cases are good |
| Code coverage | Which code no test executes | Whether executed code is verified |
| Pass rate | Trend across builds; a drop is a signal | Anything on its own: 95% with a severity-1 open is not ready |
| Defect density | Defects per module or per KLOC; where the clusters are | Quality across teams with different reporting habits |
| Defect removal efficiency | Share of defects found before release versus after | Improves only with post-release tracking |
| Open defects by severity, ageing | Release risk and whether triage is working | – |
| Flaky rate | Share of tests that fail then pass on rerun; suite trustworthiness | – |
| Time to feedback | Commit to first red; whether people will wait for the pipeline | – |
The test summary report
One page. Scope tested and not tested, execution totals by level and type, pass rate with trend, open defects by severity with the top risks in a sentence each, environment and known limitations, and a recommendation: ready, ready with conditions, not ready. The recommendation is the point; the numbers are the evidence for it.
12. Tools and stack
A reference stack by layer. Pick one per row; maintaining two in the same row is a common way to end up with neither trusted.
| Layer | Python | JVM | JS / TS | Go / C / C++ |
|---|---|---|---|---|
| Unit | pytest, unittest, unittest.mock | JUnit 5, TestNG, Mockito | Jest, Vitest, Mocha | go test, testify; GoogleTest, Catch2 |
| Integration | testcontainers-python, docker-compose | Testcontainers, Spring Test | testcontainers-node, supertest | testcontainers-go |
| API / contract | requests, httpx, Schemathesis, Pact | REST Assured, Karate, Pact JVM | Playwright request, Pact JS | net/http/httptest, gRPC test clients |
| UI / E2E | Playwright, Selenium, Robot Framework | Selenium, Selenide | Playwright, Cypress | Playwright via bindings |
| Load / perf | Locust, k6 | JMeter, Gatling | k6, Artillery | vegeta, wrk, k6 |
| Property / mutation | Hypothesis, mutmut | jqwik, PIT | fast-check, Stryker | rapid, libFuzzer |
| Static | mypy, ruff, pylint | SpotBugs, Checkstyle, SonarQube | ESLint, tsc | golangci-lint, clang-tidy, cppcheck |
Four layers are the same whatever the language:
- Chaos / resilience: Chaos Mesh, Litmus, Gremlin, Toxiproxy, tc netem
- Security: OWASP ZAP, Burp Suite, Trivy, Semgrep, Snyk, CodeQL
- Test management: Jira with Zephyr or Xray, TestRail, qTest; traceability lives here
- CI / reporting: GitHub Actions, GitLab CI, Jenkins, Tekton; Allure, ReportPortal, JUnit XML
Selenium versus Playwright versus Cypress, since it always comes up: Selenium is the oldest, broadest language support, slowest, most flaky. Playwright is the current default for new work: fast, auto-waiting, multi-browser, Python and TS. Cypress is JS-only with an excellent developer experience; it runs inside the browser, so some cross-origin and multi-tab cases are awkward.
13. Run a pipeline
A CI gate in practice: fast and cheap first, slow and expensive last, every stage a gate for the next. Run it clean, then inject a bug at different levels and watch where it is caught and what the failure message can say.
- Static — lint, types, secrets
- Unit — 2,400 tests, 40s
- Integration — real DB and broker
- API / contract — schema + Pact
- Smoke — deploy to staging, 20 checks
- System / E2E — flows + failover
- Load — k6, 10 min
Try it in your head with three injected bugs: a broken function, a schema mismatch between two services, and a failover that does not converge. The first stops at stage 2, the second at stage 3 or 4, and the third gets all the way to stage 6.
The cost of catching each bug: the unit bug was found in under a minute by a test that names the exact function. The failover bug survived five stages and was found by a test that can only say "the cluster did not converge within 30 seconds", which is where the next module begins.
14. Debugging a failure
When a system-level test goes red, the failure rarely names the culprit. The discipline is elimination in a fixed order, with evidence captured at each step.
- Reproduce. Same build, same environment, same data. If it does not reproduce, record the rate and move to intermittent-failure handling: more runs, tighter capture, timing analysis.
- Is it the product, the harness, or the environment? Check the test's own logs, the environment's health, recent infra changes. Roughly a third of system-test reds in a young suite are not product bugs. Rule those out before opening a defect.
- Localise by layer. Application logs, then OS (processes, sockets, permissions, resources), then configuration, then dependencies, then network. Each layer has its own tools; do not skip a layer because the symptom looks like a later one.
- Bisect. Which commit, which config change, which dependency bump.
git bisectwith the failing test as the oracle turns a day of guessing into an hour. - Capture. Logs, packet captures, API responses, resource graphs, core dumps, config snapshots. Attach them before the environment is reused and the evidence is gone.
- Report. The output is a defect a developer can act on without re-running anything: environment, minimal repro, expected versus actual, evidence, frequency, impact.
The Linux toolkit for step 3
journalctl and application logs; ps, top, /proc/<pid>/; ss and lsof for sockets and files; strace for what a process is actually asking the kernel; tcpdump and Wireshark at the network seam; free, vmstat, iostat, df for resources; dmesg for the kernel's view. For crashes, gdb on the core dump (bt, thread apply all bt) and know how cores are produced in containers (ulimit -c, core_pattern). For memory growth in a soak run, an ASan/LSan build or valgrind memcheck on a reduced-scale reproduction, since valgrind is 10 to 50 times slower.
15. Check yourself
Eight questions. Decide before you open the answer.
A test spins up a real Postgres in Docker, calls the service's repository layer, and checks that a row was written with the right schema. What is it? Unit test · Integration test · E2E test
Answer
Integration test. Real dependency, one component boundary, no user flow: integration.
After each deploy to staging, 15 tests run in three minutes covering login, one purchase, and the health endpoint. What are they? Sanity tests · Regression suite · Smoke tests
Answer
Smoke tests. Wide, shallow, fast, gating: smoke. Sanity would be narrow and about a specific change.
Functional testing is: A type that applies at every level · A subset of E2E testing · Another name for system testing
Answer
A type that applies at every level. Types (what property) and levels (how much system) are independent axes.
An input accepts 1 to 100. Which set is the minimum boundary-value coverage? 50 · 1 and 100 · 0, 1, 2, 99, 100, 101
Answer
0, 1, 2, 99, 100, 101. Each boundary and its neighbours on both sides. Off-by-one errors live exactly there.
A typo in the pricing page headline is: High severity, high priority · Low severity, high priority · Low severity, low priority
Answer
Low severity, high priority. Technically trivial, commercially urgent. Severity and priority are independent.
The customer's team runs the pre-agreed acceptance criteria in their own environment before signing off. This is: System testing · User acceptance testing · Conformance testing
Answer
User acceptance testing. Customer, their need, their environment, sign-off: UAT.
Your suite reports 92% line coverage. What does that tell you? Which lines never execute under test; nothing about assertion quality · The code is 92% correct · You can stop writing tests
Answer
Which lines never execute under test; nothing about assertion quality. Coverage is execution, not verification. Mutation testing measures whether tests would notice a bug.
A soak test running normal traffic for two weeks shows memory growing 40 MB a day. Which tool proves whether it's a leak and where? k6 · Selenium · valgrind or ASan on a reduced-scale repro
Answer
valgrind or ASan on a reduced-scale repro. Memory instrumentation on a debug build. k6 generates load; it does not localise leaks.
16. Glossary
- Acceptance testing — Validation by the customer or product owner against agreed criteria. UAT when done by users.
- Boundary value analysis — Testing at and around the edges of input partitions.
- Contract test — Verifies producer and consumer agree on an interface without running both together.
- Defect removal efficiency — Share of total defects found before release.
- Entry / exit criteria — Conditions that must hold to start, and to stop, a test phase.
- Equivalence partition — A class of inputs the system should treat identically.
- Flaky test — A test that passes and fails on the same code without change. Quarantine, own, fix.
- Harness — Framework plus environment, stubs, simulators, and data needed to run a suite.
- Integration test — Two or more real components exercised across their real interface.
- Mutation testing — Injecting faults into code to check whether tests detect them.
- Oracle — The mechanism a test uses to decide pass or fail.
- Regression — Re-running existing tests to confirm a change broke nothing.
- Sanity test — Narrow, deep check of a specific change before full regression.
- Smoke test — Fast, shallow check of critical paths to decide whether a build is worth further testing.
- Soak test — Sustained normal load over days to find leaks and drift.
- System test — Whole integrated product, black box, against requirements, including non-functional.
- Traceability — Mapping between requirements and the tests that cover them.
- Verification / validation — Built it right / built the right thing.
End of course. The modules are independent; return to any of them from the contents.
