Skip to content
echogeco
Blog

· 20 min read

Software testing, end to end

A working course in one post: every level, every type, how to design cases, how to plan, how to triage, and the tools people actually use.

testingqualityengineering
Out-of-focus source code in an editor on a dark monitor
On this page

A complete working course for engineers who write tests, build test frameworks, or lead test teams. Sixteen modules, about sixty minutes. The modules are independent; the contents list takes you to any of them.

The test pyramid

Many fast unit tests at the base, a few slow end-to-end tests at the top. Everything in this course hangs off it. The shape is the rule: each step up costs more to write, run, and debug, so a test belongs at the lowest level that can actually prove the behaviour.

Tier (top → bottom)What lives thereCost
AcceptanceCustomer validates the product meets their need, against agreed criteria, in their environment.few · slow · costly
System / E2EThe whole product, black box, against requirements, including load, failover, security. E2E flows are one technique inside it.
IntegrationTwo or more real components talking: service to DB, service to broker, service to service. testcontainers, WireMock. The level most often missing.
Component / APIOne deployable service with real infrastructure, driven only through its public interface. Postman, REST Assured, Schemathesis, Pact.
UnitOne function or module, dependencies mocked. Milliseconds each, thousands of them. pytest, JUnit, Jest, GoogleTest.many · fast · cheap

1. Why we test

Testing exists because software is built from assumptions, and assumptions are wrong at a predictable rate. The job of testing is to find where they are wrong before a user does, and to give the people shipping the software an honest picture of the risk they are taking.

Testing is the process of evaluating a system by exercising it, to find defects and to measure whether it meets its requirements. It shows the presence of bugs, never their absence.

Verification and validation

Verification asks: did we build the thing right? Does the implementation match the specification. Unit, integration, and system tests are mostly verification. Validation asks: did we build the right thing? Does the product meet the user's actual need. Acceptance testing and usability testing are validation. A product can pass every verification test and still fail validation; that is what a well-tested feature nobody wanted looks like.

The cost curve

A defect found while writing the code costs minutes. Found in code review, an hour. Found by an integration test, half a day. Found in system test, days, because the failure no longer names the component. Found by a customer, weeks plus the trust. Every practice in this course is a way to push defect discovery left along that curve.

Seven principles worth internalising

  • Testing shows presence of defects, not absence.
  • Exhaustive testing is impossible; risk decides where to spend effort.
  • Early testing is cheaper testing.
  • Defects cluster: a few modules hold most of the bugs.
  • Repeating the same tests stops finding new bugs (the pesticide paradox). Suites need refresh.
  • Testing is context dependent: a payment system and a game are tested differently.
  • A bug-free product can still be the wrong product.

2. What a test framework is

A test framework is the reusable machinery that lets you write, organise, run, and report tests without rebuilding the plumbing each time. Tests are the content; the framework is the structure they plug into.

Every framework, from pytest on a single library to a 40-person hardware lab, provides the same six things.

Discovery and structure

How tests are found, grouped, tagged, selected. Conventions like test_*.py, markers, suites. e.g. pytest collection, JUnit runners, Robot suites

Fixtures and lifecycle

Setup and teardown at the right scope: per test, per module, per session. Spinning up a database, a container, a topology. e.g. pytest fixtures, JUnit @BeforeEach, testcontainers

Assertions and oracles

How a test decides pass or fail. Plain equality for units; eventually() and statistical windows for distributed systems. e.g. assert, Hamcrest, AssertJ, custom async matchers

Drivers and abstraction

How tests talk to the thing under test: browser, HTTP client, device CLI, protocol stack. The layer that lets one test run against a simulator or real hardware. e.g. Selenium WebDriver, Playwright, requests, Netmiko

Execution and orchestration

Parallelism, retries, ordering, timeouts, environment provisioning, CI hooks. e.g. pytest-xdist, GitHub Actions, Jenkins, Tekton

Reporting and artifacts

Results, history, flakiness trends, and attached evidence: logs, screenshots, pcaps, core dumps. e.g. Allure, ReportPortal, JUnit XML, TestRail

Two terms get mixed up with it. A test runner is only the execution piece (pytest is both a runner and a framework). A test harness is the framework plus the environment, stubs, simulators, and data needed to run a suite end to end. In distributed systems the harness is where most of the engineering goes.

Designing one: the order that works

  1. Driver abstraction first. Tests express intent ("register a user", "cut the link"), never raw protocol. Each driver has a simulated and a real backend behind the same interface. Everything else can be retrofitted; this cannot.
  2. Topology as code. One declarative spec for the system under test drives local, CI, and lab provisioning. Tests declare what they need; the harness provides or claims it.
  3. Deterministic identity. Every run gets an ID; every entity created carries it. Reruns are idempotent and teardown is one predicate.
  4. Async oracles. eventually(predicate, timeout), always(predicate, duration), observed(events, matcher, within). Forbid sleep().
  5. Fault injection as an API. If chaos is a separate manual exercise it never runs.
  6. Evidence on failure by default. A red result with no artifacts is a red result nobody will debug.

3. Levels: the pyramid

Levels answer how much of the system is under test. Each level up integrates more, runs slower, and fails for more possible reasons.

Unit

One function, class, or module, in isolation, dependencies mocked or stubbed. Milliseconds each, thousands of them, run on every save. Owned by developers.

On coverage. 80% line coverage is the figure most teams quote; 60% is a floor. Coverage measures lines executed, not behaviour verified: a test can hit 100% of lines and assert nothing. Treat it as a detector for untested modules, never as a quality score. Branch coverage is more honest than line coverage; mutation testing (module 9) is more honest than both.

Tools: pytest, unittest (Python); JUnit 5, TestNG (Java); Jest, Vitest (JS/TS); Go testing; GoogleTest, Catch2 (C/C++). Mocking: unittest.mock, Mockito, gMock. Coverage: coverage.py, JaCoCo, gcov/lcov.

Integration

Two or more real components talking: service to database, service to message broker, two services over their real API. Catches the bugs unit tests structurally cannot: wrong schema, wrong serialisation, wrong assumption about ordering. This is the level most often missing from a beginner's mental model, and the one that catches the most bugs per test in distributed systems.

Tools: testcontainers (real Postgres/Redis/Kafka in Docker), WireMock/MockServer for external HTTP, Pact for contract tests, docker-compose for small topologies.

Component and API

One deployable service, real infrastructure, exercised only through its public interface. API testing lives here: is the endpoint correct, does it validate input, reject bad auth, return the documented schema and status codes, behave under retries. Contract testing is the specialised form that checks producer and consumer agree on the interface without running both.

Tools: Postman/Newman, REST Assured, requests + pytest, Karate, Schemathesis (fuzz from OpenAPI), Pact, Dredd.

System

The complete integrated product, deployed in a production-like configuration, tested as a black box against its requirements. Includes non-functional requirements: performance, failover, security, capacity. Done by the vendor before release. Simulators replace things outside the product, never things inside it.

End-to-end (E2E)

A complete user or business flow through the real system: log in, do the thing, see the result. E2E is a technique used inside system testing, not a synonym for it. System testing also includes load, failover, and security runs that are not flows. Keep E2E suites small: they are slow, flaky, and expensive to debug, so they cover critical paths only.

Tools: Selenium, Playwright, Cypress (web); Appium (mobile); Robot Framework (keyword-driven); custom harnesses over device APIs.

Acceptance (UAT)

The customer or product owner validates the system meets their need, usually in their environment or a staging copy of it, against pre-agreed criteria that are often contractual. System test proves the product meets its spec; acceptance proves the spec was the right spec. Variants: business acceptance, operational acceptance (can ops run it), alpha and beta.

4. Types by purpose

Types answer what property are we checking. A type is not a level: functional testing happens at every level from unit to acceptance. The two axes are independent, and conflating them is the single most common confusion in test vocabulary.

Functional

Functional

Does it do what the requirements say. Input in, expected output out. A unit test of a parser is functional; an E2E checkout flow is functional; a load test is not. Any level. Same tools as the level it runs at.

Regression

Re-running existing tests to confirm a change broke nothing. A reason for running tests, not a separate kind. The automated suite in CI is the regression suite. e.g. CI on every merge, nightly full runs

Smoke

Fast, shallow pass over the most important paths, run first. Answers one question: is this build worth testing further? Minutes long; a failure blocks everything downstream. e.g. 10 to 30 tagged tests, post-deploy

Sanity

Narrow, deep check that a specific fix or feature works before the full regression. Smoke is wide and shallow; sanity is narrow and focused. Define both in the test plan so nobody argues. e.g. targeted subset by ticket

Exploratory

Unscripted, skilled manual testing: design and execute in the same session, following the product's behaviour. Finds the bugs no script anticipated. The highest-value form of manual testing. e.g. session-based test management, charters

Installation, upgrade, recovery

Fresh install, upgrade from N-1 and N-2, downgrade, rollback, restore from backup, restart mid-operation. Routinely skipped, routinely where release-day incidents come from. e.g. versioned environment snapshots, scripted upgrade paths

Non-functional

Performance: load, stress, soak, spike

Load: expected traffic, are latency and error budgets met. Stress: push past capacity, where does it break and does it recover. Soak: normal load for days, catches leaks and drift. Spike: sudden surge. e.g. k6, JMeter, Locust, Gatling

Security

Vulnerability scanning, dependency audit, dynamic scans, penetration testing, auth and authz checks, secrets detection. Shift-left versions run in CI; pentests run pre-release by specialists. e.g. OWASP ZAP, Burp, Trivy, Semgrep, Snyk

Resilience and chaos

Inject failures deliberately: kill a pod, partition the network, add latency, skew a clock, fill a disk. Verify the system degrades the way the design says. Most production bugs in distributed systems are recovery bugs. e.g. Chaos Mesh, Litmus, Gremlin, tc netem, Toxiproxy

Compatibility, usability, accessibility

Browsers, OS versions, devices, screen readers, locales. Usability is qualitative and human-run; accessibility has automatable parts. e.g. BrowserStack, Sauce Labs, axe-core, Lighthouse

Conformance and interoperability

Follows the standard; works with other implementations. Central in telecom, payments, healthcare, and anything protocol-driven. Keep separate from functional suites: different owners, cadence, and meaning on failure. e.g. standards test suites, certification labs, plugfests

5. How tests get executed

Manual

A human executes the test, from a written script (scripted manual) or by exploring (exploratory). Scripted manual testing is what you should automate. Exploratory is what you cannot, and it is where skilled testers earn their keep: new features, UX judgement, and "what happens if I do this stupid thing". Manual is not lesser; repetitive manual is wasteful.

Automation

Tests executed by code. "Automating manual tests" is the common origin story but too narrow: load tests, chaos tests, property-based tests, and soak runs were never manual and never could be. Automation is the default for anything that runs more than a few times. The trap is automating a flaky suite and calling it done: automation only pays when the tests are reliable enough that a red result means something.

Black, white, and grey box

Black box: test through the external interface, no knowledge of internals (system, acceptance). White box: full knowledge of code and structure (unit, coverage-driven). Grey box: partial knowledge, such as using logs and metrics from inside the system to assert on externally triggered behaviour. Good system testers work grey box.

Static and dynamic

Everything above is dynamic: the code runs. Static testing examines code without running it: linting, type checking, static analysis, code review, requirement and design review. The cheapest bugs you will ever find.

Tools: mypy, pyright, ESLint, golangci-lint, clang-tidy, SonarQube, Semgrep, CodeQL.

Practices: TDD, BDD, shift-left

TDD: write the failing test first, then the code, then refactor. A design discipline more than a testing one. BDD: specify behaviour in structured natural language (Given/When/Then) that executes as tests; useful when product and QA co-own the spec, noisy when only engineers read it. Shift-left: move testing earlier, into design and development, so defects are found where they are cheap. The pyramid is shift-left made into a shape.

Tools: Cucumber, Behave, SpecFlow, pytest-bdd.

6. Test design techniques

Test design is how you get from a requirement to a finite set of cases that catch the defects worth catching. Without technique you get either ten thousand redundant cases or five that miss the boundaries. These are the ones that carry the load.

Equivalence partitioning and boundary value analysis

Divide the input space into classes the system should treat identically (partitions), test one value per class, then test the edges of each class, because off-by-one errors live at boundaries. Take an age field that accepts 18 to 65:

ValuePartitionExpected
17just below the lower boundaryrejected
18lower boundaryaccepted
19just above the lower boundaryaccepted
40mid-rangeaccepted
64just below the upper boundaryaccepted
65upper boundaryaccepted
66just above the upper boundaryrejected

Minimum cases from this one field: 17, 18, 19, 64, 65, 66, plus one mid-range and the invalid types (empty, negative, non-numeric).

Decision tables

When behaviour depends on combinations of conditions, tabulate every combination and its expected action. Forces you to ask what happens for the combinations nobody specified.

Logged inHas subscriptionTrial expiredExpected
No––Redirect to login
YesYes–Full access
YesNoNoTrial access with banner
YesNoYesPaywall

State transition testing

Model the object as states and transitions; test every valid transition, then every invalid one (an event arriving in a state that should ignore or reject it). Use it for anything with a lifecycle: orders, sessions, connections, protocol state machines. The invalid transitions are where the bugs are.

Pairwise (combinatorial)

When you have many configuration parameters, testing every combination is impossible. Most defects involve one or two parameters interacting, so a set that covers every pair of values catches most of them at a fraction of the cost. Six parameters with three values each is 729 combinations exhaustively and around 15 pairwise.

Tools: PICT, AllPairs, pairwise generators in most languages.

Use-case and scenario testing

Walk the real user journeys end to end, including the unhappy paths: the cancelled order, the timeout mid-payment, the retry. This is where E2E cases come from, and where you find that the individually correct components do not add up to a correct flow.

Error guessing and checklists

Experience-based: empty inputs, maximum lengths, Unicode, concurrent edits, clock changes, leap days, double-clicks, back button, network loss mid-request. Keep a team checklist and grow it from every escaped defect.

Risk-based prioritisation

You cannot run everything. Rank features by likelihood of failure (complexity, churn, history) times impact of failure (revenue, safety, reputation) and spend depth where the product is high on both. The risk matrix is the argument you make when someone asks why a test was not run.

7. Strategy and planning

The test life cycle

Requirement analysis → Test planning → Case design → Environment setup → Execution → Defect cycle → Closure and report

The stages overlap in any real team. The point of naming them is that each produces an artifact somebody else depends on, and skipping one is where the surprises come from.

Strategy versus plan

A test strategy is the organisation-level or product-level statement of approach: which levels and types apply, what is automated, what environments exist, what the entry and exit criteria are, who owns what. It changes rarely. A test plan is the release-level or feature-level instance: scope, schedule, estimates, resources, risks, and the specific criteria for this release. It changes every cycle.

What a plan has to contain

  • Scope: what is in and, explicitly, what is out.
  • Approach: levels, types, techniques, and the manual/automated split per area.
  • Entry criteria: what must be true before testing starts (build passes smoke, environment ready, test data loaded).
  • Exit criteria: what must be true to stop (pass rate, no open severity-1, coverage of priority-1 requirements).
  • Environments and data: what exists, who owns it, how it is reset.
  • Estimates and schedule: with the assumptions written down.
  • Risks and mitigations: including the risk that testing is cut short.
  • Roles: who designs, who executes, who triages, who signs off.

Traceability

Every requirement maps to the cases that cover it; every case maps back to a requirement or a risk. The traceability matrix is how you answer "is feature X tested" in ten seconds and how you notice a requirement with zero cases before the release, not after.

Estimation

Count cases by type, apply a per-case execution time by level (a unit case is seconds, an E2E case with setup is minutes, a manual exploratory session is an hour), add environment setup, add a defect-retest allowance of 20 to 30 percent, add a contingency. Write the assumptions next to the number. Estimates without assumptions are guesses with a decimal point.

Gating

Pre-merge: static, unit, integration, plus API tests on the touched service. Under 10 minutes or people stop running it. Post-merge and nightly: full system suite plus chaos. Weekly or pre-release: soak, load, security scans, compatibility. Anything above integration that goes flaky gets quarantined with an owner and a deadline, not a retry decorator.

8. Environments and data

Most large test efforts fail here rather than in test design. An environment nobody can reproduce, data that drifts, a shared staging server three teams fight over: these produce more false failures than any amount of bad assertions.

Environment tiers

TierWhat it isWho uses itLifetime
LocalDeveloper machine, containers for dependenciesUnit and integrationMinutes
Ephemeral CIFresh environment per pipeline runIntegration, API, smokeOne run
Shared stagingProduction-like, long-livedSystem, E2E, exploratoryWeeks, reset on schedule
PerformanceProduction-sized, isolatedLoad, soak, stressReserved per run
Pre-production / UATMirror of production with production data shapeAcceptance, release rehearsalPer release

Rules that keep environments trustworthy

  • Provision from code. If it cannot be rebuilt from a repo, it will drift.
  • Isolate per run where possible: a namespace, a tenant, a database schema per pipeline.
  • Reserve shared and scarce environments through a queue, not a chat message.
  • Pin versions of every dependency the environment includes, including test tools and simulators.
  • Reset on a schedule and after every destructive test.

Test data

Synthetic data generated per run beats copied production data: it is deterministic, it has no privacy exposure, and it can be shaped to the boundaries you are testing. Where production shape matters (performance, migrations), use anonymised subsets with referential integrity preserved. Every entity a test creates carries the run ID so cleanup is one query and reruns never collide.

Tools: factory_boy, Faker, Bogus, dbt seeds, testcontainers, Kubernetes namespaces, Terraform or Pulumi for environments.

9. Specialised techniques

Not new levels or types, but methods that find more with less.

Property-based testing

Instead of fixed examples, state a property (decode(encode(x)) == x for all x) and let the tool generate thousands of inputs, then shrink failures to the smallest reproducer. Excellent for parsers, codecs, and data pipelines. e.g. Hypothesis, QuickCheck, fast-check, jqwik

Mutation testing

Deliberately break the code (flip a condition, drop a line) and check whether the tests notice. Surviving mutants show where coverage is fake. The honest answer to "is 80% coverage good". e.g. mutmut, Stryker, PIT

Fuzzing

Feed malformed and random input at an interface to find crashes and undefined behaviour. Standard for anything that parses untrusted input. e.g. AFL++, libFuzzer, boofuzz, Schemathesis for APIs

Contract testing

Consumer writes what it expects from the provider; provider verifies it satisfies every consumer's contract. Replaces most cross-service integration tests in microservice estates. e.g. Pact, Spring Cloud Contract

Snapshot and visual regression

Store the rendered output and diff on every run. Cheap for UI and serialised structures, noisy if used for everything. e.g. Jest snapshots, Percy, Chromatic, Playwright screenshots

Test impact analysis

Map tests to the code they exercise and run only the affected subset on a change. Keeps pre-merge under the ten-minute line as the suite grows. e.g. pytest-testmon, Bazel test selection, Launchable

10. Defects and triage

Lifecycle

New → Triaged → Assigned → Fixed → Ready for retest → Verified → Closed

Side exits: Rejected (not a bug, or by design), Duplicate, Deferred (real, not this release), Cannot reproduce. A defect that bounces between Fixed and Reopened more than once is a signal the root cause was never found.

Severity versus priority

Severity is technical impact: how badly does it break the product. Priority is business urgency: how soon must it be fixed. They are independent. A typo in the legal footer is low severity, high priority. A crash in a feature no customer has enabled is high severity, low priority.

High priorityMedium priorityLow priority
Critical severityStop the releaseFix before releaseFix, may ship with workaround
Major severityFix before releaseFix this releaseNext release
Minor severityFix this releaseBacklogBacklog

What a good defect report contains

  • Title that states the failure, not the feature: "Checkout returns 500 when coupon expired mid-session", not "Checkout bug".
  • Environment and build identifiers.
  • Minimal reproduction steps, with the data used.
  • Expected versus actual, with the requirement referenced.
  • Evidence attached: logs, screenshots or recordings, packet captures, core dumps, API responses.
  • Frequency: always, intermittent with a rate, once.
  • Impact assessment: which users, which flows, workaround if any.

The test is whether a developer can act on it without re-running the test. If they have to ask a question first, the report is incomplete.

Triage

A short, regular meeting of test, development, and product that assigns severity and priority, confirms ownership, and decides deferrals. Its output is a defect list where every item has an owner and a target. Its failure mode is becoming a status meeting; keep it to decisions.

Root cause analysis

For every escaped defect and every severity-1, ask why it was not caught earlier and which level should have caught it. The answer becomes a new test at that level, or a change to the strategy. Without this loop the same class of bug escapes every release.

11. Metrics and reporting

Metrics exist to support decisions, mostly one decision: is this release ready. Anything that does not feed a decision is decoration.

MetricWhat it tells youWhat it does not
Requirement coverageWhich requirements have cases, and which have passedWhether the cases are good
Code coverageWhich code no test executesWhether executed code is verified
Pass rateTrend across builds; a drop is a signalAnything on its own: 95% with a severity-1 open is not ready
Defect densityDefects per module or per KLOC; where the clusters areQuality across teams with different reporting habits
Defect removal efficiencyShare of defects found before release versus afterImproves only with post-release tracking
Open defects by severity, ageingRelease risk and whether triage is working–
Flaky rateShare of tests that fail then pass on rerun; suite trustworthiness–
Time to feedbackCommit to first red; whether people will wait for the pipeline–

The test summary report

One page. Scope tested and not tested, execution totals by level and type, pass rate with trend, open defects by severity with the top risks in a sentence each, environment and known limitations, and a recommendation: ready, ready with conditions, not ready. The recommendation is the point; the numbers are the evidence for it.

12. Tools and stack

A reference stack by layer. Pick one per row; maintaining two in the same row is a common way to end up with neither trusted.

LayerPythonJVMJS / TSGo / C / C++
Unitpytest, unittest, unittest.mockJUnit 5, TestNG, MockitoJest, Vitest, Mochago test, testify; GoogleTest, Catch2
Integrationtestcontainers-python, docker-composeTestcontainers, Spring Testtestcontainers-node, supertesttestcontainers-go
API / contractrequests, httpx, Schemathesis, PactREST Assured, Karate, Pact JVMPlaywright request, Pact JSnet/http/httptest, gRPC test clients
UI / E2EPlaywright, Selenium, Robot FrameworkSelenium, SelenidePlaywright, CypressPlaywright via bindings
Load / perfLocust, k6JMeter, Gatlingk6, Artilleryvegeta, wrk, k6
Property / mutationHypothesis, mutmutjqwik, PITfast-check, Strykerrapid, libFuzzer
Staticmypy, ruff, pylintSpotBugs, Checkstyle, SonarQubeESLint, tscgolangci-lint, clang-tidy, cppcheck

Four layers are the same whatever the language:

  • Chaos / resilience: Chaos Mesh, Litmus, Gremlin, Toxiproxy, tc netem
  • Security: OWASP ZAP, Burp Suite, Trivy, Semgrep, Snyk, CodeQL
  • Test management: Jira with Zephyr or Xray, TestRail, qTest; traceability lives here
  • CI / reporting: GitHub Actions, GitLab CI, Jenkins, Tekton; Allure, ReportPortal, JUnit XML

Selenium versus Playwright versus Cypress, since it always comes up: Selenium is the oldest, broadest language support, slowest, most flaky. Playwright is the current default for new work: fast, auto-waiting, multi-browser, Python and TS. Cypress is JS-only with an excellent developer experience; it runs inside the browser, so some cross-origin and multi-tab cases are awkward.

13. Run a pipeline

A CI gate in practice: fast and cheap first, slow and expensive last, every stage a gate for the next. Run it clean, then inject a bug at different levels and watch where it is caught and what the failure message can say.

  1. Static — lint, types, secrets
  2. Unit — 2,400 tests, 40s
  3. Integration — real DB and broker
  4. API / contract — schema + Pact
  5. Smoke — deploy to staging, 20 checks
  6. System / E2E — flows + failover
  7. Load — k6, 10 min

Try it in your head with three injected bugs: a broken function, a schema mismatch between two services, and a failover that does not converge. The first stops at stage 2, the second at stage 3 or 4, and the third gets all the way to stage 6.

The cost of catching each bug: the unit bug was found in under a minute by a test that names the exact function. The failover bug survived five stages and was found by a test that can only say "the cluster did not converge within 30 seconds", which is where the next module begins.

14. Debugging a failure

When a system-level test goes red, the failure rarely names the culprit. The discipline is elimination in a fixed order, with evidence captured at each step.

  1. Reproduce. Same build, same environment, same data. If it does not reproduce, record the rate and move to intermittent-failure handling: more runs, tighter capture, timing analysis.
  2. Is it the product, the harness, or the environment? Check the test's own logs, the environment's health, recent infra changes. Roughly a third of system-test reds in a young suite are not product bugs. Rule those out before opening a defect.
  3. Localise by layer. Application logs, then OS (processes, sockets, permissions, resources), then configuration, then dependencies, then network. Each layer has its own tools; do not skip a layer because the symptom looks like a later one.
  4. Bisect. Which commit, which config change, which dependency bump. git bisect with the failing test as the oracle turns a day of guessing into an hour.
  5. Capture. Logs, packet captures, API responses, resource graphs, core dumps, config snapshots. Attach them before the environment is reused and the evidence is gone.
  6. Report. The output is a defect a developer can act on without re-running anything: environment, minimal repro, expected versus actual, evidence, frequency, impact.

The Linux toolkit for step 3

journalctl and application logs; ps, top, /proc/<pid>/; ss and lsof for sockets and files; strace for what a process is actually asking the kernel; tcpdump and Wireshark at the network seam; free, vmstat, iostat, df for resources; dmesg for the kernel's view. For crashes, gdb on the core dump (bt, thread apply all bt) and know how cores are produced in containers (ulimit -c, core_pattern). For memory growth in a soak run, an ASan/LSan build or valgrind memcheck on a reduced-scale reproduction, since valgrind is 10 to 50 times slower.

15. Check yourself

Eight questions. Decide before you open the answer.

A test spins up a real Postgres in Docker, calls the service's repository layer, and checks that a row was written with the right schema. What is it? Unit test · Integration test · E2E test

Answer

Integration test. Real dependency, one component boundary, no user flow: integration.

After each deploy to staging, 15 tests run in three minutes covering login, one purchase, and the health endpoint. What are they? Sanity tests · Regression suite · Smoke tests

Answer

Smoke tests. Wide, shallow, fast, gating: smoke. Sanity would be narrow and about a specific change.

Functional testing is: A type that applies at every level · A subset of E2E testing · Another name for system testing

Answer

A type that applies at every level. Types (what property) and levels (how much system) are independent axes.

An input accepts 1 to 100. Which set is the minimum boundary-value coverage? 50 · 1 and 100 · 0, 1, 2, 99, 100, 101

Answer

0, 1, 2, 99, 100, 101. Each boundary and its neighbours on both sides. Off-by-one errors live exactly there.

A typo in the pricing page headline is: High severity, high priority · Low severity, high priority · Low severity, low priority

Answer

Low severity, high priority. Technically trivial, commercially urgent. Severity and priority are independent.

The customer's team runs the pre-agreed acceptance criteria in their own environment before signing off. This is: System testing · User acceptance testing · Conformance testing

Answer

User acceptance testing. Customer, their need, their environment, sign-off: UAT.

Your suite reports 92% line coverage. What does that tell you? Which lines never execute under test; nothing about assertion quality · The code is 92% correct · You can stop writing tests

Answer

Which lines never execute under test; nothing about assertion quality. Coverage is execution, not verification. Mutation testing measures whether tests would notice a bug.

A soak test running normal traffic for two weeks shows memory growing 40 MB a day. Which tool proves whether it's a leak and where? k6 · Selenium · valgrind or ASan on a reduced-scale repro

Answer

valgrind or ASan on a reduced-scale repro. Memory instrumentation on a debug build. k6 generates load; it does not localise leaks.

16. Glossary

  • Acceptance testing — Validation by the customer or product owner against agreed criteria. UAT when done by users.
  • Boundary value analysis — Testing at and around the edges of input partitions.
  • Contract test — Verifies producer and consumer agree on an interface without running both together.
  • Defect removal efficiency — Share of total defects found before release.
  • Entry / exit criteria — Conditions that must hold to start, and to stop, a test phase.
  • Equivalence partition — A class of inputs the system should treat identically.
  • Flaky test — A test that passes and fails on the same code without change. Quarantine, own, fix.
  • Harness — Framework plus environment, stubs, simulators, and data needed to run a suite.
  • Integration test — Two or more real components exercised across their real interface.
  • Mutation testing — Injecting faults into code to check whether tests detect them.
  • Oracle — The mechanism a test uses to decide pass or fail.
  • Regression — Re-running existing tests to confirm a change broke nothing.
  • Sanity test — Narrow, deep check of a specific change before full regression.
  • Smoke test — Fast, shallow check of critical paths to decide whether a build is worth further testing.
  • Soak test — Sustained normal load over days to find leaks and drift.
  • System test — Whole integrated product, black box, against requirements, including non-functional.
  • Traceability — Mapping between requirements and the tests that cover them.
  • Verification / validation — Built it right / built the right thing.

End of course. The modules are independent; return to any of them from the contents.