For QA, vulnerability scanning and penetration testing answer different questions. A scanner repeatedly matches observable conditions against known checks. A penetration tester forms hypotheses, changes identity and state, follows trust boundaries, chains weaknesses and judges business impact. Strong assurance uses automation for breadth and repeatability, and human testing for adversarial depth—not one as a substitute for the other.

Define the architecture question before testing

Define the assurance claim before choosing a method. If the question is whether a dependency, header or exposed service matches a known rule, scanning may be efficient. If it concerns authorization, tenant isolation, workflow sequence or exploit chaining, human reasoning is essential.

Write the expected security invariant and the prohibited outcome in language that engineering, security and operations interpret the same way. Name the identity, trust transition, protected asset and authoritative state. That statement becomes the basis for scope, evidence and retesting.

Build the minimum evidence pack

  • A current component and trust-boundary diagram for the environment under test.
  • Identity, credential, network and data-flow details relevant to the decision.
  • Representative accounts, workloads and synthetic records with known ownership.
  • Logs or traces that follow the action through every material control point.
  • Explicit safety limits, stop conditions, owners and restoration steps.

Collect the application and API inventory, roles and tenants, critical journeys, data classifications, authentication paths, environment constraints, current scanner coverage, recent code changes, known exclusions, test accounts, logging access and explicit safety boundaries.

Understand what scanners do well

Automated scanners are valuable when a repeatable probe has a reliable observable result.

  • Inventory exposed hosts and routes.
  • Check known dependency or platform signatures.
  • Detect common header and transport weaknesses.
  • Run frequent baseline checks.
  • Compare results across builds.

Record scanner version, configuration, authentication, target coverage, exclusions and validation status. A tool logo alone is not evidence of complete coverage.

Recognize where scanners lose context

Automation rarely understands ownership, permitted sequence, commercial rules or the meaning of two identities in different tenants.

  • Compare owner and non-owner access.
  • Change role while preserving object reference.
  • Abuse order or payment sequence.
  • Chain a low-severity leak with another control gap.
  • Explore undocumented routes and alternate clients.

Describe each missed class as an assurance limitation, not a scanner defect. No single automated suite models every product-specific invariant.

Use penetration testing for adversarial questions

A tester adapts after each observation and chooses the next safe experiment based on product behavior.

  • Map attack surface and trust boundaries.
  • Form a prohibited-outcome hypothesis.
  • Establish a legitimate baseline.
  • Change one trust variable.
  • Stop at minimum defensible proof.

Require a traceable path from hypothesis to request, authoritative outcome, impact and remediation—not screenshots without context.

Separate coverage from finding count

More scanner alerts do not mean greater security assurance, and a short manual report does not mean little work.

  • Measure reachable surface.
  • Track authenticated roles and tenants.
  • List workflow states exercised.
  • Record blocked or excluded tests.
  • Validate duplicate and false-positive handling.

Use a coverage matrix and residual-risk statement. Finding counts are outputs, not a measure of method quality.

Design a layered QA programme

Assign deterministic checks to automation and reserve expert time for changing, contextual attack paths.

  • Run SAST, SCA and DAST at suitable pipeline stages.
  • Keep authorization regression cases.
  • Threat-model material changes.
  • Schedule independent manual tests.
  • Retest validated findings after remediation.

Map each method to the risk question it answers, its cadence, owner and escalation path.

Turn manual discoveries into regression

Once a tester reveals a stable failure mode, QA can often automate the durable invariant.

  • Preserve sanitized fixture and preconditions.
  • Assert denied action and unchanged state.
  • Test a legitimate control case.
  • Parameterize roles and tenants.
  • Keep complex chaining cases for expert review.

Do not encode dangerous proof-of-concept behavior into broad production tests. Retain safe, minimal assertions.

Evaluate vendors and tools consistently

Compare claimed coverage against your product architecture rather than generic feature lists.

  • Ask for authenticated test method.
  • Inspect manual evidence examples.
  • Confirm business-logic approach.
  • Review safety and communication controls.
  • Clarify retest and closure deliverables.

A defensible programme states what each layer can and cannot prove, rather than promising zero vulnerabilities.

Execute as controlled hypotheses

  1. Establish the legitimate baseline and capture authoritative state.
  2. Change one trust variable—identity, route, scope, object, time or environment.
  3. Observe the decision at each layer rather than relying only on the response.
  4. Stop at the minimum proof that demonstrates or rejects the hypothesis.
  5. Restore test state and record residual uncertainty or blocked coverage.

Run scanners in authenticated and unauthenticated modes where safe, triage signals, and give the manual tester clean accounts and architecture context. Coordinate testing windows and preserve evidence without exposing secrets.

Report the architectural consequence

Report automated observations and manually validated findings separately. Include scope, method, coverage, exclusions, false-positive handling, exploit evidence, business impact and residual uncertainty.

Separate observed evidence from inference. State prerequisites, repeatability, blast radius and the shared component responsible for the decision. When multiple findings have one architectural cause, keep the individual proofs but group the remediation around the common control.

Required closure evidence

  • The original proof no longer succeeds.
  • A legitimate workflow still succeeds under the intended identity and route.
  • Alternate consumers of the same pattern enforce the same invariant.
  • Telemetry records both allowed and denied decisions with useful context.
  • The architecture record and threat assumptions reflect the new control.

Remediate at the durable control point

Fix the shared root control, then add deterministic regression where possible. Improve scanner configuration for machine-detectable patterns and preserve expert retesting for contextual failures.

Retest the invariant, not just the payload

Rerun the relevant automated check, repeat the original manual proof, exercise nearby identities or states and verify a legitimate workflow. Closure requires authoritative evidence, not disappearance of one alert.

Use the NIST Technical Guide to Information Security Testing and Assessment for assessment planning and evidence discipline, and the OWASP Web Security Testing Guide for relevant application techniques. Adapt both to the system-specific trust decision described here.

The architecture decision

QA should use vulnerability scanning as continuous, repeatable detection and penetration testing as bounded adversarial validation. Their overlap is useful; their assurance claims are not interchangeable.

Ask WIMD to design the right testing mix for automated coverage, manual attack paths and durable QA regression.