A scanner and a penetration tester can examine the same application and produce very different evidence. A scanner is valuable for repeatable discovery of known patterns at scale. Manual penetration testing adds human understanding of identity, authorization, workflow, business impact and weakness combinations. Buyers should not choose between automation and humans as if only one has value; they should verify that automation supports a transparent manual method instead of being sold as the complete assessment.

The practical buying test is observable. Ask what a tester will learn about the product, which hypotheses will be exercised, how findings will be validated, and what evidence the report will contain. A provider that performs serious manual work can explain those activities without hiding behind tool names or vague claims of “100% manual testing.”

What automated scanners do well

Automated tools are efficient at broad, repeatable checks. They can crawl reachable content, fingerprint technologies, identify configuration signals, compare responses, fuzz inputs and detect patterns associated with known weaknesses. In a mature security program, scanners run frequently and help teams find regressions before an external assessment.

  • Repeat the same checks across many hosts and releases.
  • Discover common headers, exposed components and input-handling signals.
  • Create leads that a tester can investigate and validate.
  • Support regression monitoring between deeper assessments.

Their limitation is context. A tool may see two successful responses but not understand that one tenant has accessed another tenant’s invoice. It may observe a workflow without knowing that a product manager intended a four-eyes approval. It can miss multi-step abuse that requires state, timing, role changes or an understanding of business value.

What manual penetration testing should add

A model of the attack surface

The tester maps entry points, identities, privilege boundaries, data flows and critical actions. This model includes documented and undocumented APIs, alternate clients, account recovery paths, administrative functions and integrations. It lets the tester reason about what should never be possible, not just what a tool can reach.

Authorization and tenant testing

A person compares users, roles and tenants, changes object references, tests server-side enforcement and follows secondary effects. Good evidence shows the precondition, request, response and unauthorized consequence while avoiding unnecessary access to real data.

Business-logic abuse

Manual testing examines sequence and intent: repeated discounts, approval bypass, inventory manipulation, refund abuse, race conditions, unsafe state transitions or controls enforced only in the interface. These cases are product-specific and rarely reduced to a generic signature.

Chaining and impact analysis

Several individually modest weaknesses may form a serious attack path. A tester explores whether information disclosure enables account takeover, whether a low-privilege action reaches an administrative workflow, or whether client-side controls can be bypassed through the API. The report should explain the chain and the business consequence.

Validation and restraint

Manual work also removes noise. The tester reproduces findings, distinguishes exploitable conditions from informational signals, records limitations and stops before causing avoidable harm. “Manual” is not permission for unsafe experimentation; it operates under written rules of engagement.

The OWASP Web Security Testing Guide provides a broad framework across identity, authentication, authorization, sessions, input validation, business logic, client-side behaviour and APIs. Ask a vendor how the parts relevant to your application become concrete test cases, rather than accepting an OWASP logo as proof of coverage.

Evidence you can inspect before buying

  1. A scoped methodology that connects product risks and trust boundaries to test activities.
  2. A redacted sample report with reproducible evidence, business impact, severity rationale and actionable remediation.
  3. A role-and-tenant coverage plan rather than one generic authenticated account.
  4. Named quality controls: validation, peer review, duplicate consolidation and secure evidence handling.
  5. A clear statement of tools, what they support and which work remains manual.
  6. Transparent exclusions and a final coverage statement describing what was and was not tested.

A sample report is particularly revealing. Look past the visual design. Can an engineer reproduce the issue? Does the finding identify the affected component and explain the violated trust assumption? Is remediation specific enough to address root cause? Does the executive section translate technical evidence into a decision without exaggeration?

Questions to ask the proposed tester

  • Which workflows will you learn before active testing begins?
  • How will you test horizontal, vertical and cross-tenant authorization?
  • Which automated tools will you use, and how will their results be validated?
  • How do you develop product-specific abuse cases?
  • What requires a second reviewer before it enters the report?
  • How do you handle a critical finding during the engagement?
  • How will you record coverage limitations caused by access, instability or time?

The strongest answers are operational. They describe requests, identities, comparisons, state transitions, evidence and review. Generic assurances—“our experts test everything”—do not tell you whether the proposed time and staffing can support the claimed depth.

Red flags that suggest a scanner-led certificate

  • The vendor quotes before asking about roles, tenants, APIs or critical workflows.
  • The methodology is a list of commercial tools without product-specific hypotheses.
  • The report sample is dominated by generic descriptions and raw tool screenshots.
  • Every target receives the same duration regardless of functionality or access model.
  • The proposal guarantees zero false positives without allocating time for validation.
  • The provider promises complete coverage or a clean certificate before testing.
  • Retesting means rerunning the same scan rather than verifying the affected control and bypass paths.

None of these signals proves bad faith by itself. Together, they indicate that the service may be optimized for document production rather than security understanding. Ask for clarification in writing and make the answer part of the statement of work.

Avoid the opposite mistake: “manual” without engineering discipline

Manual testing is not automatically high quality. Unstructured clicking by an inexperienced tester can be less useful than a well-configured scanner. Quality comes from an explicit method, relevant expertise, careful evidence, safe execution and technical review. Buyers should evaluate the complete delivery system rather than treating “manual” as a magic word.

NIST SP 800-115 describes planning, conducting technical tests, analyzing findings and developing mitigation strategies. That lifecycle in the NIST testing guide is a better quality benchmark than counting tools or accepting a single methodology label.

Write the distinction into the SOW

Define the assets, roles, tenants and workflows. Require attack-surface discovery, relevant automated checks, manual authorization and business-logic testing, validation of reported findings, a technical quality review and a coverage statement. Specify that raw scanner observations are not findings until validated, and that limitations must remain visible in the final report.

Also protect time for discovery and analysis. If access is delayed or scope expands, use change control rather than silently reducing manual coverage. The commercial schedule should make the trade-off visible to the person accepting risk.

The practical decision

Buy a process in which automation expands reach and human testing supplies context, validation and business impact. You should be able to trace the vendor’s time from product understanding to test hypotheses, evidence, review and retesting. If the only visible activity is a scan and the only outcome is a certificate, you have not received the assurance implied by a penetration test.

Explore WIMD’s penetration-testing services and ask for the role, workflow and evidence plan that applies to your application—not a generic promise of manual testing.