A buyer needs evidence that a provider can produce a useful penetration-test report, but another customer’s report is not an acceptable sales sample. Even heavy redaction can leave hostnames, identifiers, screenshots, architecture details, timestamps or narrative clues that reveal the client.

The better evaluation uses a vendor-created synthetic report, an authorised demonstration target or a fully reconstructed finding. Assess structure, reasoning and evidence, then use a live walkthrough to test authorship. Confidentiality discipline is itself part of the vendor’s capability.

Set a confidentiality-safe request

Tell candidates exactly what is acceptable: a synthetic application, a public training target, a vendor-owned lab or a purpose-built fictional report. Require the sample to preserve realistic technical depth without retaining customer identifiers or artefacts.

Do not ask the vendor to prove credibility by breaking an NDA. If it offers a real report voluntarily, pause review and ask for its disclosure authority and sanitisation process. The response reveals how the provider will protect your data later.

  • Synthetic organisations, domains, users and IP addresses.
  • No real access tokens, personal data, source code or customer screenshots.
  • No dates or architecture combinations that can re-identify a client.
  • Written confirmation of the sample’s origin and permitted use.

Confirm the report supports two audiences

The executive section should explain what was tested, material risk themes, business implications, constraints and what decision is required. It should not claim that a point-in-time test certifies the entire organisation.

Technical sections should let engineering reproduce, triage and repair each issue. Evaluate whether the same risk is expressed consistently across the summary, finding detail and remediation plan.

  • Plain-language risk without false certainty.
  • Clear scope, dates, method and limitations.
  • Technical evidence sufficient for an authorised engineer.
  • Traceability from headline themes to individual findings.

Inspect scope and coverage evidence

A list of findings does not show what was tested. Look for a coverage section mapping applications, endpoints, roles, tenants, workflows or controls to status and constraints.

Ask how the sample represents inaccessible components, unstable environments, missing accounts and late scope changes. Honest coverage gaps are more useful than a claim that every possible vulnerability was checked.

  • In-scope targets and exclusions are unambiguous.
  • Identities and privilege levels are recorded.
  • Completed, not applicable, constrained and untested work are distinguished.
  • Coverage language matches the engagement’s actual assurance claim.

Evaluate finding anatomy

Select three findings of different severity and trace each from title to proof. A strong finding states the affected asset, identity, precondition, action, observed result, expected control and impact.

Evidence should include both the control case and prohibited case when authorization or workflow is involved. Tool output may support the finding, but the report must explain why the result matters in the application context.

  • Specific title and affected component.
  • Reproducible steps with safe redaction.
  • Expected versus observed behaviour.
  • Impact tied to reachable data, function or business state.

Check risk-rating consistency

Ask for the rating method and score several findings independently. Severity should account for prerequisites, privileges, reach, data sensitivity, scale and recovery—not just a scanner label or weakness class.

Look for transparent professional judgement. Two technically similar defects can have different business impact, but the report should make that difference visible and repeatable.

  • Rating criteria are defined and applied consistently.
  • Compensating controls and environmental facts are explicit.
  • Uncertainty does not become exaggerated certainty.
  • Executive prioritisation agrees with finding-level severity.

Judge evidence quality and data minimisation

Screenshots and payloads should show enough context to prove the issue without exposing unnecessary secrets or personal data. Tokens, cookies and identifiers must be redacted consistently while retaining the distinctions needed for reproduction.

Ask how raw artefacts are stored, transferred and deleted. The sample’s hygiene should match the provider’s proposed evidence-handling process.

  • Redaction is irreversible in delivered formats.
  • Evidence distinguishes users, tenants and objects safely.
  • Secrets and personal data are minimized by default.
  • Raw artefact access and retention are controlled.

Assess remediation usefulness

Generic advice such as validate input or enforce authorization is rarely sufficient. The report should identify the failing decision point, recommend a control pattern, note architectural trade-offs and define how success can be tested.

A provider should avoid prescribing one library or product without understanding the system. Good remediation separates immediate containment from root-cause repair where that distinction matters.

  • Recommendation targets the demonstrated root cause.
  • Ownership or affected layer is clear.
  • Short-term containment is not confused with closure.
  • Verification criteria are observable and specific.

Review aggregation and systemic insight

Multiple symptoms may share a policy or component. Assess whether the report identifies systemic causes without hiding confirmed affected routes.

The best sample helps leaders see patterns while preserving engineering detail. It states which variants were tested and where broader remediation or discovery is still required.

  • Duplicates are consolidated without losing asset coverage.
  • Common roots and affected variants are named.
  • Untested variants are not presented as confirmed.
  • Strategic actions remain linked to evidence.

Inspect retest and closure design

Each finding should contain a closure test: identity, input, expected denial or safe result, and relevant variants. A status of fixed should be based on evidence, not a developer statement.

Ask to see a synthetic retest addendum. It should preserve the original finding, record the tested build and date, show outcomes and distinguish fixed, partially fixed, not fixed and cannot verify.

  • Retest scope and number of cycles are visible.
  • Regression or alternate-path checks are identified.
  • Residual risk and partial remediation are explicit.
  • Closure records remain auditable.

Use a live authorship walkthrough

Choose one sample finding and ask the proposed report author to explain discovery, proof, severity, remediation and retest. Introduce a changed fact and ask whether the rating or advice changes.

This detects template-only samples and sales-led demonstrations. It also shows whether the person assigned to your engagement can communicate with engineers and leadership.

  1. Issue the same synthetic sample request to every candidate.
  2. Score independently before the walkthrough.
  3. Ask the named lead tester to defend one finding.
  4. Record unresolved differences and require written clarification.

Apply disqualifying confidentiality gates

Reject a sample that appears to contain live customer data without demonstrable permission, even if the technical content is impressive. Also reject evasive answers about origin, redaction or retention.

Confidentiality is not a cosmetic procurement clause. Penetration testers will receive privileged access and generate sensitive evidence. Unsafe sales behaviour predicts operational risk.

Use NIST SP 800-115 for planning, conducting and reporting concepts and the official OWASP Web Security Testing Guide for test-area and evidence context. Apply your own contractual confidentiality, privacy and retention requirements to every sample and artefact.

Treat report quality and confidentiality as one test

A sample report should demonstrate how the vendor thinks, proves, prioritises and communicates. It should never require a previous customer to absorb new risk for your procurement process.

WIMD uses synthetic cases, structured scorecards and live authorship walkthroughs to evaluate reporting. This reveals practical quality while preserving the confidentiality standard expected during the real engagement.