A penetration-testing vendor shortlist should contain only providers that have passed minimum technical, legal, operational and evidence-quality gates for your actual scope. Shortlisting is not the stage for comparing polished proposal totals. It is the stage for removing providers that cannot credibly perform the work, so final proposals come from a small group you could genuinely appoint.

Use one written checklist for every candidate. Separate mandatory gates from scored differentiators, request the same proof, record uncertainty and make exceptions visible. A famous brand, a low price or a large certification list should not compensate for a failed requirement involving authorization, data handling, tester competence or safe delivery.

Write a one-page brief describing the systems, decision deadline and assurance outcome. Include application and API types, roles and tenants, critical workflows, environments, data sensitivity, production constraints and required deliverables. This prevents the shortlist from becoming a collection of companies that offer vaguely related services.

  • Business decision: launch, customer assurance, regulatory programme, acquisition or recurring baseline.
  • Technical surfaces: web, API, mobile, cloud-linked workflows, administrative planes and integrations.
  • Identity model: peer users, privileged roles, tenants, partners, support users and service accounts.
  • Delivery constraints: dates, environments, approved hours, prohibited actions and internal dependencies.

Gate 1: prove direct scope relevance

Ask each candidate for two or three anonymised examples that match the architecture or risk—not merely the industry label. A provider may have tested many websites yet lack experience with multi-tenant authorization, GraphQL, mobile binaries, payment state or service-to-service identity.

Require the vendor to explain how it would break the scope into testable units. Strong responses discuss trust boundaries, identities, objects, workflows and interfaces. Weak responses repeat the target count and an OWASP label without identifying where depth is needed.

  • Experience with the protocols, platforms and deployment model in your brief.
  • Evidence of testing comparable authorization and business-logic complexity.
  • A credible approach to undocumented, asynchronous or internal paths where relevant.
  • Clear limitations when the requested access cannot support the required assurance.

Gate 2: identify the people who will deliver

Company capability does not guarantee assigned-team capability. Ask for the proposed lead, expected practitioner mix, reviewer, location, availability and substitution rules. Profiles may be anonymised during early discovery, but roles, experience and accountability should be concrete.

Certifications are useful supporting signals. They do not show whether a practitioner can reason through your tenant model, communicate with engineers or produce defensible evidence. Use a short technical conversation before shortlisting.

  • Named or role-defined lead tester with relevant engagement experience.
  • Independent technical reviewer for important findings and the final report.
  • Disclosure of subcontracting and the controls applied to external personnel.
  • A replacement process that preserves competence, context and accountability.

Gate 3: verify the testing method

Ask the vendor to walk from intake to closure. Listen for scope reconciliation, account validation, threat hypotheses, controlled automation, manual exploration, evidence capture, peer review, reporting and retesting. A tool list is not a methodology.

Give every candidate one representative scenario. For a SaaS platform, describe two tenants, several roles and an approval workflow. Ask what identities, objects, transitions and evidence the team needs. The response reveals manual depth more reliably than the phrase “manual VAPT.”

  • Automated discovery is triaged and used to support—not replace—manual analysis.
  • Authorization uses multiple users, roles and tenants where relationships matter.
  • Business logic is tested through state, sequence, quantity, timing and channel variation.
  • Coverage, blocked paths and exclusions are documented instead of silently omitted.

Gate 4: inspect evidence and reporting quality

Request a sanitised finding and a sample report structure. A useful finding identifies preconditions, affected identity and asset, reproducible steps, observed result, impact, severity rationale, remediation direction and retest criteria. It should prove the unsafe condition without exposing another client.

Check whether the deliverable serves different readers. Leaders need affected capabilities, systemic themes and decisions; engineers need technical evidence and root-cause direction; assurance teams need scope, dates, limitations and status that cannot be mistaken for a guarantee.

  • Evidence is specific enough to reproduce and sufficiently redacted.
  • Severity connects technical facts with stated business context.
  • Repeated symptoms are related to root cause without hiding distinct exposure.
  • A coverage and limitation statement accompanies the finding list.

Gate 5: review safety and critical escalation

The candidate should describe rules of engagement before access is granted. These should cover authorisation, target ownership, source addresses, test identities, data markers, rates, prohibited actions, third parties, stop conditions and emergency contacts.

Ask how a critical finding is handled before report delivery. A mature provider verifies the evidence, uses an agreed secure channel, reaches the designated decision-maker and continues or pauses according to the operating plan. “We put it in the report” is not enough.

  • Written authorisation and exact target boundaries.
  • Production-safe proof thresholds and operational coordination.
  • A reachable escalation path with severity and evidence expectations.
  • Incident handling if testing causes an unexpected effect.

Gate 6: protect credentials, evidence and confidential data

Penetration testing may expose credentials, tokens, source, customer data, architecture and exploitable defects. Shortlist only vendors that can explain approved transfer, encryption, least-privilege access, device controls, personnel access, retention, deletion and incident notification.

Map policy statements to working practice. Ask where testers keep captures, whether personal storage is prohibited, how report access is logged and how deletion is confirmed. Confidentiality clauses do not compensate for weak operational handling.

  • Data locations and authorised personnel are known before delivery.
  • Evidence is minimised and secrets are redacted wherever possible.
  • Retention is time-bound and aligned with remediation and retesting needs.
  • Subcontractors follow equivalent confidentiality and security controls.

Gate 7: confirm remediation and retesting support

Ask what happens after the PDF arrives. Define the clarification period, technical debrief, remediation questions, retest eligibility, turnaround and closure artefact. A provider should distinguish a technical retest from an administrative status update.

The retest approach should recreate the original preconditions and check meaningful variants when a shared control changed. It should record fixed, partially fixed, mitigated, unresolved and accepted outcomes without overstating assurance.

  • Named route for engineering clarification and evidence questions.
  • Retest scope, number of cycles, timing and prerequisites stated early.
  • Closure evidence remains linked to the original finding and affected versions.
  • Customer-facing confirmation accurately states scope and limitations.

Score differentiators only after the gates pass

Once mandatory gates are satisfied, score technical fit, team depth, evidence quality, collaboration, delivery confidence, retest value and price. Use weights connected to your risks. Preserve the reason for each score and require evaluators to cite evidence.

Do not let a weighted total hide a failed gate. A low price cannot offset uncontrolled evidence handling; a strong report sample cannot offset absence of relevant API capability. Show gate results and scores separately.

  1. Mark each mandatory item pass, fail or unresolved and identify the evidence owner.
  2. Resolve material unknowns through one equal clarification round.
  3. Score only the candidates that pass every non-negotiable gate.
  4. Select a small shortlist that procurement and technical owners could all appoint.

Use the NIST SP 800-115 planning and assessment guidance, plus the OWASP Web Security Testing Guide, as reference points when testing whether a vendor has a complete operating process.

Move qualified vendors into final proposals

A shortlist should reduce uncertainty. For every remaining provider, you should know why its team, method, evidence controls and operating model fit the stated engagement. The final request can then focus on a comparable scope, effort, schedule, contract terms and total value.

WIMD supports this evaluation with transparent scoping, identifiable delivery roles, manual authorization and workflow testing, reproducible findings, explicit limitations and evidence-led retesting. Buyers can examine that delivery system before deciding whether WIMD belongs on the shortlist.