Adding a penetration-testing provider to an approved vendor list requires evidence that the firm can deliver a repeatable assurance standard, not one impressive sales demonstration. Evaluate how it scopes unfamiliar systems, assigns qualified testers, validates findings, protects sensitive data, controls report quality and supports remediation across multiple engagements.
The decision should produce an approval record that procurement, security and application owners can reuse. Separate mandatory controls from engagement-specific fit so that a technically capable specialist is not rejected for lacking irrelevant scale, and a polished generalist is not approved without proof of testing depth.
Define your assurance standard before meeting vendors
Write down the outcomes your organization expects. These may include manual authorization testing, business-logic abuse, API coverage, tenant-isolation validation, production-safe rules of engagement, developer-ready evidence and independent retesting. Requirements should reflect the applications and risks the vendor will actually assess.
- Types of applications, APIs, mobile clients, cloud services and infrastructure in scope.
- Required attacker perspectives and credentialed access models.
- Regulatory, customer and internal reporting obligations.
- Data residency, retention and personnel-location constraints.
- Expected turnaround, critical-finding escalation and retest service levels.
A vendor can only be evaluated against a defined need. “OWASP coverage” alone does not state which identities, business workflows, trust boundaries or environments require assurance.
Examine the methodology as an operating process
Ask the provider to walk through a representative engagement from intake to closure. Look for decisions and quality gates rather than a list of tools. The process should adapt to architecture and risk while remaining documented enough to repeat.
- How is scope converted into assets, roles, tenants, interfaces and critical workflows?
- How are automated results triaged and where is manual testing required?
- How do testers record coverage, limitations and blocked paths?
- How are critical findings independently checked and escalated?
- Who reviews the report for technical accuracy and evidence quality?
- How are remediation questions and retesting handled after delivery?
Request a redacted methodology artefact or coverage template. A verbal claim of “manual testing” is weaker than an observable workflow showing how manual hypotheses, evidence and peer review are recorded.
Evaluate the people who will perform the work
Company credentials do not reveal who will test your application. Ask for the delivery team structure, relevant experience, supervision model and substitution rules. Certifications may support a profile, but they do not replace evidence of application reasoning and clear technical communication.
- Named or role-defined lead tester and report reviewer.
- Experience with your architecture, protocols and business model.
- Ratio of senior review to practitioner work.
- Process for staffing changes and subcontractors.
- Language, time-zone and availability for live engineering collaboration.
For recurring approval, confirm that quality does not depend on a single exceptional tester. Ask how junior work is supervised and how the provider calibrates severity and evidence across teams.
Test the claim of manual depth
Use a scenario discussion
Present a small anonymized scenario: several roles, two tenants, an API, an approval workflow and a support function. Ask how the vendor would plan testing. A strong answer identifies identity pairs, state transitions, trust boundaries, abuse cases and missing information before naming tools.
Inspect sample evidence
Review a redacted report or finding. It should contain reproducible steps, raw request and response evidence where relevant, affected asset and role, preconditions, observed impact, business context, remediation direction and limitations. Confirm that sensitive values are handled safely.
Ask for a coverage statement
A mature provider explains what was tested, sampled, blocked and excluded. A clean report without a coverage record cannot tell you whether the system resisted testing or important paths were never exercised.
Assess finding validation and severity governance
Ask how the provider distinguishes a potential weakness from a reportable finding. High-impact conclusions should require reproduction, evidence and peer review. Severity should combine technical characteristics with stated environmental context without allowing commercial pressure to inflate or suppress ratings.
- Minimum evidence standard for each severity level.
- Independent review for critical and high findings.
- Handling of uncertain impact, inaccessible paths and false positives.
- Process for client disagreement and evidence-based reconsideration.
- Rules for linking chained weaknesses and shared root causes.
Review data protection as part of technical quality
Penetration testing can expose credentials, tokens, source code, architecture, customer records and exploitable defects. Review how the vendor minimizes, encrypts, separates, accesses, transfers, retains and deletes those materials.
- Approved systems for evidence and report exchange.
- Personnel access controls, logging and least privilege.
- Device, remote-work and subcontractor controls.
- Retention periods and verifiable deletion process.
- Incident notification and cooperation obligations.
Map contractual claims to operational evidence. A policy that allows uncontrolled screenshots or indefinite local storage does not meet a strong assurance standard.
Evaluate delivery reliability
Approved vendors must perform consistently under scheduling pressure. Ask for a sample plan, dependencies, escalation route and report QA timeline. Check references for responsiveness, evidence quality and remediation support—not merely whether a report arrived.
Review capacity honestly. A small specialist may be excellent for focused work but unsuitable for several simultaneous global engagements. A large provider may have capacity but variable practitioner depth. Approve fit by service class where useful.
Make post-report support part of approval
- Technical debrief for security and engineering.
- Reasonable clarification window with named ownership.
- Context-aware remediation consultation.
- Defined retest scope, timing, evidence and final status.
- Treatment of partially fixed, mitigated and accepted findings.
- Closure artefacts suitable for customers and auditors without overstating assurance.
Run a controlled pilot when the risk justifies it
For a strategic vendor, use a bounded pilot on a representative application or workflow. Score the provider on scoping quality, access discipline, coverage transparency, finding accuracy, communication, report usefulness and closure support. Do not create artificial vulnerabilities that reward guessing; evaluate normal delivery behaviour.
Use an approval scorecard with gates
Use pass/fail gates for legal authority, confidentiality, data handling and conflicts. Score technical depth, delivery capability and service fit separately. Preserve written evidence and exceptions. A weighted total should not allow excellent presentation to compensate for a failed mandatory security control.
Use the OWASP Web Security Testing Guide and NIST SP 800-115 as reference points when assessing whether a provider’s process covers planning, execution, evidence analysis and reporting.
Set conditions for continued approval
- Approval scope: service types, regions, data classes and engagement limits.
- Expiry or review date and evidence required for renewal.
- Performance measures such as report defects, retest quality and missed deadlines.
- Trigger review after personnel, ownership, data-processing or material quality changes.
- Route for suspending new work when a critical control fails.
The practical decision
Approve a penetration-testing provider when its methodology, assigned people, manual reasoning, evidence standard, peer QA, data protection and remediation support meet your defined assurance needs repeatedly. Record the approval boundaries and validate performance through real engagements instead of treating onboarding as a permanent endorsement.
Evaluate WIMD against your security-vendor standard with a transparent discussion of methodology, evidence, delivery controls and retesting.