Run a penetration-testing vendor technical evaluation call as a structured working session, not a sales presentation. Give each candidate the same anonymised system scenario, ask the proposed delivery lead to reason aloud, test the claims made in written material and record evidence against your shortlist gates.

The call is not an unpaid penetration test and should not demand confidential methods or another client’s data. Its purpose is to determine whether the people and process behind the proposal can understand your risk, ask useful questions, make safe testing decisions and communicate evidence clearly.

Prepare the call before inviting vendors

Send a short brief several days in advance. Describe product type, interfaces, identity model, important workflows, environment choices, data sensitivity and target outcome. Remove customer names, secrets and implementation detail that is unnecessary for evaluation.

Name the participants you expect: the proposed lead tester or technical owner, not only account management. On your side include the person who understands architecture, the security decision owner and a note-taker. Procurement may attend but should not turn the session into price negotiation.

  • Share the same brief and agenda with every candidate.
  • State that responses will be scored and may require written follow-up.
  • Ask the vendor to identify who is speaking and their likely delivery role.
  • Define topics that remain confidential until a later due-diligence stage.

Open with team accountability

Ask who performs discovery, hands-on testing, finding verification, report review and retesting. Clarify whether the people on the call are proposed for delivery and what happens if availability changes. A strong technical speaker who never touches the engagement is a weak assurance signal.

Discuss seniority in terms of decisions, not years alone. Who approves a high-risk production proof? Who challenges a severity rating? Who helps when an identity workflow behaves unexpectedly? Look for clear responsibility and escalation.

  • Expected lead, practitioners and independent reviewer.
  • Relevant experience tied to your architecture and risk.
  • Subcontractor, location and substitution transparency.
  • Availability for engineering discussion during testing and remediation.

Use one architecture scenario to reveal thinking

Present a compact scenario: two tenants, three roles, a web client, public APIs, an asynchronous export and a support function. Ask the candidate to sketch the first testing hypotheses and identify missing information. Do not guide the answer toward your preferred terminology.

Strong candidates clarify ownership, role relationships, workflow states, alternate channels and service trust. They ask for representative identity pairs and safe data. Weak candidates jump directly to scanners or recite vulnerability categories without adapting them.

  1. Ask which trust boundaries and business consequences deserve priority.
  2. Ask which accounts, objects, logs and documentation are prerequisites.
  3. Ask how the plan changes if staging and production differ.
  4. Ask how coverage and untested paths will be recorded.

Test the manual-testing claim

Ask the tester to choose one role boundary or workflow and describe how manual hypotheses are formed. Useful answers vary identity, object relationship, state, sequence, value, timing and channel, while using tools for repeatability and evidence rather than judgment.

Follow up with: “What would convince you this is not vulnerable?” This exposes whether the vendor relies on one negative response or builds broader evidence across representative cases. A mature answer acknowledges sampling and residual uncertainty.

  • Peer-to-peer and vertical authorization examples.
  • Business rules expressed as invariants that can be violated.
  • Chaining of low-level weaknesses into meaningful impact.
  • A distinction between automated breadth and human analysis.

Discuss automation without rewarding theatre

Ask which tools support asset discovery, schema coverage, parameter variation and regression. Then ask how output is triaged, rate controlled and connected to manual investigation. The useful answer is a workflow; a long list of products is not.

  • Automation has a defined purpose in each test phase.
  • Request volume and mutation are bounded by rules of engagement.
  • Tool findings do not enter the report without contextual validation.
  • Manual discoveries can be expanded systematically across related paths.

Run a sample-finding discussion

Provide a small synthetic finding or ask the vendor to use a sanitised one. Have the proposed lead explain preconditions, evidence, impact, severity, root cause, remediation and retest criteria. Ask what would be removed from the report for confidentiality and what must remain for reproducibility.

Challenge one assumption politely. Observe whether the vendor responds with evidence, revises the conclusion or becomes defensive. The call should reveal a process for resolving technical disagreement without commercial pressure.

  • Identity, asset, version and state are explicit.
  • The observed effect supports the claimed impact.
  • Severity separates technical facts from customer-supplied context.
  • Remediation fixes the failed control rather than hiding one payload.

Examine safety and critical escalation

Ask the candidate to design boundaries for production-like testing: source addresses, marked accounts, value or rate limits, prohibited actions, monitoring, stop conditions and an emergency contact. Good testers explain how they obtain enough proof without maximising harm.

Present a critical-finding scenario. Ask who verifies it, who contacts the customer, what evidence is sent, how communication is protected and whether testing continues. Look for an agreed decision path rather than improvised heroics.

  • Testing authority and third-party boundaries are verified.
  • Stop conditions have a named decision owner.
  • Critical notification is prompt, secure and evidence-backed.
  • Unexpected operational effects enter an incident process.

Ask how coverage and limitations are communicated

Ask what the provider records when no vulnerability is found. The answer should include assets, identities, workflows, interfaces, hypotheses, sampling and blocked paths. Without that record, a clean report can be misread as complete assurance.

Use a difficult example: one role account never works or an integration is unavailable. A trustworthy vendor explains the impact on assurance, attempts resolution, documents the gap and does not quietly reuse effort elsewhere without approval.

  • Coverage is understandable beyond raw request counts.
  • Blocked prerequisites are escalated during testing.
  • Limitations appear in status updates and the final report.
  • Scope changes require explicit impact and approval.

Close with reporting, remediation and retesting

Ask how draft factual review works, who owns corrections, how developer questions are handled and when the report becomes final. Confirm finding identifiers remain stable and evidence changes are traceable.

Then ask the team to describe a retest. It should reconstruct original conditions, execute the proof, examine relevant variants and record the observed result. Risk acceptance and compensating controls should not be presented as though the vulnerability was technically removed.

  • Executive and engineering audiences receive appropriate detail.
  • Clarification and debrief have defined owners and windows.
  • Retest scope, cycles, prerequisites and turnaround are explicit.
  • Closure language remains bounded to the tested version and paths.

Score what happened in the room

Immediately after the call, score the same criteria for every vendor. Record quotations or specific evidence, not impressions such as “seemed experienced.” Give credit for precise questions and honest limitations; penalise unsupported certainty and evasion.

Do not let one charismatic speaker determine the outcome. Verify written claims, profiles, sample evidence and contractual controls separately. The call is one input to a defensible shortlist, not a substitute for due diligence.

  1. Each evaluator scores independently before group discussion.
  2. The panel reconciles major differences using evidence from the call.
  3. Material unanswered questions receive one written clarification deadline.
  4. Only candidates meeting every mandatory gate progress to final proposals.

The OWASP Application Security Verification Standard and NIST SP 800-115 give shared language for controls and testing process. Use the call to determine whether the vendor can apply those references to your system.

Turn the call into a defensible shortlist decision

Advance the vendor only when the call adds credible evidence about the assigned team, application-specific reasoning, testing depth, reporting and safety. Preserve scores, questions and follow-ups so later commercial comparison does not rewrite the technical conclusion.

WIMD uses technical evaluation as a two-way readiness check. Buyers can challenge the proposed method and evidence standard, while WIMD identifies the identities, architecture details and operating boundaries required to deliver meaningful assurance.