A renewal decision should use evidence from the completed engagement, not only price, finding count or personal rapport. A technically responsible vendor can find few issues in a well-secured product; a noisy vendor can produce many low-value observations.
Evaluate whether the provider controlled scope, exercised relevant attack paths, produced reproducible evidence, communicated risk promptly, protected sensitive material and helped accountable teams reach verified closure. Compare performance with the objectives and constraints agreed at selection time.
The review should involve security, engineering, product or service owners, operations, procurement and risk stakeholders who actually experienced the work. Separate provider performance from buyer-caused readiness failures, and record improvements required from both sides.
Return to the original engagement objectives
Review the decision the test was meant to support, the agreed scope, deliverables, schedule and success criteria. A vendor cannot be fairly judged against expectations never documented.
Distinguish product security outcomes from service performance. The provider owns testing quality; the buyer owns readiness, timely decisions and remediation.
- Business and assurance objective.
- Authorised scope and method.
- Required deliverables.
- Known constraints and buyer dependencies.
Assess mobilisation and readiness discipline
Evaluate kickoff quality, dependency discovery, account and environment checks, communication setup and escalation preparation. Strong mobilisation finds blockers before paid testing time is lost.
Record whether the vendor gave clear readiness criteria and whether delays were escalated with practical options. Also record buyer failures that affected delivery.
- Clear onboarding and ownership.
- Readiness checklist quality.
- Early blocker visibility.
- Accurate schedule recovery.
Assess scope and authority control
Check whether testers stayed within authorised targets, techniques, identities and production limits and whether discoveries entered controlled change management.
Unauthorised testing is a serious performance concern. Equally, silently ignoring in-scope areas or changing depth without disclosure weakens assurance.
- Rules-of-engagement compliance.
- Scope-change records.
- Third-party boundary respect.
- Final scope reconciliation.
Assess coverage relevance
Compare completed coverage with the product’s real attack surface: roles, tenants, APIs, clients, workflows, administration and trust boundaries. Look for explicit limitations.
Do not use request counts as a proxy. Ask whether the vendor tested the decisions and failure modes that matter to the service and whether important gaps were visible early.
- Identity and authorization depth.
- Critical business workflows.
- API and client coverage.
- Cloud and integration boundaries.
Assess technical depth and judgement
Review the balance of automated discovery, manual analysis, business-logic testing, chained impact and safe validation. Findings should distinguish exploitable weaknesses from hardening advice.
Consider whether assigned specialists matched web, API, mobile, cloud or technology needs and whether senior review improved complex conclusions.
- Manual and workflow testing.
- Relevant specialist skill.
- Controlled impact validation.
- Creative but safe investigation.
Assess evidence and false-positive control
Sample findings for reproducibility, target and identity context, control cases, redaction and proof of authoritative impact. Check how quickly disputed evidence was re-examined.
A low false-positive rate matters, but so does willingness to correct mistakes transparently. Repeated unverifiable or exaggerated findings create engineering cost and erode trust.
- Reproducible steps and context.
- Sufficient, minimal evidence.
- Peer review and severity calibration.
- Transparent correction process.
Assess urgent escalation and safety
Review critical notifications, acknowledgement tracking, stop decisions and coordination with operations or incident response. Confirm that urgent risk did not wait for the final report.
Examine any outages, excessive load, data exposure or rule violations and the provider’s candour in reporting them. Safe execution is part of technical competence.
- Timely critical notification.
- Minimum-safe confirmation.
- Production and data safeguards.
- Incident transparency and recovery support.
Assess routine communication
Status updates should have shown completed and remaining coverage, blockers, changes, actions and milestone forecasts. Meetings should have resolved decisions rather than repeated vague activity.
Consider responsiveness across time zones and the quality of communication with both engineers and risk owners. Constant messaging is not the same as useful transparency.
- Predictable reporting cadence.
- Actionable blockers and owners.
- Honest forecast changes.
- Appropriate audience and confidentiality.
Assess report quality and usability
The final report should align executive and technical content, preserve limitations, explain severity and give actionable remediation direction. Check version control and delivery security.
Ask engineering whether findings entered tickets without extensive rewriting and whether leaders understood the risk without technical distortion.
- Accurate scope and coverage.
- Clear business and technical impact.
- Root-cause-focused remediation.
- Controlled, consistent final artefacts.
Assess remediation and retest support
Evaluate clarification quality, remediation discussions, retest readiness, turnaround and finding-by-finding closure evidence. The provider should remain independent while helping teams understand the control objective.
Check whether systemic variants were considered and whether partial fixes remained accurately open. A clean retest letter should never be obtained by narrowing evidence after the fact.
- Responsive technical clarification.
- Clear verification criteria.
- Release-specific retesting.
- Honest residual status.
Assess confidentiality and vendor security
Review evidence handling, access control, encryption, retention, deletion and any subcontractor use. Confirm that leavers and role changes were managed.
A provider stores concentrated attack knowledge. Security exceptions, unexplained copies or unauthorised subcontractors may outweigh good technical work.
- Evidence access and transfer.
- Retention and deletion proof.
- Personnel and subcontractor control.
- Incident and disclosure history.
Assess team continuity and independence
Compare the proposed and delivered team, senior oversight, turnover and availability. Determine whether knowledge continuity improved efficiency without creating complacency.
Reassess conflicts and independence, especially if the provider also designed or implemented controls. Rotation can add a fresh viewpoint; continuity can add product depth. Choose deliberately.
- Named-team delivery.
- Senior quality ownership.
- Knowledge transfer and continuity.
- Conflict and independence safeguards.
Compare commercial value and decide
Reconcile scope, change orders, invoices, delays and delivered evidence. Value includes reduced uncertainty and usable closure, not just day rate or finding volume.
Choose renew, renew with corrective commitments, competitive benchmark, rotate specialist coverage or replace. Set measurable improvements and owners for the next cycle regardless of provider.
- Score evidence against original objectives.
- Separate vendor and buyer performance causes.
- Record strengths, failures and required improvements.
- Approve renewal, competition or transition with safeguards.
Use NIST SP 800-115 for assessment-process context and the OWASP Web Security Testing Guide for application testing context. Apply the organisation’s procurement, supplier-risk, independence and evidence-protection requirements.
Make renewal an evidence-based assurance decision
The best vendor review examines how the engagement reduced uncertainty and enabled verified action. It recognises both strong technical work and operational discipline, while separating customer-side blockers from provider failures.
WIMD recommends a documented scorecard and improvement plan so continuity, rotation or replacement strengthens the next assessment instead of simply changing logos.