AI security assurance: read the scope of the report

Evaluate an AI security supplier's assurance evidence against the service you intend to buy: the assessed components, test dates or reporting period, excluded systems, open findings and customer responsibilities. A report supports conclusions within those boundaries. Record the remaining questions before accepting the service.

An AI security platform may include a browser component, desktop software, a cloud console, model services and support tooling. Evidence about one part needs an explicit connection to the other parts your organisation will use.

Start with the service boundary

The UK NCSC's cloud-provider guidance recommends examining the evidence behind assurance claims and checking that its scope covers how and where the customer uses the service. It also asks buyers to consider the supplier's configuration of underlying cloud services.

Create a short scope record alongside the proposed deployment. Name the product, hosting arrangement, relevant client components, administrative functions and third-party services. Match each to the scope statement supplied with the assurance document.

For example, a fictional platform's web-console test may cover authenticated tenant access while excluding its desktop update mechanism. Record that exclusion and ask what separate assurance covers the update path. This is a procurement example, without any claim about a particular vendor.

Distinguish the assurance documents

A penetration test examines a defined target under agreed conditions. NCSC's penetration-testing guidance treats it as a point-in-time activity and recommends recording excluded components as limitations. Ask which roles, interfaces and deployment versions were included.

A SOC 2 report examines a described system and its controls. AICPA's description criteria distinguish a type 1 report covering the description and control design at a specified date from a type 2 report that also addresses operating effectiveness over a period. The detail can include functions outside the system boundary, subservice arrangements and controls customers must operate themselves. The criteria download requires a free account.

An ISO management-system certificate has a stated scope and issuing body. Check that scope and the certificate's status through the issuer or relevant accreditation register, such as the JAS-ANZ register. A failed register search needs follow-up with the issuer before drawing a conclusion.

These documents answer different questions. Keep their dates, boundaries and responsible parties visible in the same buying decision.

Ask how findings reached closure

A useful attestation should let a buyer understand the remediation process without receiving detailed exploitation instructions. Ask for the date and scope of testing, the status of material findings, who accepted residual risk and what verification followed the fixes.

NIST SP 800-115 discusses remediation plans and subsequent verification. It distinguishes technical retesting from relying on signed documentation and records the risk of accepting a fix without technically checking it. This is longstanding US guidance, with terminology that needs adapting to the purchasing context.

Separate a developer's fix date, the deployment date and the retest date. If a finding was closed through a compensating control, identify the control, its owner and the conditions under which that decision remains valid.

An unresolved finding may require a narrower deployment, additional safeguards or deferral. Make that decision through the organisation's risk process. A severity label alone cannot explain the business exposure.

Protect the evidence while reviewing it

Detailed security reports can expose sensitive system information. NIST warns that inappropriate release of assessment material can increase exploitation risk. Request the level of detail needed for the decision through an agreed confidential channel, and restrict its internal circulation.

NCSC recommends looking for evidence that independent validation forms part of a governed process. A redacted attestation, a scope statement and a remediation discussion may answer some procurement questions. A sensitive unresolved issue may need a more controlled review with the supplier and assessor.

Confirm who issued each document and how authenticity can be checked. Preserve the document version and review date. Keep supplier reports out of public tickets, broad chat channels and unapproved AI summarisation tools.

For the monitoring service's own data handling, use the separate privacy checklist. Assurance-document access creates another information-handling responsibility.

Close the gap between the report and your use case

Suppose the supplier's assessment predates a newly added agent connector. Ask what changed, what security review covered that change and when refreshed assurance will include it. The existing report can remain useful for unchanged components while the connector receives its own evidence request.

Record customer-operated controls explicitly: identity configuration, administrative access, supported deployment settings and any monitoring obligations named in the report. Assign each to an owner who can demonstrate it in the proposed environment.

Finish with a concise acceptance record: evidence received, relevant scope, material exclusions, findings status, customer responsibilities and the next review trigger. Combine that record with a controlled tenant-isolation evaluation where appropriate.

This guide offers procurement criteria. It does not evaluate any vendor's private report or assert a Promptective certification. Ask for evidence that fits the service and decisions your team will actually depend on.

Discuss your AI security requirements with Promptective.