AI Verify as a source of test evidence
What the toolkit can provide
AI Verify is a testing framework and software toolkit. Its official repository describes technical tests for supported classification and regression models using tabular and image data, with modular plugins. Coverage depends on the version and plugin. The project does not provide a general guarantee of safety or the absence of risks and bias. See the official README.
A test result can support an assessment of a particular system property. It is not an AI Act compliance certificate and does not automatically satisfy an article of the law. The name “AI Verify 2.0” alone does not establish coverage of all generative AI risks, intellectual property issues or agent security.
Define the question and the evidence first
| Question | Possible test evidence | What else is needed |
|---|---|---|
| Is performance acceptable for the intended purpose? | Results on representative data and failure scenarios | A justified threshold, uncertainty, consequences of errors and operational monitoring |
| Do impacts differ between groups? | Error metrics and relevant fairness analyses | Lawful handling of attributes, context and expert interpretation |
| Is the model robust against selected input changes? | Results of specific robustness tests | Scenario coverage and security of the wider system and dependencies |
| Can the test be reproduced? | Model, plugin, data and configuration versions | Retained evidence, a repeatable procedure and change control |
| Can a person exercise effective oversight? | Results from a user scenario and records of interventions | Authority, training, relevant context and a functioning operational process |
Selected results may support, for example, documentation of accuracy and robustness under AI Act Article 15 for high-risk systems. Assess the scope and application date of those requirements separately.
Fairness has no universal 0.1 threshold
Statistical parity, equal opportunity, equalised odds and predictive parity measure different properties and may not be achievable simultaneously. Select metrics for the specific decision, purpose and risk. A difference below 0.1 or 5% alone does not demonstrate an absence of discrimination.
Before testing, define the affected groups, sample size and representativeness, uncertainty, rationale for the metric and response to an unacceptable impact. Do not process sensitive attributes merely because a testing library accepts them; check the legal conditions.
A reproducible process
- Describe the system, intended use and property being tested.
- Select a supported toolkit version and a specific plugin using its documentation.
- Prepare test data you are entitled to use and establish a baseline.
- Record versions, settings, limitations and unsuccessful tests.
- Have a responsible specialist interpret the result and state where the conclusion cannot be generalised.
- Link findings to corrective actions or a justified decision, including the conditions for repeating the test.
Work outside the testing tool
Address the inventory and legal classification, responsibilities, required documentation, contracts, transparency, incident procedures and any applicable conformity assessment. Providers of high-risk systems and deployers have different obligations. AI Act Article 18 concerns retention of selected provider documentation; Article 19 concerns automatically generated logs. Article 52 sets out a procedure for GPAI models with systemic risk, rather than general chatbot transparency.
Verified on 6 September 2026: AI Verify repository and documentation, consolidated AI Act. Continue with MEASURE and preparing for expert assessment.