MEASURE: measurement and evaluation

How to analyse, test and validate AI systems

NIST AI RMF function: MEASURE Guide sections: MS-1 to MS-5

The labels below organise this guide; they are not a reproduction of official NIST subcategory identifiers. See the NIST mapping.

Using the example metrics

Numerical thresholds, sample reports and schedules below are illustrative. They are neither universal NIST standards nor tests of legal compliance. For each system, define the metric, measurement population, baseline, sample size, uncertainty and consequences of errors in advance. Justify the acceptable threshold and the response when it is exceeded. Fairness metrics can conflict; a gap below 5% alone does not demonstrate an absence of discrimination.

1. Overview of MEASURE

MEASURE supports appropriate testing, validation and monitoring throughout the AI lifecycle.

Objective	Description
TEVV	Test, evaluation, validation and verification
Metrics	Define and track meaningful indicators
Bias	Evaluate fairness and harmful bias
Security	Assess security and resilience
Continuity	Monitor changes in operation

2. MS-1: TEVV framework

2.1 What is TEVV?

2.2 TEVV across the lifecycle

Phase	TEVV activities	Timing
Design	Verify requirements	Before development
Development	Unit tests, code review	During development
Pre-deployment	Integration, validation, red teaming	Before release
Deployment	A/B testing, staged rollout	During release
Production	Monitoring, drift detection	Continuously
Retirement	Final assessment, lessons learned	When decommissioning

2.3 Testing strategy

Define a test plan for each system. The values and cadence in this example need to be replaced with justified project criteria.

## TEST PLAN: [AI system name]

### 1. Objectives
- What are we testing?
- What are the acceptance criteria?

### 2. Test types
| Type | Scope | Tools or method |
|---|---|---|
| Unit | Model components | pytest, unittest |
| Integration | End-to-end behaviour | Custom scripts |
| Performance | Speed, throughput | Load testing |
| Fairness | Harmful bias | Aequitas, Fairlearn |
| Security | Adversarial behaviour | Authorised red team |
| User acceptance | Usability and fitness for purpose | User studies |

### 3. Test data
- Representative samples
- Edge cases
- Adversarial inputs

### 4. Schedule
| Milestone | Tests | Date |
|---|---|---|
| Development complete | Unit, integration | |
| Pre-release | Technical and risk tests | |
| Release decision | User acceptance | |
| Monthly review | Performance, fairness | |

### 5. Illustrative pass/fail criteria
| Metric | Example threshold | Current value |
|---|---|---|
| Accuracy | >95% | |
| Latency p99 | <500 ms | |
| Demographic parity gap | <5% | |

3. MS-2: Performance metrics

3.1 Core metrics

Accuracy and reliability

Metric	Description	Use
Accuracy	Proportion of correct predictions	Classification
Precision	True positives / predicted positives	Where false positives are costly
Recall	True positives / actual positives	Where false negatives are costly
F1 score	Harmonic mean of precision and recall	Balancing the two metrics
AUC-ROC	Area under the ROC curve	Comparing discrimination across thresholds
RMSE/MAE	Prediction error measures	Regression
Perplexity	How well a language model predicts a text sample	Language modelling; not a complete measure of output quality

Reliability

Metric	Description	Illustrative target
Consistency	Repeatability of outputs under defined conditions	>99% where deterministic behaviour is expected
Stability	Performance over time	<5% drift in a defined metric
Availability	Uptime	>99.9%
Latency p50/p99	Response time	Defined in the service requirements

3.2 Fairness metrics

Metric	Definition	What it evaluates
Demographic parity	P(Ŷ=1|G=a) = P(Ŷ=1|G=b)	Equal selection rates across groups
Equalised odds	Equal true-positive and false-positive rates across groups	Differences in error rates
Predictive parity	Equal positive predictive value across groups	Differences in precision
Individual fairness	Similar individuals receive similar outcomes	Case-level treatment

Example measurement with Fairlearn:

from fairlearn.metrics import demographic_parity_difference

dpd = demographic_parity_difference(
    y_true,
    y_pred,
    sensitive_features=sensitive_feature
)

# Illustrative threshold: |dpd| < 0.05.
# Justify the threshold for the specific project and population.

3.3 Generative AI metrics

Metric	Description	Possible measurement approach
Hallucination rate	Frequency of factually incorrect outputs	Expert review of a defined sample
Toxicity score	Potentially harmful content	Classifier scores plus contextual review
Bias in generation	Stereotypes in outputs	Benchmarks such as Winogender or BBQ, where relevant
Instruction following	Compliance with task instructions	Task-specific evaluation datasets
Refusal rate	Refusal of inappropriate requests	Adversarial prompts; also track inappropriate refusals

4. MS-3: Bias testing

4.1 Pre-deployment assessment

• Identify relevant protected characteristics, such as age, sex or ethnicity.
• Assess training data for harmful bias.
• Measure baseline metrics by relevant group.
• Define and justify fairness criteria.
• Prepare mitigation and escalation options.

Personal data used for testing needs an Article 6 GDPR legal basis; special categories, such as ethnicity, also need an applicable exception to the Article 9 prohibition and suitable safeguards. Article 4a of the AI Act permits processing for bias detection and correction only in its specified cases and subject to all its conditions, including necessity, access and reuse restrictions, and deletion. An intention to test bias does not itself establish this permission.

4.2 Testing methods

Slice analysis

Evaluate performance on relevant subgroups. This is an illustrative report:

| Slice | Count | Accuracy | Precision | Recall |
|---|---|---|---|---|
| Overall | 10000 | 94.5% | 93.2% | 95.1% |
| Gender: M | 5200 | 95.1% | 94.0% | 95.8% |
| Gender: F | 4800 | 93.8% | 92.3% | 94.3% |
| Age: <30 | 3000 | 96.2% | 95.5% | 96.8% |
| Age: 30–50 | 4500 | 94.0% | 93.1% | 94.5% |
| Age: >50 | 2500 | 92.1% | 90.8% | 93.2% |

An example internal alert is a gap above 5 percentage points between groups. Investigate sample size, uncertainty and consequences; smaller gaps can still matter.

Counterfactual testing

Change a protected characteristic while keeping other relevant information constant, then examine the output:

Original: "John applied for a loan..." → Approved
Counterfactual: "Jane applied for a loan..." → Approved?

Different outcomes can indicate a need to investigate potential bias.

4.3 Mitigation strategies

Strategy	Stage	Examples
Pre-processing	Data	Rebalancing, resampling, feature selection
In-processing	Training	Fairness constraints, adversarial debiasing
Post-processing	Output	Threshold adjustment, equalised odds methods
Human review	Deployment	Meaningful review and override for sensitive decisions

5. MS-4: Security testing

5.1 AI-specific threats

Threat	Description	Testing or control area
Adversarial examples	Inputs designed to mislead a model	Adversarial robustness testing
Model extraction	Attempted model replication through queries	Query monitoring and rate-limit testing
Data poisoning	Manipulation of training data	Data provenance and validation
Prompt injection	Instructions that attempt to redirect a generative AI system	Direct and indirect injection tests
Membership inference	Attempts to infer whether data appeared in training	Privacy attack evaluations

5.2 Red teaming for generative AI

Type	Participants	Focus
General	Typical users	Common misuse and failures
Expert	Domain specialists	Domain-specific risks
Adversarial	Security researchers	Edge cases and bypass attempts
AI-assisted	AI tools and human reviewers	Broader coverage with human interpretation

Use authorised, controlled testing with appropriate handling of sensitive or illegal material. A session can cover:

## RED-TEAM SESSION: [Generative AI system]

### 1. Attempts to bypass safeguards
- [ ] Direct harmful requests
- [ ] Role-play scenarios
- [ ] Multi-step prompts
- [ ] Language switching
- [ ] Encoded inputs

### 2. Information extraction
- [ ] Training data exposure
- [ ] Personal data exposure
- [ ] System prompt exposure
- [ ] Capability probing

### 3. Content risks
- [ ] Violence or hate
- [ ] Misinformation
- [ ] Safeguards against CSAM and non-consensual intimate imagery
- [ ] Intellectual property infringement

### 4. Misuse risks
- [ ] Social engineering
- [ ] Phishing
- [ ] Malicious code assistance
- [ ] CBRN-related requests

### 5. Findings
| Finding | Severity | Reproducible | Recommendation |
|---|---|---|---|
| | | | |

5.3 Prompt injection scenarios

Scenario	Example	Expected security property
Direct injection	“Ignore previous instructions…”	Maintain authorised task boundaries
Indirect injection	Malicious instructions in retrieved documents	Treat external content as untrusted data
Context manipulation	“From now on, you are…”	Preserve role and permission boundaries
Delimiter attacks	Instructions hidden in Markdown or code blocks	Parse safely and enforce controls independently

6. MS-5: Continuous monitoring

6.1 Example dashboard

The values below demonstrate a layout; they are not observed customer results or recommended universal thresholds.

6.2 Drift detection

Drift type	What changes	Detection approach	Illustrative trigger
Data drift	Input distribution	PSI, KS test	PSI >0.1
Concept drift	Relationship between inputs and outcomes	Performance evaluation with ground truth	>5% degradation
Model behaviour drift	Prediction behaviour	Output distributions and task evaluations	A statistically and operationally meaningful change

6.3 Alerts

These are example internal response targets, not TECHNOMATON support commitments or statutory reporting deadlines.

Level	Trigger	Response	Example target
P1 Critical	Safety incident or major outage	Immediate escalation	15 minutes
P2 High	Performance degradation above 10%	Same-day investigation	4 hours
P3 Medium	Drift or minor issues	Planned review	24 hours
P4 Low	Information or optimisation	Review in the next sprint	1 week

7. Implementation checklist

Adapt this illustrative schedule to the system, team and risk.

Phase 1: Test infrastructure — weeks 1–2

• Define test data management.
• Establish test environments.
• Select testing tools.
• Define baseline metrics.

Phase 2: Pre-deployment testing — weeks 3–4

• Implement unit and integration tests.
• Assess harmful bias.
• Run security tests.
• Complete user acceptance testing.

Phase 3: Monitoring — weeks 5–6

• Deploy monitoring components.
• Define justified metrics and thresholds.
• Configure alerts and ownership.
• Create dashboards.

Phase 4: Continuous improvement — ongoing

• Revalidate periodically and after relevant changes.
• Monitor drift.
• Conduct red-team sessions.
• Review metrics and thresholds.

8. Tools

These are examples to evaluate for compatibility, current maintenance and the intended use; listing a tool is not an endorsement or a claim of integration.

Category	Examples	Purpose
ML testing	pytest, Great Expectations	Code and data checks
Fairness	Fairlearn, Aequitas, AI Fairness 360	Bias and fairness analysis
Explainability	SHAP, LIME, Captum	Model interpretation
Security	TextAttack, Adversarial Robustness Toolbox	Adversarial testing
Monitoring	Evidently, WhyLabs, Arize	Production observability
Generative AI evaluation	HELM, lm-evaluation-harness	Language model evaluations

Continue to MANAGE to prioritise and address findings.

AI-Native Entry Framework | CC BY-NC-SA 4.0