Generative AI risks

Twelve generative AI risk areas and ways to manage them

Source: NIST AI 600-1, Generative Artificial Intelligence Profile, July 2024

1. Introduction

Generative AI creates text, images, audio or other content from learned patterns. This guide draws on NIST AI 600-1. It is not a legal definition or a binding severity classification.

The NIST AI RMF and its generative AI profile are voluntary. The controls, templates and priorities below are our suggested examples to adapt; they do not reproduce NIST actions verbatim or determine statutory duties.

What makes these risks distinctive?

Aspect	Predictive or classification AI	Generative AI
Outputs	Predictions and classifications	Synthetic content
Predictability	Depends on the model and inputs	Also depends on generation settings and interaction context
Scale	Depends on architecture and infrastructure	Depends on architecture and infrastructure
Misuse	Depends on purpose, access and impact	Additional scenarios involving synthetic content and connected tools
Unexpected behaviour	Possible outside tested conditions	Can also emerge from open-ended tasks and interactions

2. Overview of the 12 risk areas

The grouping below is an example of internal prioritisation, not a severity rating assigned by NIST. Environmental impacts, for example, depend on the scale and conditions of operation.

3. Risk areas in detail

R1: CBRN information or capabilities

Scope: easier access to information or capabilities that could enable chemical, biological, radiological or nuclear harm.

Possible scenarios include assistance with dangerous substances, biological design tools that extend specialist capabilities, and combinations of generative AI with other tools that increase the potential for harm.

Control	Type	Implementation example
Content filtering	Preventive	Restrict inappropriate CBRN-related assistance
Query monitoring	Detective	Identify suspicious patterns within an appropriate monitoring policy
User verification	Preventive	Verify access for sensitive use cases
Incident escalation	Corrective	Escalate to appropriate specialists or authorities where relevant

Legal assessment: examine the precise prohibited practices and other applicable laws. CBRN as a whole is not a separate prohibited category under AI Act Article 5.

R2: Confabulation

Scope: content that is presented confidently but is false or misleading.

Language models generate plausible continuations rather than automatically checking every statement against ground truth. Errors can include invented legal citations, incorrect medical information, fabricated facts in articles or inaccurate technical instructions.

Control	Type	Implementation example
Fact-checking policy	Preventive	Define which outputs require independent verification
Retrieval-augmented generation	Preventive	Use reviewed sources; retrieval does not guarantee correct citations or eliminate hallucinations
Calibrated indicators	Detective	Use validated uncertainty measures; confident wording is not a probability of correctness
Human review	Detective	Specialist review of consequential outputs
Automated checks	Detective	Detect selected inconsistencies and verify claims where possible
User education	Preventive	Explain capabilities and limitations

Example internal policy:

## VERIFYING GENERATIVE AI OUTPUTS

1. Do not use generative AI as the sole authority for legal documents,
   medical information, financial decisions or critical infrastructure.
2. Verify consequential factual claims using appropriate independent sources.
3. Disclose AI involvement where required by law, contract or internal policy.
4. Record significant false outputs and use them to improve evaluation.

R3: Dangerous, violent or hateful content

Scope: easier production of violent, radicalising or hateful content.

Risk scenarios include bypassing safeguards, generating extremist material, instructions for violence or self-harm, and spreading hatred.

Control	Type	Implementation example
Content moderation	Preventive	Input and output controls
Safeguard testing	Preventive	Regular authorised tests of bypass resistance
Acceptable use rules	Preventive	Clear terms and prohibited behaviour
Appropriate event records	Detective	A proportionate audit trail
Reporting mechanism	Corrective	Accessible reporting and escalation
Account restrictions	Corrective	Enforce the applicable policy

R4: Data privacy

Scope: exposure of personal data from training or use, and inference of sensitive information.

Models may reproduce personal data, infer sensitive attributes from apparently ordinary information, or enable processing that conflicts with purpose limitation and other GDPR principles.

Control	Type	Implementation example
Data classification	Preventive	Define what may enter each approved service
Personal data detection	Detective	Scan relevant outputs with human escalation
Training restrictions	Preventive	Agree and configure appropriate limits on provider use of data
DPIA	Preventive	Complete where processing is likely to result in high risk under GDPR Article 35
Anonymisation	Preventive	Apply suitable preprocessing and assess re-identification risk
Access controls	Preventive	Restrict users, systems and connected tools

Privacy checklist:

Establish a legal basis for processing.
Provide appropriate information to data subjects.
Minimise personal data.
Respect purpose limitation.
Enable applicable data subject rights.
Put an Article 28 GDPR contract or other legal act in place where the provider acts as a processor.
Complete a DPIA where required.

R5: Environmental impacts

Scope: environmental effects of training and operating generative AI models.

Consumption depends on the model, hardware, utilisation and workload.
Emissions depend on the energy mix and measurement boundary.
Water impacts depend on cooling and energy production in the relevant location. A universal conversion such as “one language model equals a fixed number of flights” is not useful.

Control	Type	Implementation example
Provider assessment	Preventive	Compare relevant, supported environmental information
Model efficiency	Preventive	Use a smaller model where it meets task requirements
Caching	Preventive	Reduce unnecessary repeated work where appropriate
Footprint measurement	Detective	Track consumption and estimates with stated boundaries
Relevant reporting	Detective	Include material impacts in applicable reporting

R6: Harmful bias and homogenisation

Scope: stereotyped outputs, unequal performance across groups and excessive uniformity of information or decisions.

Examples include gender stereotypes, poorer performance for minority groups or languages, degradation associated with unsuitable synthetic training data, and correlated outcomes from widespread use of similar models.

Control	Type	Implementation example
Bias assessment	Preventive	Evaluate before deployment
Diverse evaluation	Preventive	Test relevant groups, languages and contexts
Fairness monitoring	Detective	Track justified metrics and investigate impacts
Human oversight	Detective	Meaningful review in consequential uses such as hiring
Diverse expertise	Preventive	Include different perspectives in evaluation
Feedback channels	Corrective	Enable people to report harmful outcomes

Possible evaluation datasets include WinoBias and Winogender for gender bias, BBQ for social biases, and RealToxicityPrompts for toxicity. Assess whether each dataset fits the intended use; a benchmark result is not a universal fairness verdict.

R7: Human–AI configuration

Scope: risks arising from interaction between people and AI, including overreliance, unwarranted distrust and anthropomorphism.

Risk	Description	Possible response
Automation bias	Excessive trust in outputs	Critical thinking and verification practices
Algorithmic aversion	Unjustified rejection of useful outputs	Relevant explanations and evidence
Emotional entanglement	Unhealthy reliance on an AI interaction	Clear identity, appropriate boundaries and support options
Anthropomorphism	Mistaking human-like language for human qualities	Interface design that communicates the system’s role and limits

Example internal interaction guidance:

## HUMAN–AI INTERACTION

1. Make the system's identity clear.
   - Tell users when they are interacting with an AI assistant.
   - Avoid design choices that mislead users about its role or capabilities.

2. Support critical judgement.
   - AI can make mistakes.
   - Verify consequential information.
   - Assign accountable people and meaningful decision authority.

3. Set appropriate boundaries.
   - Do not present the system as a substitute for qualified human support.
   - Do not infer human feelings or understanding from conversational style.
   - Provide a suitable route to human help when needed.

R8: Information integrity

Scope: misinformation, deliberate disinformation, deepfakes and erosion of trust.

Scenarios include large-scale fabricated news, synthetic video or audio, invented evidence and content that undermines confidence in reliable information.

Control	Type	Implementation example
Content provenance	Preventive	Provenance metadata or watermarking where suitable
Synthetic media detection	Detective	Detection tools used with known limitations
Disclosure	Preventive	Labelling required by law, contract or policy
Fact-checking	Detective	Verify relevant claims and sources
Media literacy	Preventive	Train users to assess information critically

Provenance approaches include digital watermarking, metadata standards such as C2PA, fingerprinting and authenticated publishing processes. Provenance can help establish origin and editing history; it does not by itself prove that a claim is true.

R9: Information security

Scope: generative AI as both a tool for and a target of cyberattacks.

Potential misuse includes phishing, malicious code assistance and social engineering. Attacks against AI systems include prompt injection, data poisoning and model extraction.

Control	Type	Implementation example
Input validation	Preventive	Validate schema and size; cleaning prompt text alone is insufficient
Prompt injection controls	Preventive	Separate untrusted content and limit the impact of a bypass
Rate limits	Preventive	Restrict abuse
Access controls	Preventive	Authenticate and authorise actions
Logging	Detective	Maintain a proportionate audit trail
Incident response	Corrective	Prepare an AI-specific response procedure

To reduce prompt injection risk:

Treat user text, websites, documents and tool outputs as untrusted input. Instructions inside them must not grant permissions.
Enforce permissions and action parameter validation outside the model. Apply least privilege, data isolation and appropriate outbound communication limits.
Require an authorised person to approve consequential actions. Show the specific action and target being approved.
Validate outputs before using them in code, SQL, HTML or another tool. Context-aware escaping addresses those injection surfaces; it is not a universal defence against prompt injection.
Test direct and indirect attacks and repeat relevant tests after model changes. Logging, filters and structured prompts provide additional layers, not a guarantee that the risk is eliminated.

See OWASP: Prompt Injection.

R10: Intellectual property

Scope: infringement, reproduction of training material and uncertainty about rights in inputs and outputs.

Scenarios include reproducing protected content, misusing a person’s identity and misunderstanding rights in generated material. The legal assessment depends on the jurisdiction, material, human contribution and intended use.

Control	Type	Implementation example
IP policy	Preventive	Rules for inputs, outputs and publication
Rights review	Detective	Review before publication or commercial use
Attribution and disclosure	Preventive	Meet applicable licence, contractual and legal requirements
Contract review	Preventive	Understand supplier terms and risk allocation
Training data information	Detective	Review available supplier documentation

Example policy structure:

## INTELLECTUAL PROPERTY AND GENERATIVE AI

1. Rights in outputs
   - Treat generated material as input to further work and review.
   - Assess human contribution and applicable rights before publication.

2. Permitted inputs and use
   - Use third-party material only where the proposed use is permitted.
   - Do not make unsupported claims of originality or ownership.

3. Attribution
   - Disclose AI assistance where relevant or required.
   - Follow applicable licences and supplier terms.

4. Due diligence
   - Check outputs for infringement and misleading similarity.
   - Record relevant sources, permissions and review decisions.

R11: Obscene, degrading or abusive content

Scope: harmful content including child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII).

An organisation should define strict prohibitions and an effective response process for these forms of abuse.

Control	Type	Implementation example
Prohibited-use policy	Preventive	Clear rules and technical restrictions
Content controls	Preventive	Appropriate filtering and access boundaries
Detection	Detective	Suitable tools and trained escalation roles
Reporting and response	Corrective	Follow applicable law and the relevant authority or platform process
Controlled testing	Preventive	Verify safeguards without creating or distributing illegal material

Define the applicable reporting process with qualified specialists for the jurisdiction, organisation’s role and event, including the recipient and whether a duty to report to the US National Center for Missing & Exploited Children arises. Include protection of affected people and lawful evidence handling.

AI Act: the new Article 5(1)(ba) and (bb) prohibitions, added by Regulation (EU) 2026/1744, apply from 2 December 2026. They cover specified realistic intimate or sexually explicit material depicting an identifiable person without freely given, specific, informed, unambiguous and explicit consent, and material or performances defined in Article 2(c) and (e) of Directive 2011/93/EU; (bb) preserves grounds excluding unlawfulness under national law. Article 5(1a) limits placing on the market or putting into service to cases involving either the system’s intended purpose, or a reasonably foreseeable and reproducible outcome without significant technical modification where proportionate and appropriate safeguards reliably preventing such generation or manipulation and correcting identified or reported misuse are absent. Prohibited use requires use for that purpose. For the prohibition in point (ba) only, Article 5(1b) does not treat as manipulation changes that neither increase exposure of depicted intimate parts nor change the nature of explicitly depicted sexual activities. Assess the complete Article 5(1a)–(1b) and Article 113(a).

R12: Value chain and component integration

Scope: risks from third-party components, unclear provenance and insufficiently assessed dependencies.

Examples include an unknown origin of a pretrained model, biased external datasets, vulnerable libraries and errors in benchmark labels.

Control	Type	Implementation example
Supplier due diligence	Preventive	Assessment before acquisition
Model documentation	Preventive	Review model cards and relevant limitations
Component inventory	Detective	Maintain software and model dependency records
Contractual requirements	Preventive	Agree relevant service and security conditions
Exit planning	Preventive	Prepare for replacing a supplier or component
Periodic review	Detective	Monitor relevant changes and vulnerabilities

4. Implementation matrix

Example starting priorities

These are discussion prompts, not fixed sector ratings. A system’s purpose, users and possible harm determine the actual priorities.

Organisation or use context	Example starting risk areas
General enterprise use	R2, R4, R9, R6, R12
Healthcare	R2, R4, R6, R11, R1
Finance	R2, R4, R6, R9, R10
Media and marketing	R8, R10, R2, R6, R3
Education	R2, R6, R4, R8, R7
Critical infrastructure	R9, R1, R8, R12, R4

Possible early improvements

Action	Illustrative effort	Potential impact	Risk areas
Define verification for consequential claims	Low	High	R2, R8
Establish personal data rules for approved tools	Low	High	R4
Review content safeguards	Medium	High	R3, R11
Clarify AI identity and disclosure	Low	Medium	R7, R8
Assess relevant suppliers	Medium	High	R12

5. Assess the risks in your organisation

Use a quick scan to identify questions that need deeper analysis. For a full assessment, document the system, applicable risk areas, evidence, existing controls, gaps, owners and residual-risk decisions. A questionnaire count or a completion percentage does not establish compliance.

The assessment overview explains how to choose the right depth. Combine these risk areas with MAP, MEASURE and MANAGE.

6. References

AI-Native Entry Framework | CC BY-NC-SA 4.0