AI governance and risk

1. What AI governance means

AI governance is the set of responsibilities, policies, decision processes, and evidence used to manage AI systems throughout their lifecycle. It helps organizations decide which uses are acceptable, how risks are assessed, who approves deployment, how people are affected, and when a system needs review or retirement.

Governance is not a single document or a claim that an AI system is “safe.” It connects product intent and organizational accountability to engineering controls, measurement, oversight, and response.

Use case → Inventory → Risk assessment → Controls and owners
    → Evaluation and approval → Monitoring → Review or retirement
              ↖ changes, incidents, and new evidence ↙

For an application security engineer, governance is a supporting skill: translate requirements into testable controls, surface technical limitations, and provide reliable evidence to the people accountable for risk decisions.

2. AI use-case inventory and ownership

An inventory helps teams discover AI systems and avoid “shadow AI”—tools or model integrations used without the right visibility and review. It can begin as a simple register and grow to match the organization's risk and obligations.

For each use case, record enough information to locate the system and route decisions:

• Purpose, intended users, affected people, business owner, and technical owner.
• System type: hosted model, internal model, retrieval-augmented generation (RAG), agent, or third-party AI feature.
• Data inputs and outputs, sensitivity, source, retention, access rules, and cross-border or tenant boundaries where applicable.
• Model/provider, relevant versions, integrations, tools/actions, dependencies, and deployment environment.
• Risk assessment status, applicable reviews, key mitigations, evaluation evidence, approval conditions, and next review date.

Keep the inventory proportionate. Do not collect sensitive prompts or full datasets merely to fill in a register; record useful metadata and use approved systems for any sensitive evidence.

3. AI risk and impact assessment

Risk assessment asks what could go wrong, who might be affected, how severe the effects could be, and what controls reduce the chance or impact. Consider the intended use and plausible misuse—not only model accuracy.

• Security: prompt injection, data leakage, unauthorized tool calls, compromised dependencies, abuse, and denial of service.
• Privacy and data protection: collection, purpose, access, retention, deletion, sensitive data, and unintended inference.
• Reliability and safety: incorrect or inconsistent outputs, failures under unusual inputs, overreliance, and impacts of automated actions.
• Fairness and user impact: who benefits or is disadvantaged, accessibility, meaningful human review, and ways to contest important outcomes.
• Transparency and accountability: how people know AI is involved, who owns decisions, and how limitations and incidents are communicated.
• Legal and contractual obligations: use context, jurisdiction, data rights, sector rules, procurement terms, and provider commitments.

Rate risks using your organization's process, state assumptions, assign owners, and record residual risk and review dates. A framework helps structure thought; it does not determine legal applicability or make an inherently unacceptable use acceptable.

4. AI data governance, privacy, and quality

AI systems may use prompts, uploaded files, training or fine-tuning data, retrieved documents, telemetry, and generated outputs. Track these data flows just as you would for any application, and pay special attention to what crosses a model-provider or tenant boundary.

• Define what data the feature needs; minimize collection and exclude secrets or personal data that are not necessary.
• Confirm user authorization before retrieval and tool access. Enforce access in application code and data services, not by asking a model to respect instructions.
• Record where data is processed, how long it is kept, whether it is used for provider training, and how deletion requests are handled.
• Protect datasets and prompts in logs, evaluation tools, analytics, and support workflows; restrict access and retention.
• Document relevant data provenance, quality limitations, consent or usage rights, and known gaps.
• Test for cross-user and cross-tenant disclosure, including retrieval, caches, citations, and generated summaries.

Privacy and security are related but distinct. A technically secure system can still use data in a way that is unexpected, excessive, or inconsistent with its approved purpose.

5. Model and vendor risk

Using an external model or AI service introduces dependencies beyond ordinary uptime. Review what the provider promises, what the product actually configures, and which risks remain with your organization.

• Understand input/output retention, training use, data location, subprocessors, deletion, and breach notification terms.
• Review service availability, version changes, model deprecation, rate limits, and incident communication.
• Identify the provider's security documentation, independent assurance, access controls, and support for audit or investigation.
• Record how model or API changes are evaluated before rollout, including regressions in safety, quality, privacy, and security.
• Plan for outages, provider changes, account compromise, and a way to disable or replace the dependency.

Vendor questionnaires are evidence inputs, not guarantees. Validate important claims against contracts, architecture, configuration, and tests, with procurement and legal teams as appropriate.

6. Lifecycle governance and controls

AI governance continues after an initial approval. Changes to prompts, models, datasets, retrieval sources, tools, user groups, and deployment context can change risk and should trigger review based on defined thresholds.

1. Propose: state the intended use, users, expected benefit, affected people, and alternatives.
2. Assess: inventory the system, map data and dependencies, assess risks, and identify applicable policy or legal reviews.
3. Design: assign control owners for access, privacy, security, human oversight, evaluation, and fallback behavior.
4. Evaluate: test realistic tasks, misuse cases, subgroup or context limitations where relevant, security boundaries, and failure modes.
5. Approve and deploy: record accountable decisions, conditions, monitoring expectations, rollback criteria, and user communications.
6. Monitor and change: review incidents, metrics, provider/model changes, complaints, and new evidence; reassess or retire when necessary.

For AI security specifically, ensure that a model cannot independently grant access. Enforce authorization in the application, limit tool permissions, validate outputs, require confirmation for consequential actions, and retain proportionate audit evidence.

7. Monitoring, incidents, and accountability

AI incidents can include personal or confidential data exposure, unsafe or unauthorized actions, misleading outputs with material impact, harmful bias, model/provider compromise, or sustained service abuse. The response process should make it clear who can contain, investigate, communicate, and decide whether service resumes.

• Define escalation routes spanning product, security, privacy, legal, safety, and operations teams as appropriate.
• Monitor security and service indicators without collecting excessive prompt or personal data.
• Preserve enough version, configuration, and event information to investigate while applying retention and access limits.
• Prepare containment actions: disable a tool, revoke a credential, restrict a data source, roll back a model/prompt, or pause the feature.
• Use incidents and near misses to improve the risk assessment, controls, tests, and user guidance.

Accountability must be assigned to people and organizational roles. A model, checklist, or governance framework cannot take responsibility for a deployment decision.

8. Frameworks and regulation: useful context, not a substitute for advice

Frameworks offer ways to organize risk and management practices. Regulations impose obligations when they apply. Applicability depends on the system, intended use, organization, geography, sector, and role; confirm decisions with qualified legal and compliance professionals.

• NIST AI Risk Management Framework (AI RMF): organizes risk work around Govern, Map, Measure, and Manage. It is voluntary guidance, not a law.
• ISO/IEC 42001: specifies requirements for an AI management system within a defined organizational scope. Certification, where sought, concerns that management system and scope; it does not prove a model cannot fail.
• ISO/IEC 23894: provides guidance for managing AI-related risk.
• EU Artificial Intelligence Act: establishes obligations based on system risk, intended purpose, and organizational roles, with application dates and duties that phase in. Check the current official text and timeline.
• OWASP Top 10 for LLM Applications and MITRE ATLAS: security awareness and adversary resources useful for testing; neither is a regulation or a complete governance system.

For the security-engineering path, learn enough of these resources to explain how technical findings map to risks, owners, controls, and evidence. See the standards and compliance guide for an overview.

9. Practice and recap

Continue with the AI Application Security guide for technical controls, or visit the Level 4 technology roadmap.

Further resources

Free practical resources

• NIST AI Risk Management Framework — voluntary guidance for managing AI risks across the lifecycle.
• NIST AI RMF Playbook — free suggested actions and questions organized around the AI RMF functions; adapt relevant items to a use case instead of treating it as a checklist.
• NIST AI RMF Core and Profiles — browse the outcomes, then use the fictional use-case exercise above to turn one outcome into owners and evidence.
• ISO/IEC 42001 and ISO/IEC 23894 — AI management-system requirements and AI risk-management guidance.
• EU Artificial Intelligence Act (official legal text) — consult the current text and seek qualified advice on applicability.
• OWASP Top 10 for LLM Applications and MITRE ATLAS — security risk and adversary references to complement governance work.

Watch AI governance playlists and talks

• IBM Technology: AI Ethics and Governance Explained — accessible introductory material from a channel with approximately 1.81M subscribers.

Subscriber counts are approximate and change over time; video view counts vary. This playlist is an introduction, not compliance training; use the official NIST AI RMF, ISO, and legal references above for authoritative requirements.

This tutorial is educational, not legal advice, a compliance assessment, or a certification checklist. Laws and standards can change; verify current requirements with qualified professionals.