AI Governance Checklist: Your LLM Architecture Comes First

Deloitte reports a 53-point gap between AI ambition and AI governance maturity. 74% of enterprises plan to deploy agentic AI within two years, while only 21% have a mature model for governing autonomous agents.

For engineering teams, an AI governance checklist only becomes useful when it maps to architecture, because policy language cannot show who called which model, which provider handled the request, or where the audit trail lives.

Three routing postures, managed gateway, self-hosted gateway, and direct API, map differently against the governance requirements your LLM stack can actually satisfy.

Where the governance gap becomes concrete

The governance gap shows up when agentic AI leaves pilot projects and enters production workflows. For engineering leads, that gap becomes concrete when model calls touch customer context, route through external providers, and create audit questions the stack cannot answer.

Deloitte also found data privacy and security ranking among leaders’ top concerns about AI. That puts engineering leads in an awkward position. Your team may already route model calls into production workflows while governance still lives in a spreadsheet, a policy document, or a meeting agenda.

Why a 53-point gap matters

The gap matters because agentic AI does not stay inside demos. It calls tools, touches customer context, triggers downstream workflows, and creates spend that someone eventually has to explain.

Your stack has to answer who called which model, which provider processed the request, what it cost, and where the audit trail lives.

Why governance checklists don’t produce audit logs

OWASP gives compliance leaders a useful starting point. The OWASP Top 10 for LLM Applications Cybersecurity and Governance Checklist is written for “leaders across executive, tech, cybersecurity, privacy, compliance, and legal areas, DevSecOps, MLSecOps, and Cybersecurity teams and defenders.” That scope makes sense when the job is risk assessment, vendor management, steering committees, and policy ownership.

Checking governance areas produces a PDF, not spend visibility, provider attribution, or a single auditable endpoint. That PDF can guide the program, but it cannot show whether an internal tool used approved models for high-risk use cases, or produce the audit trail your team needs on Monday.

Your checklist may be right while your stack still cannot prove what happened.

Your routing architecture is your first governance layer

The first step in LLM governance is routing model traffic through an architecture that can record usage, provider, cost, and access decisions. This is the point where API governance stops being a policy phrase and becomes an engineering surface. If the request path cannot capture the evidence, the governance program starts with a blind spot.

Three architectural postures

A managed gateway sends model calls through a third-party routing layer for provider access, usage records, and routing controls.

A self-hosted gateway puts that layer inside your infrastructure, trading more control for more operational responsibility.

Direct API access sends each service straight to model providers, leaving governance primitives to each team.

What each posture gives you by default

Here is the practical baseline for what each routing posture can prove before your team adds custom controls.

Posture	Spend visibility	Provider attribution	Data sovereignty	RBAC	Audit trail
Managed gateway (OpenRouter, Portkey)	Yes. OpenRouter activity dashboard, per API key	Yes. Provider attribution per request	Yes. Zero Data Retention routing controls	Per-key and workspace controls; full RBAC in Portkey	Usage and provider records; full audit logs in Portkey
Self-hosted gateway (LiteLLM)	Yes. Spend tracking with virtual keys	Yes. Logs provider per route	Yes. Infrastructure and logs stay under your control	Yes. LiteLLM Enterprise	Yes. LiteLLM Enterprise audit logs plus Prometheus
Direct API	No	No	No shared control plane	No	No

The table shows which governance evidence exists by default and which your team still has to build.

Why direct API access leaves gaps

Direct API access gives you zero governance primitives. Six provider keys scattered across twelve microservices won’t give you one place to inspect spend, provider attribution, access controls, or audit history. The moment usage spreads beyond one team, your routing architecture becomes the difference between a report and a cleanup project.

The next step is to turn that architecture baseline into a governance checklist your stack can actually answer.

A governance checklist that maps to your stack

A useful governance checklist separates controls your routing layer can prove from controls your organization still has to own. Treat the checklist as five pillars:

• Inventory: which AI systems, models, and providers are in use
• Accountability: who owns each system, tool, and approval path
• Access: which teams can use approved tools for high-risk use cases
• Evidence: which logs support audit trail, data residency, and spend questions
• Compliance: which controls require policy review, bias assessment, incident response plans, or EU AI Act documentation

The engineering question is which ones your current stack can prove without a scramble.

AI inventory and model tracking

Your AI inventory starts with the request path. If every model call routes through one endpoint, you can see which applications use which models and how that usage changes over time. If each team calls providers directly, your inventory exists only where someone remembered to document it.

A spreadsheet can list approved systems. A routing layer can show whether production traffic matches the list.

Spend visibility and budget controls

Spend visibility is often the earliest warning system you get. OpenRouter exposes per-key usage through the activity dashboard, which gives engineering leads a place to inspect usage before finance or security turns it into a fire drill. Self-hosted gateways can provide the same view once your team configures tracking, dashboards, and alerts.

Direct API access leaves you with provider billing pages and whatever each service logs. That can work while one team owns one feature. It breaks down when multiple teams ship AI features against different providers and no one can answer which project created the spike.

Provider attribution and data residency

If your prompt contains customer data, the provider that handled it becomes a compliance-relevant fact. You need to know where requests flow, not just which model responded.

OpenRouter exposes provider attribution per request and supports Zero Data Retention routing. Direct API access leaves each provider relationship isolated.

Access controls, approved tools, and high-risk use cases

High-risk use cases, such as hiring, lending, healthcare triage, or financial decisions, need tighter rules than internal summarization.

OpenRouter scopes access at the API-key and workspace level, so you can separate keys by team or application and set per-key budgets. Teams that need centralized RBAC and route-level enforcement inside one platform can layer on Portkey or LiteLLM Enterprise.

Audit trail and continuous monitoring

These controls matter for any governance program because they prove what happened after deployment, not just what the policy intended.

A managed gateway can quickly provide part of that evidence. A self-hosted gateway can give you deeper evidence if you build logging and monitoring around it. Direct API access gives you no shared trail unless your team creates one across every provider integration.

EU AI Act, agentic AI, and what architecture cannot satisfy

The EU AI Act adds requirements that no routing layer satisfies by itself. For high-risk AI systems, the regulation covers obligations around technical documentation, transparency, human oversight, conformity assessment, post-market monitoring, and serious-incident reporting under Regulation (EU) 2024/1689. Those requirements need process, ownership, and review outside the request path.

Agentic AI makes that boundary more important because autonomous systems can call tools, take actions, and move through workflows without a human approving each step. Your routing layer can show what the system called and where the request went, but your governance framework still has to define when humans intervene and how incidents get handled.

What a managed gateway gives you, and what it doesn’t

A managed gateway gives you a fast governance baseline rather than a complete governance program. It helps you centralize model access, see usage, and control routing, but it does not remove the need for policy, procurement review, or workload-specific compliance decisions.

What OpenRouter provides by default

The value is centralization. Instead of scattered provider keys across services, you get one routing layer for model access, provider selection, usage visibility, and data-policy controls such as Zero Data Retention routing. That gives engineering leads a practical place to start answering who called what model, which provider handled it, and what it cost.

Those controls matter because they sit close to the request path. They reduce cleanup work later. They also make the next governance decision more explicit: keep the control in routing, add it in application code, or handle it through procurement and compliance review.

Portkey’s framing is mostly right

In its January 2026 guide, Portkey argues that “observability becomes the foundation” for governance, auditing, optimization, and confidence in production outcomes. That sequence is right. Observe the system before you claim you govern it.

Use a managed gateway when your first problem is fragmented LLM access and weak visibility. When your requirements expand into formal redaction workflows, custom authorization, or audit evidence beyond routing logs, treat those as separate design decisions.

Self-hosted versus managed governance

Self-hosted and managed gateways solve the same visibility problem with different ownership models.

When LiteLLM wins

Self-hosting LiteLLM is a common recommendation for teams that want full control, and that advice has a real basis. LiteLLM has 40K+ GitHub stars and a feature set built for platform teams that want direct control over their gateway.

Use self-hosted when control matters more than speed. It fits teams that need full data sovereignty, custom authentication, internal logging standards, model access controls, and close integration with existing platform systems. It also fits teams that already have the capacity to run another production service.

When a managed gateway wins

Use a managed gateway when you cannot answer who spent what, on which models, going to which providers.

This gives you immediate evidence for spend, provider selection, and usage patterns without turning gateway operations into a new platform project.

Operating cost of self-hosting

Self-hosting moves the governance cost into your team’s backlog instead of removing it. You own deployment, logging infrastructure, uptime, upgrades, and incident response, including the questions that decide whether the gateway becomes reliable infrastructure or another fragile service:

• Who patches it?
• Who handles provider failures?
• Who maintains dashboards and alerts?
• Who explains missing logs during an audit?

If you have that team, self-hosting can be the right decision. If you do not, managed routing gives you a cleaner first step. Either way, the architecture choice is already a governance choice.

Next steps

• Audit your current LLM API access pattern: count how many provider API keys exist across your services and confirm who has access to each.
• Map your team’s routing architecture to one of the three postures: managed gateway, self-hosted gateway, or direct API.
• Answer the four baseline governance questions with your current setup: who is spending what, on which models, going to which providers, and does any of it produce an audit log?
• If you cannot answer those questions, route one service through a managed gateway this sprint.
• Set a review trigger: when your team hits a compliance audit, procurement review, or Series B due diligence, revisit the self-hosted versus managed-platform decision.