AI Foundry & OpenAI Posture Check

Purpose

Assess the security, reliability, and cost efficiency of Azure OpenAI and AI Foundry deployments. Detects the most common anti-patterns that teams make when building AI-powered products.

Procedure

Step 0: Discover AI resources

az cognitiveservices account list \
  --query "[?kind=='OpenAI' || kind=='AIServices'].{name:name, kind:kind, rg:resourceGroup, location:location}" -o table

Category 1: Security

1. Managed Identity (12 pts): Check identity type and disableLocalAuth

   az cognitiveservices account show --name <account> --resource-group <rg> \
     --query "{identity:identity.type, disableLocalAuth:properties.disableLocalAuth}" -o json
   

2. Network isolation (13 pts): Check public access, firewall rules, private endpoints
3. Content filtering (10 pts): Verify RAI policies assigned to all deployments

   az cognitiveservices account deployment list --name <account> --resource-group <rg> \
     --query "[].{name:name, model:properties.model.name, raiPolicy:properties.raiPolicyName}" -o table
   

Category 2: Reliability & Operations

1. Model versions (7 pts): Check all deployments are on GA versions, not deprecated

   az cognitiveservices account deployment list --name <account> --resource-group <rg> \
     --query "[].{name:name, model:properties.model.name, version:properties.model.version}" -o table
   

2. Diagnostic settings (7 pts): Verify RequestResponse + Audit logs to Log Analytics
3. Resource locks (5 pts): Check CanNotDelete lock on production accounts
4. Multi-region (5 pts): Verify accounts in 2+ regions for resilience
5. 429 throttling (6 pts): Query Log Analytics for throttle rate in last 24h

   AzureDiagnostics
   | where TimeGenerated > ago(24h) and ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
   | summarize TotalRequests=count(), Throttled=countif(resultSignature_d == 429) by Resource
   

Category 3: Cost & Efficiency

1. Rate limits (5 pts): Check deployments have explicit TPM capacity set
2. Model diversity (3 pts): Verify mix of models (not premium for everything)
3. PTU utilization (5 pts): If PTU exists, check utilization > 60%
4. Token consumption trend (5 pts): Query token usage over 7 days for anomalies

Category 4: Architecture

1. AI Gateway (APIM) (7 pts): Check for API Management with OpenAI backend

   az apim list --query "[].{name:name, sku:sku.name}" -o table
   

2. Environment separation (5 pts): Verify dev/prod use separate OpenAI accounts

Scoring

Sample output

References

• Azure OpenAI Security: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity
• Content Filtering: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters
• Model Retirements: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements
• BCDR: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/business-continuity-disaster-recovery