Chapter 8: Monitoring, Observability & Alerting¶
Seeing Everything – The Foundation for AI Agents
Part of: The DevOps Engineer's Guide to Effective AI Usage — and Becoming a Software Orchestrator
🧭 Foundation link: This chapter applies the Workflow Structure from Chapter 1 to operations: observability as the feedback loop that tells you whether AI-assisted changes actually work. The Software Orchestrator mindset applies throughout — structure lives in the repository, and any tool can implement it.
Table of Contents¶
- Executive Summary – Why Monitoring Matters for AI
- Part 1: Monitoring vs. Observability – Understanding the Difference
- Part 2: Monitoring Architecture – What to Monitor and How
- Part 3: Alerting Strategy – When to Alert and Who to Notify
- Part 4: Dashboards & Visualization – Making Data Actionable
- Part 5: AI Agent Monitoring – Special Considerations for Chapter 10
- Part 6: Workflow Application for Monitoring Workflows
- Part 7: Iteration Points – Your Feedback Needed
- Appendix: Monitoring Templates & Configurations
1. Executive Summary – Why Monitoring Matters for AI ¶
The Hard Truth About Monitoring¶
┌─────────────────────────────────────────────────────────────┐
│ WHY MONITORING MATTERS FOR AI AGENTS │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Without Monitoring] │
│ • You can't see what's broken │
│ • AI Agents operate in the dark │
│ • Incidents detected by customers │
│ • No data for AI Agents to learn from │
│ • No audit trail for compliance │
│ │
│ [With Monitoring] │
│ • You see problems before customers do │
│ • AI Agents have data to make decisions │
│ • Incidents detected and resolved quickly │
│ • AI Agents learn from historical data │
│ • Full audit trail for compliance │
│ │
│ [Key Insight] │
│ Chapters 3-7 built the structure and guardrails │
│ Chapter 8 provides the visibility │
│ Chapter 10 AI Agents need this visibility to operate │
│ │
└─────────────────────────────────────────────────────────────┘
Why This Chapter Exists¶
Chapter 3 taught you: Structured IaC (InfraCtl)
Chapter 4 taught you: Structured Deployment (Ansible)
Chapter 5 taught you: Structured CI/CD (Pipelines + Runners)
Chapter 6 taught you: Production Deployment & Release Management
Chapter 7 taught you: Governance, Safety & Compliance
Chapter 8 teaches you: Monitoring, Observability & Alerting – the visibility that makes Chapters 3-7 (and eventually Chapter 10 AI Agents) observable and accountable
Chapter 10 will teach you: AI Agents that USE this monitoring data to make decisions
The Core Thesis¶
"You can't automate what you can't observe. This chapter provides the monitoring, observability, and alerting foundation that Chapters 3-7 operate within, and that Chapter 10 AI Agents need to make informed decisions."
What You'll Learn¶
| Section | What You'll Gain | Why It Matters |
|---|---|---|
| Part 1: Monitoring vs. Observability | Understand the difference | Choose the right tools |
| Part 2: Monitoring Architecture | What to monitor and how | Comprehensive visibility |
| Part 3: Alerting Strategy | When to alert and who | Avoid alert fatigue |
| Part 4: Dashboards | Make data actionable | Quick decision-making |
| Part 5: AI Agent Monitoring | Special considerations | Chapter 10 preparation |
| Part 6: Workflow Application | Apply the repository-centered workflow to monitoring | Daily productivity |
2. Part 1: Monitoring vs. Observability – Understanding the Difference ¶
2.1 The Key Distinction¶
┌─────────────────────────────────────────────────────────────┐
│ MONITORING vs. OBSERVABILITY │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Monitoring] │
│ • WHAT: Known unknowns │
│ • Question: "Is the system working?" │
│ • Approach: Pre-defined metrics and alerts │
│ • Example: CPU > 80% → alert │
│ • Best for: Known failure modes │
│ │
│ [Observability] │
│ • WHAT: Unknown unknowns │
│ • Question: "Why is the system broken?" │
│ • Approach: Logs, metrics, traces (three pillars) │
│ • Example: Query any metric, correlate across services │
│ • Best for: Complex, distributed systems │
│ │
│ [The Relationship] │
│ Monitoring is a subset of observability │
│ You need both for production readiness │
│ AI Agents need observability to make good decisions │
│ │
└─────────────────────────────────────────────────────────────┘
2.2 The Three Pillars of Observability¶
┌─────────────────────────────────────────────────────────────┐
│ THREE PILLARS OF OBSERVABILITY │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Pillar 1: Metrics] │
│ • WHAT: Numerical measurements over time │
│ • Examples: CPU usage, memory, request rate, error rate │
│ • Tools: Prometheus, Datadog, CloudWatch │
│ • AI Agent Use: Decision thresholds, anomaly detection │
│ │
│ [Pillar 2: Logs] │
│ • WHAT: Timestamped records of events │
│ • Examples: Application logs, access logs, audit logs │
│ • Tools: ELK Stack, Splunk, CloudWatch Logs │
│ • AI Agent Use: Root cause analysis, pattern detection │
│ │
│ [Pillar 3: Traces] │
│ • WHAT: Request flow across services │
│ • Examples: Distributed traces, span data │
│ • Tools: Jaeger, Zipkin, AWS X-Ray │
│ • AI Agent Use: Service dependency mapping, latency analysis│
│ │
│ [You Need All Three] │
│ Metrics: Tell you WHAT is happening │
│ Logs: Tell you WHY it's happening │
│ Traces: Tell you WHERE it's happening │
│ │
└─────────────────────────────────────────────────────────────┘
2.3 Monitoring Maturity Levels¶
┌─────────────────────────────────────────────────────────────┐
│ MONITORING MATURITY LEVELS │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Level 1: Reactive] │
│ • Monitor: Nothing until something breaks │
│ • Alert: Customers report issues │
│ • Response: Firefighting │
│ • AI Agent Role: Not ready for AI Agents │
│ │
│ [Level 2: Proactive] │
│ • Monitor: Key metrics (CPU, memory, disk) │
│ • Alert: Threshold-based alerts │
│ • Response: On-call responds to alerts │
│ • AI Agent Role: Basic monitoring, human decides │
│ │
│ [Level 3: Predictive] │
│ • Monitor: Business metrics + technical metrics │
│ • Alert: Anomaly detection, trend analysis │
│ • Response: Prevent issues before they happen │
│ • AI Agent Role: AI can recommend based on trends │
│ │
│ [Level 4: Autonomous] │
│ • Monitor: Full observability (metrics, logs, traces) │
│ • Alert: AI-driven alerting, smart correlation │
│ • Response: AI Agents auto-remediate low-risk issues │
│ • AI Agent Role: Chapter 10 ready │
│ │
│ [Recommendation] │
│ Aim for Level 3 before implementing AI Agents (Level 4) │
│ │
└─────────────────────────────────────────────────────────────┘
2.4 Monitoring Requirements by Environment¶
| Environment | Monitoring Level | Alerting | Retention | AI Agent Access |
|---|---|---|---|---|
| Development | Basic metrics | Email only | 30 days | Full access |
| Staging | Enhanced metrics + logs | Slack + email | 90 days | Full access |
| Production | Full observability (3 pillars) | PagerDuty + Slack + email | 7 years | Read-only, write with approval |
3. Part 2: Monitoring Architecture – What to Monitor and How ¶
3.1 Monitoring Layers¶
┌─────────────────────────────────────────────────────────────┐
│ MONITORING LAYERS │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Layer 1: Infrastructure] │
│ • CPU, memory, disk, network │
│ • VM/container health │
│ • Load balancer health │
│ • Database connections │
│ • Tools: Prometheus, CloudWatch, Datadog │
│ │
│ [Layer 2: Application] │
│ • Request rate, error rate, latency │
│ • Business metrics (signups, purchases) │
│ • Application logs │
│ • Distributed traces │
│ • Tools: New Relic, AppDynamics, custom metrics │
│ │
│ [Layer 3: Pipeline] │
│ • CI/CD pipeline status │
│ • Deployment frequency │
│ • Deployment success rate │
│ • Rollback frequency │
│ • Tools: GitHub Actions metrics, custom dashboards │
│ │
│ [Layer 4: AI Agent] (Chapter 10) │
│ • AI Agent decision rate │
│ • AI Agent confidence scores │
│ • AI Agent escalation rate │
│ • AI Agent accuracy │
│ • Tools: Custom AI Agent monitoring (Section 6) │
│ │
└─────────────────────────────────────────────────────────────┘
3.2 Key Metrics to Track¶
┌─────────────────────────────────────────────────────────────┐
│ KEY METRICS BY LAYER │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Infrastructure Metrics (RED Method)] │
│ • Rate: Requests per second │
│ • Errors: Error rate (%) │
│ • Duration: Latency (p50, p95, p99) │
│ • Saturation: CPU, memory, disk usage │
│ │
│ [Application Metrics (Four Golden Signals)] │
│ • Latency: Time to serve requests │
│ • Traffic: Demand on system │
│ • Errors: Rate of failed requests │
│ • Saturation: How "full" the service is │
│ │
│ [Pipeline Metrics (DORA Metrics)] │
│ • Deployment Frequency: How often you deploy │
│ • Lead Time for Changes: Commit to deploy │
│ • Change Failure Rate: % of deployments causing issues │
│ • Mean Time to Recovery: Time to fix incidents │
│ │
│ [AI Agent Metrics] (Chapter 10) │
│ • Decision Accuracy: % of correct decisions │
│ • Confidence Score: AI confidence in decisions │
│ • Escalation Rate: % escalated to humans │
│ • Auto-Remediation Success: % successful auto-fixes │
│ │
└─────────────────────────────────────────────────────────────┘
3.3 Monitoring Configuration Template¶
File: monitoring/config/prometheus-rules.yml
# Prometheus Monitoring Rules
groups:
- name: infrastructure
interval: 30s
rules:
- alert: HighCPUUsage
expr: avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) < 0.2
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage detected"
description: "CPU usage is above 80% for 5 minutes"
- alert: HighMemoryUsage
expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.9
for: 5m
labels:
severity: warning
annotations:
summary: "High memory usage detected"
description: "Memory usage is above 90% for 5 minutes"
- alert: HighDiskUsage
expr: (node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes > 0.85
for: 10m
labels:
severity: warning
annotations:
summary: "High disk usage detected"
description: "Disk usage is above 85% for 10 minutes"
- name: application
interval: 30s
rules:
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Error rate is above 5% for 5 minutes"
- alert: HighLatency
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "High latency detected"
description: "P99 latency is above 1 second for 5 minutes"
- name: pipeline
interval: 60s
rules:
- alert: PipelineFailure
expr: ci_pipeline_status{status="failed"} == 1
for: 0m
labels:
severity: warning
annotations:
summary: "CI/CD pipeline failed"
description: "Pipeline {{ $labels.pipeline }} failed"
- alert: HighRollbackRate
expr: sum(rate(deployment_rollback_total[1h])) / sum(rate(deployment_total[1h])) > 0.1
for: 1h
labels:
severity: warning
annotations:
summary: "High rollback rate detected"
description: "Rollback rate is above 10% in the last hour"
3.4 Log Aggregation Configuration¶
File: monitoring/config/fluentd-config.yml
# Fluentd Log Aggregation Configuration
<system>
log_level info
</system>
<source>
@type tail
path /var/log/application/*.log
pos_file /var/log/fluentd/application.log.pos
tag application.*
<parse>
@type json
</parse>
</source>
<source>
@type tail
path /var/log/audit/*.log
pos_file /var/log/fluentd/audit.log.pos
tag audit.*
<parse>
@type json
</parse>
</source>
<match application.**>
@type elasticsearch
host elasticsearch.monitoring.svc
port 9200
index_name application-logs
<buffer>
@type file
path /var/log/fluentd/buffer/application
flush_interval 5s
</buffer>
</match>
<match audit.**>
@type elasticsearch
host elasticsearch.monitoring.svc
port 9200
index_name audit-logs
<buffer>
@type file
path /var/log/fluentd/buffer/audit
flush_interval 5s
</buffer>
</match>
3.5 Distributed Tracing Configuration¶
File: monitoring/config/jaeger-config.yml
# Jaeger Distributed Tracing Configuration
service_name: my-application
sampler:
type: probabilistic
param: 0.1 # Sample 10% of traces
reporter:
log_spans: true
local_agent:
reporting_host: jaeger.monitoring.svc
reporting_port: 6831
tags:
environment: production
version: ${APP_VERSION}
service: ${SERVICE_NAME}
4. Part 3: Alerting Strategy – When to Alert and Who to Notify ¶
4.1 Alert Severity Levels¶
┌─────────────────────────────────────────────────────────────┐
│ ALERT SEVERITY LEVELS │
├─────────────────────────────────────────────────────────────┤
│ │
│ [P1: Critical] │
│ • Impact: Production down, customers affected │
│ • Response Time: <15 minutes │
│ • Notification: PagerDuty + Slack + Phone │
│ • On-Call: Primary + Secondary │
│ • Examples: Complete outage, security breach, data loss │
│ │
│ [P2: High] │
│ • Impact: Major functionality impaired │
│ • Response Time: <30 minutes │
│ • Notification: PagerDuty + Slack │
│ • On-Call: Primary │
│ • Examples: Partial outage, performance degradation │
│ │
│ [P3: Medium] │
│ • Impact: Minor functionality impaired │
│ • Response Time: <2 hours │
│ • Notification: Slack │
│ • On-Call: Primary (during business hours) │
│ • Examples: Non-critical bug, UI issues │
│ │
│ [P4: Low] │
│ • Impact: Minimal, workaround available │
│ • Response Time: <24 hours │
│ • Notification: Email │
│ • On-Call: No on-call, ticket created │
│ • Examples: Cosmetic issues, documentation gaps │
│ │
└─────────────────────────────────────────────────────────────┘
4.2 Alert Routing Configuration¶
File: monitoring/config/alertmanager-routes.yml
# Alertmanager Routing Configuration
route:
receiver: default
group_by: ['alertname', 'severity', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: pagerduty-critical
continue: true
- match:
severity: critical
receiver: slack-critical
- match:
severity: warning
receiver: slack-warning
- match:
severity: info
receiver: email-info
- match:
team: security
receiver: slack-security
- match:
team: infrastructure
receiver: slack-infra
receivers:
- name: default
email_configs:
- to: team@example.com
- name: pagerduty-critical
pagerduty_configs:
- service_key: ${PAGERDUTY_SERVICE_KEY}
severity: critical
- name: slack-critical
slack_configs:
- api_url: ${SLACK_WEBHOOK_CRITICAL}
channel: '#incidents-critical'
title: '🚨 CRITICAL ALERT'
text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
- name: slack-warning
slack_configs:
- api_url: ${SLACK_WEBHOOK_WARNING}
channel: '#incidents-warning'
title: '⚠️ WARNING ALERT'
- name: email-info
email_configs:
- to: team@example.com
send_resolved: true
- name: slack-security
slack_configs:
- api_url: ${SLACK_WEBHOOK_SECURITY}
channel: '#security-alerts'
title: '🔒 SECURITY ALERT'
- name: slack-infra
slack_configs:
- api_url: ${SLACK_WEBHOOK_INFRA}
channel: '#infrastructure-alerts'
title: '🖥️ INFRASTRUCTURE ALERT'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'service']
4.3 Alert Fatigue Prevention¶
┌─────────────────────────────────────────────────────────────┐
│ ALERT FATIGUE PREVENTION │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Problem] │
│ • Too many alerts │
│ • Team ignores alerts │
│ • Real incidents missed │
│ • On-call burnout │
│ │
│ [Solutions] │
│ • Alert on symptoms, not causes │
│ • Use multi-condition alerts │
│ • Implement alert deduplication │
│ • Regular alert review (monthly) │
│ • Auto-resolve stale alerts │
│ • Require runbook for every alert │
│ │
│ [Alert Quality Checklist] │
│ □ Is this alert actionable? │
│ □ Does it have a runbook? │
│ □ Is the threshold appropriate? │
│ □ Is the severity correct? │
│ □ Is the right team notified? │
│ □ Has this alert fired in the last 30 days? │
│ □ If no fires in 30 days, should it be removed? │
│ │
│ [Monthly Alert Review] │
│ • Review all alerts that fired │
│ • Remove alerts that never fire │
│ • Adjust thresholds based on data │
│ • Update runbooks │
│ • Document lessons learned │
│ │
└─────────────────────────────────────────────────────────────┘
4.4 Alert Runbook Template¶
# Alert Runbook Template
## Alert Name: [Alert Name]
## Severity: [P1/P2/P3/P4]
## Description:
[What this alert means]
## Trigger Conditions:
[When this alert fires]
## Impact:
[What is affected when this alert fires]
## Immediate Actions:
1. [Step 1]
2. [Step 2]
3. [Step 3]
## Investigation:
1. [Check metric X]
2. [Check log Y]
3. [Check trace Z]
## Resolution:
1. [Fix step 1]
2. [Fix step 2]
3. [Verify fix]
## Rollback:
[If fix makes things worse, how to rollback]
## Escalation:
- If not resolved in 30 minutes: Escalate to [role]
- If not resolved in 1 hour: Escalate to [role]
## Related Alerts:
- [Related alert 1]
- [Related alert 2]
## Related Runbooks:
- [Related runbook 1]
- [Related runbook 2]
## Last Updated: [DATE]
## Owner: [NAME/ROLE]
5. Part 4: Dashboards & Visualization – Making Data Actionable ¶
5.1 Dashboard Types¶
┌─────────────────────────────────────────────────────────────┐
│ DASHBOARD TYPES │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Executive Dashboard] │
│ • Audience: Leadership, non-technical │
│ • Metrics: Business KPIs, uptime, incidents │
│ • Refresh: Hourly │
│ • Example: System health, customer impact │
│ │
│ [Operations Dashboard] │
│ • Audience: On-call, operations team │
│ • Metrics: All technical metrics, alerts │
│ • Refresh: Real-time │
│ • Example: Service health, active incidents │
│ │
│ [Development Dashboard] │
│ • Audience: Developers │
│ • Metrics: Deployment metrics, test results │
│ • Refresh: Real-time │
│ • Example: Pipeline status, code coverage │
│ │
│ [AI Agent Dashboard] (Chapter 10) │
│ • Audience: Engineering, AI team │
│ • Metrics: AI Agent decisions, accuracy, escalations │
│ • Refresh: Real-time │
│ • Example: AI Agent performance, human overrides │
│ │
└─────────────────────────────────────────────────────────────┘
5.2 Dashboard Best Practices¶
┌─────────────────────────────────────────────────────────────┐
│ DASHBOARD BEST PRACTICES │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Design Principles] │
│ • Start with questions, not metrics │
│ • One dashboard, one purpose │
│ • Use appropriate visualizations │
│ • Include context (baselines, thresholds) │
│ • Make it actionable │
│ │
│ [What to Include] │
│ • Current status (green/yellow/red) │
│ • Trends over time │
│ • Key metrics (limited to 5-10) │
│ • Links to related dashboards │
│ • Links to runbooks │
│ │
│ [What to Avoid] │
│ • Too many metrics (dashboard overload) │
│ • Metrics without context │
│ • Static dashboards (no time range selection) │
│ • Dashboards without owners │
│ • Dashboards that no one looks at │
│ │
│ [Maintenance] │
│ • Review dashboards quarterly │
│ • Remove unused dashboards │
│ • Update as services change │
│ • Document dashboard purpose │
│ │
└─────────────────────────────────────────────────────────────┘
5.3 Grafana Dashboard Template¶
File: monitoring/dashboards/production-overview.json
{
"dashboard": {
"title": "Production Overview",
"tags": ["production", "overview"],
"timezone": "browser",
"panels": [
{
"title": "System Health",
"type": "stat",
"targets": [
{
"expr": "up{environment=\"production\"}",
"legendFormat": "{{service}}"
}
],
"thresholds": [
{"value": 0, "color": "red"},
{"value": 1, "color": "green"}
]
},
{
"title": "Error Rate",
"type": "graph",
"targets": [
{
"expr": "sum(rate(http_requests_total{status=~\"5..\",environment=\"production\"}[5m])) / sum(rate(http_requests_total{environment=\"production\"}[5m])) * 100",
"legendFormat": "Error Rate %"
}
],
"thresholds": [
{"value": 1, "color": "yellow"},
{"value": 5, "color": "red"}
]
},
{
"title": "Latency (P99)",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{environment=\"production\"}[5m])) by (le))",
"legendFormat": "P99 Latency"
}
],
"thresholds": [
{"value": 0.5, "color": "yellow"},
{"value": 1, "color": "red"}
]
},
{
"title": "Deployment Status",
"type": "table",
"targets": [
{
"expr": "deployment_info{environment=\"production\"}",
"format": "table"
}
]
},
{
"title": "Active Incidents",
"type": "alertlist",
"alerts": {
"state": ["alerting"],
"tags": ["production"]
}
}
],
"refresh": "30s",
"time": {
"from": "now-1h",
"to": "now"
}
}
}
6. Part 5: AI Agent Monitoring – Special Considerations for Chapter 10 ¶
6.1 AI Agent Metrics to Track¶
┌─────────────────────────────────────────────────────────────┐
│ AI AGENT METRICS (Chapter 10 Preview) │
├─────────────────────────────────────────────────────────────┤
│ │
│ [Decision Metrics] │
│ • Total decisions made │
│ • Decisions by type (deploy/rollback/escalate) │
│ • Decision confidence scores │
│ • Decision accuracy (vs. human decisions) │
│ │
│ [Performance Metrics] │
│ • Decision latency (time to decide) │
│ • Action execution time │
│ • API call success rate │
│ • Rate limit hits │
│ │
│ [Safety Metrics] │
│ • Escalation rate (to humans) │
│ • Human override rate │
│ • Boundary violations │
│ • Emergency stop activations │
│ │
│ [Learning Metrics] │
│ • Model accuracy over time │
│ • False positive rate │
│ • False negative rate │
│ • Learning implementation rate │
│ │
└─────────────────────────────────────────────────────────────┘
6.2 AI Agent Monitoring Configuration¶
File: monitoring/config/ai-agent-rules.yml
# AI Agent Monitoring Rules (Chapter 10)
groups:
- name: ai-agent
interval: 30s
rules:
- alert: AI Agent Low Confidence
expr: avg(ai_agent_confidence_score[5m]) < 0.7
for: 5m
labels:
severity: warning
annotations:
summary: "AI Agent confidence is low"
description: "AI Agent average confidence is below 70%"
- alert: AI Agent High Escalation Rate
expr: sum(rate(ai_agent_escalations_total[1h])) / sum(rate(ai_agent_decisions_total[1h])) > 0.3
for: 1h
labels:
severity: warning
annotations:
summary: "AI Agent escalation rate is high"
description: "AI Agent is escalating more than 30% of decisions"
- alert: AI Agent Boundary Violation
expr: increase(ai_agent_boundary_violations_total[1h]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "AI Agent boundary violation detected"
description: "AI Agent attempted to violate boundaries"
- alert: AI Agent Decision Accuracy Drop
expr: avg(ai_agent_decision_accuracy[24h]) < 0.85
for: 24h
labels:
severity: warning
annotations:
summary: "AI Agent decision accuracy has dropped"
description: "AI Agent accuracy is below 85% over 24 hours"
6.3 AI Agent Dashboard Template¶
File: monitoring/dashboards/ai-agent-overview.json
{
"dashboard": {
"title": "AI Agent Overview",
"tags": ["ai-agent", "automation"],
"panels": [
{
"title": "AI Agent Decisions",
"type": "stat",
"targets": [
{
"expr": "sum(ai_agent_decisions_total)",
"legendFormat": "Total Decisions"
}
]
},
{
"title": "Decision Confidence",
"type": "gauge",
"targets": [
{
"expr": "avg(ai_agent_confidence_score)",
"legendFormat": "Avg Confidence"
}
],
"thresholds": [
{"value": 0.5, "color": "red"},
{"value": 0.7, "color": "yellow"},
{"value": 0.85, "color": "green"}
]
},
{
"title": "Escalation Rate",
"type": "graph",
"targets": [
{
"expr": "sum(rate(ai_agent_escalations_total[1h])) / sum(rate(ai_agent_decisions_total[1h])) * 100",
"legendFormat": "Escalation Rate %"
}
],
"thresholds": [
{"value": 20, "color": "yellow"},
{"value": 30, "color": "red"}
]
},
{
"title": "Decision Accuracy",
"type": "graph",
"targets": [
{
"expr": "avg(ai_agent_decision_accuracy)",
"legendFormat": "Accuracy %"
}
],
"thresholds": [
{"value": 85, "color": "yellow"},
{"value": 95, "color": "green"}
]
},
{
"title": "Boundary Violations",
"type": "stat",
"targets": [
{
"expr": "sum(ai_agent_boundary_violations_total)",
"legendFormat": "Violations"
}
],
"thresholds": [
{"value": 0, "color": "green"},
{"value": 1, "color": "red"}
]
},
{
"title": "Recent Decisions",
"type": "table",
"targets": [
{
"expr": "ai_agent_decisions_total",
"format": "table"
}
]
}
],
"refresh": "30s"
}
}
6.4 AI Agent Audit Trail¶
# AI Agent Audit Trail Requirements
## What to Log:
- All AI Agent decisions (with rationale)
- All AI Agent actions (deploy, rollback, escalate, block)
- All human approvals/rejections of AI recommendations
- All AI Agent boundary violations
- All AI Agent emergency stop activations
- All AI Agent rule changes
## Log Format:
```json
{
"timestamp": "2024-01-15T10:30:00Z",
"agent_id": "deployment-agent-01",
"decision": "deploy",
"version": "v1.0.1",
"environment": "staging",
"risk_level": "low",
"confidence_score": 0.92,
"approval_required": false,
"approver": null,
"outcome": "success",
"duration": "45s",
"rationale": "PATCH version, tests passed, security scan passed"
}
```
## Retention:
- AI Agent decisions: 7 years
- AI Agent boundary violations: 7 years
- AI Agent rule changes: 7 years
- AI Agent learning updates: 2 years
## Access:
- Engineers: Read own AI Agent decisions
- Team leads: Read team AI Agent decisions
- Security: Read all AI Agent logs
- Compliance: Read all AI Agent logs
- Auditors: Read all AI Agent logs (time-limited)
7. Part 6: Workflow Application for Monitoring Workflows ¶
This section applies the Chapter 2 model to monitoring work: keep dashboards, alert rules, and runbook references grounded in the repository, use CLI validation and scripted changes where possible, and use the IDE to review and explain what changed.
7.1 Optional IDE Example for Monitoring¶
If you use an editor assistant, keep it behind the repository and CLI workflow.
{
"models": [
{
"title": "🔵 Qwen-2.5-Coder (Monitoring Code)",
"provider": "openai",
"model": "qwen-2.5-coder",
"apiKey": "${QWEN_API_KEY}",
"apiBase": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"default": true
},
{
"title": "🟢 DeepSeek-V3 (Monitoring Logic)",
"provider": "openai",
"model": "deepseek-chat",
"apiKey": "${DEEPSEEK_API_KEY}",
"apiBase": "https://api.deepseek.com/v1"
},
{
"title": "🟠 Claude-3.5-Sonnet (Alert Review)",
"provider": "anthropic",
"model": "claude-3-5-sonnet-20241022",
"apiKey": "${ANTHROPIC_API_KEY}"
}
],
"customCommands": [
{
"name": "monitoring-metric",
"prompt": "Generate monitoring metric configuration for {{{ input }}}. CRITICAL: 1) Follow monitoring architecture from Chapter 8, 2) Include appropriate thresholds, 3) Include alert routing, 4) Include runbook reference. Follow Chapter 8 templates.",
"description": "Generate monitoring metric configuration"
},
{
"name": "alert-rule",
"prompt": "Generate alert rule for {{{ input }}}. Include: 1) Alert expression, 2) Severity level, 3) Notification channels, 4) Runbook reference. Follow Chapter 8 alerting strategy.",
"description": "Generate alert rule"
},
{
"name": "alert-runbook",
"prompt": "Generate alert runbook for {{{ input }}}. Include: 1) Alert description, 2) Trigger conditions, 3) Immediate actions, 4) Investigation steps, 5) Resolution steps, 6) Escalation procedure. Follow Chapter 8 runbook template.",
"description": "Generate alert runbook"
},
{
"name": "dashboard-panel",
"prompt": "Generate Grafana dashboard panel for {{{ input }}}. Include: 1) Panel type, 2) Query expression, 3) Thresholds, 4) Visualization options. Follow Chapter 8 dashboard best practices.",
"description": "Generate Grafana dashboard panel"
},
{
"name": "ai-agent-metric",
"prompt": "Generate AI Agent monitoring metric for {{{ input }}}. Include: 1) Metric definition, 2) Alert thresholds, 3) Dashboard panel, 4) Audit trail requirements. Follow Chapter 8 AI Agent monitoring (Chapter 10 preparation).",
"description": "Generate AI Agent monitoring metric"
}
]
}
7.2 VSCode Snippets for Monitoring¶
File: ~/.vscode/snippets/monitoring.json
{
"Prometheus Alert Rule": {
"prefix": "prom-alert",
"body": [
"- alert: ${1:AlertName}",
" expr: ${2:expression}",
" for: ${3:5m}",
" labels:",
" severity: ${4:warning}",
" annotations:",
" summary: \"${5:Alert summary}\"",
" description: \"${6:Alert description}\"",
" runbook: \"${7:URL to runbook}\""
],
"description": "Prometheus alert rule template"
},
"Alert Runbook": {
"prefix": "alert-runbook",
"body": [
"# Alert Runbook: ${1:Alert Name}",
"",
"## Severity: ${2:P1/P2/P3/P4}",
"",
"## Description:",
"${3:What this alert means}",
"",
"## Trigger Conditions:",
"${4:When this alert fires}",
"",
"## Immediate Actions:",
"1. ${5:Step 1}",
"2. ${6:Step 2}",
"3. ${7:Step 3}",
"",
"## Investigation:",
"1. ${8:Check metric X}",
"2. ${9:Check log Y}",
"3. ${10:Check trace Z}",
"",
"## Resolution:",
"1. ${11:Fix step 1}",
"2. ${12:Fix step 2}",
"3. ${13:Verify fix}",
"",
"## Escalation:",
"- If not resolved in 30 minutes: Escalate to ${14:role}",
"- If not resolved in 1 hour: Escalate to ${15:role}",
"",
"## Last Updated: ${16:DATE}",
"## Owner: ${17:NAME/ROLE}"
],
"description": "Alert runbook template"
},
"Grafana Panel": {
"prefix": "grafana-panel",
"body": [
"{",
" \"title\": \"${1:Panel Title}\",",
" \"type\": \"${2:graph}\",",
" \"targets\": [",
" {",
" \"expr\": \"${3:prometheus_expression}\",",
" \"legendFormat\": \"${4:Legend}\"",
" }",
" ],",
" \"thresholds\": [",
" {\"value\": ${5:0}, \"color\": \"${6:red}\"},",
" {\"value\": ${7:1}, \"color\": \"${8:green}\"}",
" ]",
"}"
],
"description": "Grafana panel template"
},
"AI Agent Metric": {
"prefix": "ai-agent-metric",
"body": [
"# AI Agent Metric: ${1:Metric Name}",
"",
"## Definition:",
"${2:What this metric measures}",
"",
"## Expression:",
"```promql",
"${3:prometheus_expression}",
"```",
"",
"## Thresholds:",
"- Warning: ${4:threshold}",
"- Critical: ${5:threshold}",
"",
"## Alert:",
"- Name: ${6:alert_name}",
"- Severity: ${7:P1/P2/P3/P4}",
"- Notification: ${8:channels}",
"",
"## Dashboard:",
"- Panel Type: ${9:type}",
"- Refresh: ${10:30s}",
"",
"## Audit Trail:",
"- Log: ${11:YES/NO}",
"- Retention: ${12:7 years}"
],
"description": "AI Agent monitoring metric template"
}
}
8. Part 7: Iteration Points – Your Feedback Needed ¶
8.1 This Chapter's Core Message¶
"You can't automate what you can't observe. This chapter provides the monitoring, observability, and alerting foundation that Chapters 3-7 operate within, and that Chapter 10 AI Agents need to make informed decisions."
8.2 Questions for Your Feedback¶
□ Question 1: Does the monitoring vs. observability distinction come through clearly?
- Is this the right framing for your experience?
- What would make it clearer?
□ Question 2: Are the monitoring layers comprehensive?
- Do you monitor infrastructure, application, and pipeline?
- What's missing?
□ Question 3: Is the alerting strategy practical?
- Do you have alert severity levels?
- What would you change?
□ Question 4: Are the dashboard best practices useful?
- Do your dashboards follow these principles?
- What would you add?
□ Question 5: Is the AI Agent monitoring section helpful?
- Does this prepare you for Chapter 10?
- What metrics are missing?
□ Question 6: Is the workflow application practical?
- Do the custom commands make sense?
- What workflows would save you time?
□ Question 7: What's missing?
- What topics should be added?
- What should be removed or condensed?
9. Appendix: Monitoring Templates & Configurations ¶
9.1 Monitoring Checklist¶
# Monitoring Implementation Checklist
## Infrastructure Monitoring:
□ CPU, memory, disk metrics collected
□ Network metrics collected
□ Load balancer health monitored
□ Database connections monitored
□ Alerts configured for critical thresholds
## Application Monitoring:
□ Request rate monitored
□ Error rate monitored
□ Latency (p50, p95, p99) monitored
□ Business metrics tracked
□ Distributed tracing enabled
## Pipeline Monitoring:
□ CI/CD pipeline status monitored
□ Deployment frequency tracked
□ Deployment success rate tracked
□ Rollback frequency tracked
□ DORA metrics calculated
## Alerting:
□ Alert severity levels defined
□ Alert routing configured
□ Alert runbooks created
□ Alert fatigue prevention implemented
□ Monthly alert review scheduled
## Dashboards:
□ Executive dashboard created
□ Operations dashboard created
□ Development dashboard created
□ AI Agent dashboard prepared (Chapter 10)
□ Dashboard review scheduled quarterly
## AI Agent Monitoring (Chapter 10):
□ AI Agent decision metrics defined
□ AI Agent confidence tracking enabled
□ AI Agent escalation rate monitored
□ AI Agent boundary violations logged
□ AI Agent audit trail configured
## Sign-Off:
□ Engineering Lead: ________________ Date: ________
□ Operations Lead: ________________ Date: ________
□ Security Lead: ________________ Date: ________
9.2 The Chapter 8 Checklist¶
# Chapter 8: Monitoring, Observability & Alerting - Checklist
## Monitoring Architecture:
□ Infrastructure monitoring enabled (Section 3.1)
□ Application monitoring enabled (Section 3.1)
□ Pipeline monitoring enabled (Section 3.1)
□ AI Agent monitoring prepared (Section 6)
## Alerting:
□ Alert severity levels defined (Section 4.1)
□ Alert routing configured (Section 4.2)
□ Alert runbooks created (Section 4.4)
□ Alert fatigue prevention implemented (Section 4.3)
## Dashboards:
□ Executive dashboard created (Section 5.1)
□ Operations dashboard created (Section 5.1)
□ Development dashboard created (Section 5.1)
□ AI Agent dashboard prepared (Section 6.3)
## AI Agent Preparation (Chapter 10):
□ AI Agent metrics defined (Section 6.1)
□ AI Agent monitoring configured (Section 6.2)
□ AI Agent audit trail prepared (Section 6.4)
## Key Principle:
"You can't automate what you can't observe. Monitoring is the foundation for AI Agents."
Chapter Summary¶
The Core Message¶
┌─────────────────────────────────────────────────────────────┐
│ CHAPTER 8 IN ONE SENTENCE │
├─────────────────────────────────────────────────────────────┤
│ │
│ "You can't automate what you can't observe. This chapter │
│ provides the monitoring, observability, and alerting │
│ foundation that Chapters 3-7 operate within, and that │
│ Chapter 10 AI Agents need to make informed decisions." │
│ │
└─────────────────────────────────────────────────────────────┘
Key Takeaways¶
✅ Monitoring vs. observability – Understand the difference
✅ Monitoring architecture: Infrastructure, application, pipeline, AI Agent
✅ Alerting strategy: Severity levels, routing, runbooks
✅ Dashboards: Executive, operations, development, AI Agent
✅ AI Agent monitoring: Special considerations for Chapter 10
✅ Workflow application: monitoring templates, CLI checks, and optional IDE support
✅ Chapter 10: AI Agents need this monitoring data to decide
Connection to Other Chapters¶
| Chapter | Connection |
|---|---|
| Chapter 3 | InfraCtl structure → Monitoring validates structure |
| Chapter 4 | Ansible structure → Monitoring validates deployment |
| Chapter 5 | CI/CD structure → Monitoring validates pipelines |
| Chapter 6 | Production deployment → Monitoring validates production |
| Chapter 7 | Governance → Monitoring enforces governance |
| Chapter 8 | Monitoring, Observability & Alerting |
| Chapter 9 | Continuous Improvement → Monitoring provides data |
| Chapter 10 | AI Agents → USE this monitoring data to decide |
Book Progress¶
✅ Chapter 1: AI Foundations (Symbolic + Data-Driven)
✅ Chapter 2: The Self-Improving Repository
✅ Chapter 3: Structured IaC (InfraCtl)
✅ Chapter 4: Structured Deployment (Ansible)
✅ Chapter 5: Structured CI/CD (Pipelines + Runners)
✅ Chapter 6: Production Deployment & Release Management
✅ Chapter 7: Governance, Safety & Compliance
✅ Chapter 8: Monitoring, Observability & Alerting
Next:
□ Chapter 9: Continuous Improvement & Learning
□ Chapter 10: AI Agents (Culmination)
□ Index: Quick Reference & Publishing