datadog_observability

Searching Logs

Use search_datadog_logs for individual log retrieval and analyze_datadog_logs for SQL-based aggregation queries.

Common log search patterns:

# Errors from a specific service
service:payment-api status:error

# Logs from a host in the last hour
host:web-prod-01

# Logs containing a specific trace ID
trace_id:abc123def456

# Errors with a specific HTTP status
@http.status_code:500 service:api-gateway

# Logs from a Kubernetes pod
kube_namespace:production kube_deployment:checkout-service

SQL-based log analysis with analyze_datadog_logs:

-- Count errors by service in the last hour
SELECT service, count(*) as error_count
FROM logs
WHERE status = 'error'
GROUP BY service
ORDER BY error_count DESC

-- Average response time by endpoint
SELECT @http.url_details.path, avg(@duration) as avg_duration
FROM logs
WHERE service = 'api-gateway'
GROUP BY @http.url_details.path

Querying Metrics

Use search_datadog_metrics to discover metrics, get_datadog_metric_context for metadata, and get_datadog_metric for time series data.

Common metric patterns:

# System metrics
system.cpu.user, system.mem.used, system.disk.used

# Container metrics
docker.cpu.usage, kubernetes.cpu.requests

# Application metrics
trace.servlet.request.hits, trace.servlet.request.duration

# Custom metrics
app.payment.processed, app.queue.depth

Always specify a time range when querying metrics to avoid retrieving excessive data.

Investigating Traces

Use get_datadog_trace for complete trace details and search_datadog_spans for span-level queries.

Trace investigation workflow:

1. Search for slow or errored spans with search_datadog_spans
2. Get the full trace with get_datadog_trace using the trace ID
3. Identify the bottleneck service or operation
4. Correlate with search_datadog_logs using the trace ID
5. Check related metrics with get_datadog_metric

Working with Monitors

Use search_datadog_monitors to find monitors by name, tag, or status.

Common monitor queries:

# Find all triggered monitors
Search for monitors with status "Alert"

# Find monitors for a specific service
Search for monitors tagged with service:payment-api

# Find monitors by name
Search for monitors matching "CPU" or "memory"

Incident Investigation Workflow

For structured incident investigation:

1. search_datadog_incidents — find recent or active incidents
2. get_datadog_incident — get full incident details and timeline
3. search_datadog_monitors — check which monitors triggered
4. search_datadog_logs — search for errors around the incident time
5. get_datadog_metric — check key metrics for anomalies
6. get_datadog_trace — inspect request traces for latency or errors
7. search_datadog_hosts — verify infrastructure health
8. search_datadog_service_dependencies — map affected services

Working with Dashboards and Notebooks

• Use search_datadog_dashboards to find dashboards by title or tag
• Use search_datadog_notebooks and get_datadog_notebook for investigation notebooks that document past analyses

Troubleshooting