
Description
Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud / Prometheus / Loki / Tempo / Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s / Docker / EC2 discovery, relabeling, modules (`import.file/git/http`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` / `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent / OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging "Alloy isn't sending anything" — even when the user says "set up the agent", "write me a scrape config", "drop these logs before sending", or "OTel collector config" without naming Alloy.
SKILL.md
Grafana Alloy
OpenTelemetry-compatible collector — one binary for metrics + logs + traces + profiles.
Prerequisites
alloyCLI installed (brew install grafana/grafana/alloy,apt install alloy, orgrafana/alloyDocker image)- An endpoint to ship to (Grafana Cloud, Prometheus, Loki, Tempo, Pyroscope)
- API key + username for that endpoint, exported as
GRAFANA_API_KEYetc.
Common Workflows
1. Write + validate a config locally
// config.alloy — metrics → Grafana Cloud
prometheus.scrape "app" {
targets = [{"__address__" = "localhost:9090"}]
forward_to = [prometheus.remote_write.cloud.receiver]
scrape_interval = "30s"
}
prometheus.remote_write "cloud" {
endpoint {
url = sys.env("PROMETHEUS_URL")
basic_auth {
username = sys.env("PROM_USER")
password = sys.env("GRAFANA_API_KEY")
}
}
}
# 1. Format + syntax-check (catches typos before run)
alloy fmt config.alloy
alloy validate config.alloy
# 2. Run it
alloy run config.alloy
# or as a service: systemctl restart alloy
# 3. Verify all components are healthy via the UI on port 12345
curl -s http://localhost:12345/api/v0/web/components \
| jq '.[] | select(.health.state != "healthy") | {id, state:.health.state, msg:.health.message}'
# Expect: empty (nothing unhealthy). Otherwise the row shows the failing component + reason.
# 4. Verify samples are flowing
curl -s http://localhost:12345/metrics \
| grep -E '^prometheus_remote_storage_(samples_total|enqueue_retries_total)' | head
# samples_total should be > 0 and rising; retries should be 0.
2. Add log shipping (file → Loki)
loki.source.file "app_logs" {
targets = [{ __path__ = "/var/log/app/*.log", job = "app" }]
forward_to = [loki.write.cloud.receiver]
}
loki.write "cloud" {
endpoint {
url = sys.env("LOKI_URL")
basic_auth {
username = sys.env("LOKI_USER")
password = sys.env("GRAFANA_API_KEY")
}
}
}
# Verify in Grafana → Explore → Loki:
# {job="app"}
# Expect lines streaming. If empty:
# - check /var/log/app/*.log actually exists + readable by the alloy user
# - http://localhost:12345 → loki.source.file.app_logs → "Targets" tab
3. Receive OTLP traces and ship to Tempo
otelcol.receiver.otlp "default" {
grpc { endpoint = "0.0.0.0:4317" }
http { endpoint = "0.0.0.0:4318" }
output { traces = [otelcol.exporter.otlp.tempo.input] }
}
otelcol.exporter.otlp "tempo" {
client {
endpoint = "tempo-xxx.grafana.net/tempo:443"
auth = otelcol.auth.basic.grafana_cloud.handler
}
}
otelcol.auth.basic "grafana_cloud" {
username = sys.env("TEMPO_USER")
password = sys.env("GRAFANA_API_KEY")
}
# Verify in Grafana → Explore → Tempo → service.name = your-service
# At the Alloy level, watch the receiver:
curl -s http://localhost:12345/metrics | grep otelcol_receiver_accepted_spans
Full pattern set (Kubernetes discovery + relabel, complete Cloud pipeline for all 4 signals): references/collection-patterns.md.
Reference
references/config-syntax.md— block/attribute/expression grammar,import.*,remotecfg, clusteringreferences/components.md— full component catalog with purpose + typical argsreferences/collection-patterns.md— end-to-end pipelines (K8s pods, OTLP, profiles, logs)
Troubleshooting
alloy validate→ "component not found" → version too old;alloy --versionand upgrade- UI shows component
unhealthy→ click into it on http://localhost:12345 for the live error prometheus_remote_storage_samples_dropped_totalrising → checkenqueue_retries_totaland the remote_write endpoint URL + creds- No spans landing in Tempo → check
otelcol_receiver_refused_spansand the exporter'sotelcol_exporter_sent_spans/otelcol_exporter_send_failed_spans
Resources
More skills from the skills repository
View all 48 skillsadaptive-metrics
optimize Grafana Cloud metrics costs
Jul 12Cost OptimizationGrafanaMetricsObservabilityadmin
manage Grafana Cloud accounts and RBAC
Jul 12Access ControlAuthGrafanaOperationsadmission-control
implement admission control webhooks
Jul 12ArchitectureSecurityValidationalerting-irm
configure Grafana Alerting and Incident Management
Jul 12AlertingGrafanaIncident ResponseMonitoringapp-observability
monitor application performance in Grafana Cloud
Jul 12APMDistributed TracingGrafanaLLM +1app-sdk-concepts
scaffold and configure Grafana apps
Jul 12ArchitectureDeploymentEngineeringGrafana
More from Grafana
View publisherfaro-setup-web
instrument web apps with Grafana Faro
faro-web-sdk
Jul 12Distributed TracingFrontendMonitoringObservabilityconfiguring-yesoreyeram-infinity-datasource
configure Grafana Infinity data source
grafana-infinity-datasource
Jul 12API DevelopmentAuthenticationConfigurationGrafanaquerying-yesoreyeram-infinity-datasource
query data with Infinity datasource
grafana-infinity-datasource
Jul 15CSVData AnalysisGrafanaGraphQL +1agento11y
manage Grafana Agent Observability resources
gcx
Jul 18AgentsGrafanaMonitoringObservabilityagento11y-eval-starter
scaffold evaluation experiments for agents
gcx
Jul 21AgentsEvalsExperimentsGrafanaagento11y-instrument
instrument LLM apps for agent observability
gcx
Jul 21AgentsGrafanaInstrumentationLLM +1