OpenTelemetry & Observability¶

CloudSlash ships with full OpenTelemetry instrumentation across the engine pipeline, AI agent, and plugin lifecycle. Export traces to any OTel-compatible backend.

OTel Tracing¶

Setting Up

Pass the --otel-endpoint flag when starting the daemon:

cs daemon --otel-endpoint http://localhost:4318

CloudSlash sends traces via OTLP HTTP to the specified endpoint. The service name is cloudslash.

Jaeger Integration¶

Run Jaeger all-in-one for local tracing:

# docker-compose.yml
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "4318:4318"   # OTLP HTTP
      - "16686:16686" # Jaeger UI

docker compose up -d
cs daemon --otel-endpoint http://localhost:4318

Open http://localhost:16686 and search for service cloudslash.

Grafana Tempo Integration¶

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

cs daemon --otel-endpoint http://tempo:4318

What's Traced¶

Engine Pipeline¶

Every phase of the scan pipeline gets its own OTel span under the tracer cloudslash/engine:

Span	Description
`engine.run`	Root span for the full pipeline
`engine.init`	Pricing client + BadgerDB initialization
`engine.time_travel`	Temporal snapshot resolution
`engine.plugin_scan`	Plugin discovery + gRPC scan per provider
`engine.heuristics`	Heuristic engine execution
`engine.pattern`	Isomorphic pattern recognition
`engine.ghost_linker`	Cross-provider edge discovery
`engine.formal_verify`	Oracle reachability proof
`engine.sau`	s.a.u saga engine execution
`engine.policy`	CEL policy evaluation
`engine.signal_analysis`	Cost/waste aggregation

AI Agent¶

Every tool call gets its own span under tracer cloudslash-agent:

Span	Attributes
`agent.tool_call`	`tool.name`, `tool.args_bytes`, `agent.round`, `agent.active_tier`
`agent.tier_selected`	`tier` (`router` or `deep`)
`agent.escalation`	Records mid-turn router→deep escalations

Plugin gRPC Lifecycle¶

Span	Description
`plugin.handshake`	Version compatibility check
`plugin.scan`	Full scan stream duration
`plugin.quarantine`	ApplyQuarantine RPC duration
`plugin.remediate`	Remediate RPC duration

Prometheus Metrics¶

The daemon exposes Prometheus metrics at GET /metrics.

Key metrics:

Metric	Type	Description
`cloudslash_policy_violations_total`	Counter	Policy violations by `rule_id`, `resource_type`, `enforcement_mode`
`cloudslash_scan_duration_seconds`	Histogram	Full pipeline duration
`cloudslash_nodes_discovered_total`	Counter	Nodes per provider per scan
`cloudslash_waste_usd_total`	Gauge	Current projected waste in USD

Structured JSON Logging¶

For machine-parseable logs (log aggregators, ELK, Loki):

cs daemon --json

All log lines emit as JSON objects:

{"level":"info","ts":"2026-05-27T10:00:00Z","msg":"plugin scan complete","provider":"aws","nodes":2847,"duration_ms":4230}

Debug¶

Matrix Mode

cs daemon --verbose

Enables detailed debug logging with per-goroutine stack traces and engine state dumps. The TUI renders this as animated "Matrix Mode": falling character streams showing live engine activity.

Don't use: verbose in production

Verbose mode generates extremely high log volume. Use OTel traces for production observability instead.

Health Endpoints¶

Endpoint	Purpose
`GET /healthz`	Liveness probe: returns `200 OK` when daemon is alive
`GET /readyz`	Readiness probe: returns `200 OK` when graph is hydrated and engine is ready

Use these for Kubernetes liveness/readiness probes or load balancer health checks.