--- name: composal-telemetry description: "Install and configure OpenTelemetry in an application, instrument its important request and background-work paths, and run evidence-backed performance verification with Composal managed telemetry. Use when asked to add OTEL, observe a verification run, diagnose latency or resource use, compare an optimization, or design a safe shadow-traffic test." --- # Composal Telemetry Instrument the application so a verification run explains where time and work went, then use that evidence to propose and verify optimizations. Prefer the application's official OpenTelemetry SDK and maintained instrumentation packages; package names and feature coverage differ by language and change over time. ## Inspect before installing Identify the runtime, framework, dependency manager, request entrypoints, background workers, databases, caches, queues, and outbound clients. Check for an existing SDK, agent, collector, trace propagation, structured logging, or metrics stack before adding another provider. Use the current official OpenTelemetry page for that language and the project's lockfile; do not guess package names or add a second global SDK. Choose the smallest supported approach that covers the app: - Start with stable zero-code or framework instrumentation when the language and runtime support it. - Add maintained library instrumentation for actual dependencies in the repo. - Add manual spans and metrics only around application-owned work that automatic instrumentation cannot explain. - Preserve an existing compatible OTel setup and extend it instead of replacing it. Record any signal the language SDK does not support stably. Do not claim full logs, metrics, or traces coverage merely because one exporter starts. ## Configure managed OTLP Composal managed ingestion is organization-tenanted. Keep the token in a secret store or process environment; never commit it, print it, put it in a shell argument, or copy it into telemetry. ```sh export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.composal.ai export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20${COMPOSAL_TOKEN},X-Vex-Organization=${COMPOSAL_ORG_SLUG}" export OTEL_SERVICE_NAME= export OTEL_RESOURCE_ATTRIBUTES='service.version=,deployment.environment.name=' ``` Treat the block as configuration shape, not a command to paste literally: inject the real secret and organization through the deployment system, and confirm the selected SDK's environment-variable support and header escaping. Use the base endpoint above so conforming OTLP/HTTP exporters append `/v1/traces`, `/v1/metrics`, and `/v1/logs`. Composal's initial customer surface is OTLP/HTTP; do not switch to gRPC unless the service documentation says it is public. The token must belong to a current member of the selected organization. A missing or inaccessible organization must fail closed. Never set or trust `vex.organization_id`, `vex.organization_public_id`, or `vex.organization_slug`; the managed gateway overwrites those reserved resource attributes from authorization. ## Instrument useful boundaries Make one request or job reconstructable across services: - Trace inbound HTTP/RPC, outbound HTTP/RPC, database queries, cache calls, queue publish/consume, and important background jobs. Preserve W3C trace context at process and queue boundaries. - Add child spans for application-owned phases that can dominate latency, such as template rendering, serialization, model/tool calls, batch construction, file processing, and external-provider waits. - Correlate structured logs with trace and span ids. Capture errors with useful exception metadata, but do not record credentials, request bodies, prompts, customer content, or unbounded payloads by default. - Add request/job duration, throughput, error, retry, queue-delay, and resource saturation metrics. Add business metrics only when they explain the run. - Use bounded attributes such as route templates, operation names, dependency names, result classes, and deployment version. Never use user ids, raw URLs, SQL text, stack traces, or request ids as metric dimensions. Sampling must retain errors and the verification run's critical paths. Do not claim absence of a problem when sampling could have dropped the evidence. ## Verify the pipeline Before performance conclusions, prove all configured signals reach the managed endpoint from the real execution environment. Run one uniquely named canary operation and confirm its service name, version, environment, trace continuity, logs, metrics, and error behavior. A successful application request is not proof that export succeeded; inspect exporter diagnostics without exposing headers. If `otel.composal.ai` or the organization-scoped analysis surface is not deployed or reachable, finish the instrumentation and local exporter validation, state the blocked managed verification, and do not invent query results. ## Measure and optimize Define the workload and success metric before changing code. Capture a baseline, identify the dominant spans and correlated resource pressure, make one coherent optimization, and rerun the same workload on the same build topology. Report the distribution and sample size, not only an average. Separate observed facts from hypotheses and require fresh telemetry before claiming a speedup. Read [references/verification-runs.md](references/verification-runs.md) for baseline/comparison evidence, performance diagnosis, and safe shadow-traffic rules. ## Finish with evidence Leave the SDK initialization, dependency changes, runtime configuration template, and focused tests in the repository. Report: - languages, services, signals, and boundaries instrumented; - the organization and time window, never the token; - canary trace or run identifiers that contain no secret or customer data; - exporter limitations, sampling, and missing coverage; - baseline versus candidate results and whether the acceptance threshold passed. Do not disable existing monitoring, deploy credentials, replay production traffic, or change live sampling/retention without the user's authorization.