# Observability

## Getting Started
Copilot can configure trace collection for services by setting the following in a manifest:
```yaml
observability:
  tracing: awsxray
```

For [Request-Driven Web Services](../concepts/services.en.md#request-driven-web-service), Copilot will enable App Runner's baked-in [tracing configuration](https://docs.aws.amazon.com/apprunner/latest/dg/monitor-xray.html).

For [Load-Balanced Web Services](../concepts/services.en.md#load-balanced-web-service), [Backend Services](../concepts/services.en.md#backend-service), and [Worker Services](../concepts/services.en.md#worker-service), Copilot will deploy the [AWS OpenTelemetry Collector](https://github.com/aws-observability/aws-otel-collector) as a [sidecar](./sidecars.en.md).

## Instrumenting Your Service
Instrumenting your service to send telemetry data is done through [language specific SDKs](https://opentelemetry.io/docs/instrumentation/). 
Examples are provided in OpenTelemetry's documentation for each supported language.
You can also view documentation and examples provided by [AWS Distro for OpenTelemetry](https://aws-otel.github.io/docs/introduction).

### Example Application

This is a small [Express.js](https://expressjs.com/) service with instrumentation configured for all of its endpoints. To get started, install the required depedencies:

```
npm install express \
	@opentelemetry/api \
	@opentelemetry/sdk-trace-node \
	@opentelemetry/auto-instrumentations-node \
	@opentelemetry/exporter-trace-otlp-grpc \
	@opentelemetry/id-generator-aws-xray \
	@opentelemetry/propagator-aws-xray
```

Then, save the following to `tracer.js`:

```js title="tracer.js" linenums="1"
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { Resource } = require("@opentelemetry/resources");
const { trace } = require("@opentelemetry/api");
const { AWSXRayIdGenerator } = require("@opentelemetry/id-generator-aws-xray");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { AWSXRayPropagator } = require("@opentelemetry/propagator-aws-xray");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

module.exports = (serviceName) => {
  const tracerConfig = {
    idGenerator: new AWSXRayIdGenerator(),
    instrumentations: [getNodeAutoInstrumentations()],
    resource: Resource.default().merge(
      new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
      })
    ),
  };

  const tracerProvider = new NodeTracerProvider(tracerConfig);
  const otlpExporter = new OTLPTraceExporter();

  tracerProvider.addSpanProcessor(new BatchSpanProcessor(otlpExporter));
  tracerProvider.register({
    propagator: new AWSXRayPropagator(),
  });

  return trace.getTracer("example-instrumentation");
};
```

`tracer.js` exports a function that returns a tracer configured to [automatically](https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node#user-content-supported-instrumentations) export traces from an Express.js server using [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md). We'll use this function in `app.js`, passing in a service name, `copilot-observability`.

```js title="app.js" linenums="1"
'use strict';
const tracer = require('./tracer')('copilot-observability');
const app = require("express")();
const port = 8080;

app.get("/", (req, res) => {
	res.send("Hello World");
});

app.listen(port, () => {
	console.log(`Listening for requests on http://localhost:${port}`);
});
```

Now, if you deploy this service using Copilot and enable observability in the manifest, you'll be able to [see traces generated by this service!](./observability.en.md#viewing-traces-in-cloudwatch)

### Including Trace Logs
!!!attention
	This section is not applicable to Request-Driven Web Services 

Because Copilot configures the [ECS resource detector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#amazon-ecs) on the collector, traces generated by your service will include the log group your service is logging to.
If you include the trace ID in your logs, logs associated with a trace will show up along with the trace in X-Ray.
This can be helpful for understanding and debugging a trace.

X-Ray [formats their trace IDs](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html#xray-api-traceids) a little [differently than OpenTelemetry](https://opentelemetry.io/docs/reference/specification/trace/api/#spancontext), so you'll need a function like this to take an OpenTelemetry trace ID and format it for X-Ray:
```js
function getXRayTraceId(span) {
	const id = span.spanContext().traceId;
	if (id.length < 9) {
		return id;
	}

	return "1-" + id.substring(0, 8) + "-" + id.substring(8);
}
```

Then, you can include the X-Ray trace ID in your logs like so:
```js
console.log("[%s] A useful log message", getXRayTraceId(span));
```

## Viewing Traces in CloudWatch
After you have instrumented your service and deployed it using Copilot, you're ready to view traces from your service!
Depending on how you've instrumented your service, you will likely need to send a few requests to it before traces will appear in the AWS console.

First, open the CloudWatch console, and click on `X-Ray traces/Service map` in the menu. Here you can see a visual map of services interacting:
![X-Ray Service Map](https://user-images.githubusercontent.com/10566468/166842664-da44756f-7a4b-4e5d-9981-42927b0deb65.png)

Next, you can view details of a specific trace by clicking on `X-Ray traces/Traces` in the menu and selecting a trace from the list.

In this example, you can see a service, `js-copilot-observability`, running some internal Express.js middleware, and then using the [AWS SDK for Javascript](https://aws.amazon.com/sdk-for-javascript/) to call `s3:listBuckets`:
![X-Ray Trace Details](https://user-images.githubusercontent.com/10566468/166842693-65558de5-5a6b-4777-b687-812406580fb6.png)