Skip to main content

OpenTelemetry v2 integration

View Markdown

The OpenTelemetry v2 integration connects the Temporal Go SDK to OpenTelemetry through the Plugin API. Use it to propagate application trace context across Temporal calls. You can also opt in to Temporal SDK operation spans and SDK metrics.

This guide demonstrates two approaches:

  • Automatic instrumentation: The Worker plugin creates spans for Temporal SDK operations and reports Temporal SDK metrics.
  • Custom instrumentation: The plugin propagates trace context, but you create the spans that describe your application.

This guide assumes that you understand OpenTelemetry fundamentals. For OpenTelemetry concepts and exporter options, refer to the OpenTelemetry documentation.

Code snippets in this guide are taken from the OpenTelemetry v2 sample. Refer to the sample for the complete, runnable code.

Prerequisites​

  • Set up your local development environment by following Set up your local development environment.
  • Leave the Temporal development server running to test the sample locally.
  • Install Docker to run Jaeger for the local tracing examples.

Install the plugin​

Install the OpenTelemetry v2 plugin:

go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest

The samples also use the OpenTelemetry OTLP gRPC and Prometheus exporters:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@latest
go get go.opentelemetry.io/otel/exporters/prometheus@latest
go get github.com/prometheus/client_golang@latest

Configure tracing​

Create and install a replay-safe tracer provider in every process that creates the plugin or calls temporalotel.Tracer. The replay-safe provider gives Workflow spans consistent identities when Workflow code replays.

opentelemetry-v2/setup.go

func InitializeGlobalTracerProvider(
	ctx context.Context,
	serviceName string,
) (*temporalotel.ReplaySafeTracerProvider, error) {
	exporter, err := otlptracegrpc.New(
		ctx,
		otlptracegrpc.WithEndpoint("127.0.0.1:4317"),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, err
	}

	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)
	return provider, nil
}

The samples use distinct OpenTelemetry service names for each instrumented process. This makes process boundaries clear when one trace contains spans from both a Client and a Worker.

The OTLP exporter uses an insecure loopback connection for this local example. Configure transport security for your production telemetry backend.

Enable automatic instrumentation​

Set AddTemporalSpans to true to have the plugin create spans for Temporal SDK operations performed by the Worker. Provide MetricsHandlerOptions to install the plugin's Temporal SDK metrics handler. A non-nil metrics configuration enables the complete SDK metric set.

opentelemetry-v2/automatic-instrumentation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

UseMonotonicCounters represents Temporal SDK counters as OpenTelemetry Int64Counter instruments instead of Int64UpDownCounter instruments.

Export Temporal SDK metrics​

Create and install a meter provider backed by the OpenTelemetry Prometheus exporter:

opentelemetry-v2/setup.go

func InitializeGlobalMeterProvider(serviceName string) (*sdkmetric.MeterProvider, error) {
	exporter, err := otelprometheus.New()
	if err != nil {
		return nil, err
	}

	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(exporter),
		sdkmetric.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetMeterProvider(provider)
	return provider, nil
}

Start a loopback-only HTTP endpoint that serves the Prometheus handler:

opentelemetry-v2/setup.go

func StartPrometheusEndpoint() (*http.Server, error) {
	mux := http.NewServeMux()
	mux.Handle("/metrics", promhttp.Handler())

	server := &http.Server{
		Addr:              "127.0.0.1:9090",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}
	listener, err := net.Listen("tcp", server.Addr)
	if err != nil {
		return nil, err
	}

	log.Println("Prometheus metrics available at http://127.0.0.1:9090/metrics")
	go func() {
		if err := server.Serve(listener); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Println("Prometheus endpoint failed:", err)
		}
	}()
	return server, nil
}

Opening the listener before starting the serving goroutine makes an address conflict fail during Worker startup. The endpoint at http://127.0.0.1:9090/metrics returns metrics in Prometheus format for an external Prometheus server to scrape.

Run the automatic instrumentation sample​

From the root of a samples-go checkout, start Jaeger and the Worker:

docker compose -f opentelemetry-v2/docker-compose.yaml up -d
go run opentelemetry-v2/automatic-instrumentation/worker/main.go

In another terminal, start the Workflow:

temporal workflow execute \
  --task-queue opentelemetry-v2 \
  --type Workflow \
  --input '"Temporal"'

The trace begins with the Worker's automatic spans, since only the Worker installs the plugin in this sample. These spans describe Temporal SDK operations that the Worker performs, such as running the Workflow and its Activity.

Open the Jaeger UI, select temporal-otel-v2-automatic-worker, and inspect the Worker trace. Open http://127.0.0.1:9090/metrics to inspect the Temporal SDK metrics.

Add custom instrumentation​

By default, the plugin propagates context: spans your application creates remain connected across Clients, Workflows, Activities, and other Temporal calls. The custom samples focus on that span propagation.

Propagate from a Workflow to an Activity​

Use temporalotel.Tracer to create replay-safe spans in Workflow code. Use an ordinary OpenTelemetry tracer in Activity code. Passing the span-derived Workflow context to workflow.ExecuteActivity connects the Activity span to the Workflow span.

opentelemetry-v2/workflow-activity-propagation/opentelemetry.go

const instrumentationName = "github.com/temporalio/samples-go/opentelemetry-v2/workflow-activity-propagation"

func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}

func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}

The Worker creates a default plugin so it can propagate the custom Workflow span to the Activity:

opentelemetry-v2/workflow-activity-propagation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()

From the root of a samples-go checkout, start Jaeger and the Worker:

docker compose -f opentelemetry-v2/docker-compose.yaml up -d
go run opentelemetry-v2/workflow-activity-propagation/worker/main.go

In another terminal, start the Workflow:

temporal workflow execute \
  --task-queue opentelemetry-v2 \
  --type Workflow \
  --input '"Temporal"'

Open the Jaeger UI, select temporal-otel-v2-custom-workflow-activity-propagation-worker, and find the trace containing the workflow-operation and activity-operation spans created by the sample.

Propagate from a Client to an Update​

Install the plugin in both processes when a custom client span must cross a Temporal call. The client-side plugin injects the span context into the Update headers. The Worker plugin extracts it into the Update handler's Workflow context.

Create the replay-safe tracer provider before the plugin in both processes, and add the default plugin when you create each Temporal Client. Keep the send-update span open until the Update reaches the completed stage and returns its result:

opentelemetry-v2/client-update-propagation/starter/main.go

func sendUpdate(
	ctx context.Context,
	c client.Client,
	workflowID string,
	name string,
) (string, error) {
	ctx, span := otel.Tracer(instrumentationName).Start(ctx, "send-update")
	defer span.End()

	handle, err := c.UpdateWorkflow(ctx, client.UpdateWorkflowOptions{
		WorkflowID:   workflowID,
		UpdateName:   clientupdate.UpdateName,
		WaitForStage: client.WorkflowUpdateStageCompleted,
		Args:         []interface{}{name},
	})
	if err != nil {
		return "", fmt.Errorf("unable to send Workflow Update: %w", err)
	}

	var result string
	if err := handle.Get(ctx, &result); err != nil {
		return "", fmt.Errorf("unable to get Workflow Update result: %w", err)
	}
	return result, nil
}

The Update validator creates a replay-safe validate-update span, and the handler creates a handle-update span, changes the Workflow's completion state, and returns the result:

opentelemetry-v2/client-update-propagation/workflow.go

func Workflow(ctx workflow.Context) (string, error) {
	var result string
	updateCompleted := false

	err := workflow.SetUpdateHandlerWithOptions(
		ctx,
		UpdateName,
		func(ctx workflow.Context, name string) (string, error) {
			_, span := temporalotel.Tracer(instrumentationName).Start(ctx, "handle-update")
			defer span.End()

			result = fmt.Sprintf("Hello, %s!", name)
			updateCompleted = true
			return result, nil
		},
		workflow.UpdateHandlerOptions{
			Validator: func(ctx workflow.Context, name string) error {
				_, span := temporalotel.Tracer(instrumentationName).Start(ctx, "validate-update")
				defer span.End()

				if name == "" {
					return fmt.Errorf("name cannot be empty")
				}
				return nil
			},
		},
	)
	if err != nil {
		return "", fmt.Errorf("unable to register Update handler: %w", err)
	}

	if err := workflow.Await(ctx, func() bool { return updateCompleted && workflow.AllHandlersFinished(ctx) }); err != nil {
		return "", err
	}
	return result, nil
}

The Workflow waits for the Update to set its completion state and for workflow.AllHandlersFinished to report all Update handlers done before returning.

From the root of a samples-go checkout, start the Worker:

go run opentelemetry-v2/client-update-propagation/worker/main.go

In another terminal, run the starter:

go run opentelemetry-v2/client-update-propagation/starter/main.go

In Jaeger, select temporal-otel-v2-custom-client-update-propagation-client and open the trace containing send-update, validate-update, and handle-update. The trace crosses into temporal-otel-v2-custom-client-update-propagation-worker, connected by the context the default plugins propagate.

Shut down telemetry providers​

Install the tracer and meter providers as OpenTelemetry globals before constructing the plugin or a Workflow tracer, so they're available to instrument every Temporal call the process makes.

When the process exits, close the Temporal Client or stop the Worker first, then shut down the Prometheus-format endpoint, meter provider, and tracer provider, each with its own fresh, bounded context rather than one that might already be canceled. Shutting down the tracer provider flushes the batch span processor, so spans created earlier in the process still reach the exporter before it exits.

Propagate trace context and baggage​

The plugin automatically propagates OpenTelemetry trace context and baggage across Temporal calls, using a composite W3C Trace Context and Baggage propagator by default instead of the global OpenTelemetry propagator. Use PluginOptions.TextMapPropagator to provide a different propagator.

Temporal headers that carry baggage can be persisted in Workflow Event Histories. Do not put credentials, tokens, or other sensitive data in OpenTelemetry baggage. Set PluginOptions.DisableBaggage to true to turn off baggage propagation.

For other application context, see Context Propagation.

Resources​