Span kind: client, server, internal

3 min read

Short answer

Span kind says what role a span plays in a call. CLIENT is an outgoing request whose response the caller waits for; SERVER handles an incoming one. PRODUCER and CONSUMER are the two ends of asynchronous work. INTERNAL, the default, is work that doesn't cross a process boundary.

On this page

An agent’s trace mixes work done inside the process with calls out to other systems: an LLM API, an RPC node. Span kind is the one field that tells the two apart.

OpenTelemetry span kinds in an agent service: SERVER for the incoming request, INTERNAL for agent and tool spans, CLIENT for calls to the LLM API and the RPC node, PRODUCER and CONSUMER around a job queue.

What are the five kinds?

Kind The span describes Agent example
CLIENT A request to a remote service; the caller waits for the response An LLM API call, an RPC call
SERVER Handling a remote request while the client waits Your agent’s HTTP endpoint
PRODUCER Starting or scheduling an operation Putting a job on a queue
CONSUMER Processing what a producer started The worker that takes the job
INTERNAL An operation inside the application A tool call, an agent loop step

The definitions are from the trace API spec. Kind is a hint to the backend about how to assemble the trace. A CLIENT span in one service and a SERVER span in another usually describe two ends of the same call.

What kinds does a real agent trace have?

These are the kinds from a make demo run, as exported:

Span Kind From
invoke_agent INTERNAL AI SDK
step 1 INTERNAL AI SDK
chat CLIENT AI SDK
execute_tool pay_vendor INTERNAL AI SDK
send 31337 CLIENT hashspan
confirm 31337 CLIENT hashspan

The GenAI conventions allow both kinds for invoke_agent: CLIENT for a remote agent, INTERNAL for one running in the same process. The demo agent runs in-process.

How do I set it?

Pass it when you start the span. The API has no way to change it afterwards.

import { SpanKind } from '@opentelemetry/api';
tracer.startSpan('execute_tool pay_vendor').end(); // INTERNAL is the default
tracer.startSpan('send 8453', { kind: SpanKind.CLIENT }).end(); // a call to an RPC node
for (const s of exporter.getFinishedSpans()) console.log(s.name, SpanKind[s.kind]);
execute_tool pay_vendor INTERNAL
send 8453 CLIENT

In JavaScript SpanKind.CLIENT is 2, but the exported OTLP data in Jaeger says 3. OTLP reserves 0 for “unspecified”, so every value moves up by one.

FAQ

What is the difference between a CLIENT and an INTERNAL span?

A CLIENT span describes a request to a remote service while the caller waits for the response, such as an HTTP or RPC call. An INTERNAL span describes work inside the application, such as a function or a tool call.

What is the default span kind?

INTERNAL. If you don't pass a kind when starting a span, it's INTERNAL.

Why does the kind number differ between my code and the exported data?

The JavaScript enum starts at INTERNAL = 0, while OTLP reserves 0 for UNSPECIFIED and starts INTERNAL at 1. Compare names, not numbers.

Go further

Type to search the docs, Learn topics and the blog.