# hashspan documentation > Trace your AI agents' on-chain transactions with OpenTelemetry. Coding agents: install the hashspan skill with `npx skills add https://hashspan.dev --skill hashspan` (https://github.com/selimaytac/hashspan/blob/main/skills/hashspan/SKILL.md); every skill and ready prompt: https://hashspan.dev/agents.md. # Introduction Source: https://hashspan.dev/docs hashspan is an open-source TypeScript library that turns each transaction an AI agent sends into OpenTelemetry spans, keyed by the transaction hash, inside the agent's own trace. It works the same for a TypeScript service that sends EVM transactions through viem, the Coinbase CDP SDK, x402 or MPP (mppx), or through a [wallet service](https://hashspan.dev/docs/integrations#wallet-services), such as a payment or payout worker. Other libraries, such as ethers, have no adapter, and chains other than EVM are not supported. Figure: Jaeger showing a reverted confirm 31337 span inside an agent trace, with blockchain.tx.status, the fee, gas used and the decoded revert reason WithdrawalLimitExceeded An agent's reverted transaction, as a confirm span in its own trace (Jaeger, from the [local lab](https://hashspan.dev/docs/local-lab)). Know OpenTelemetry but not blockchains? Start with [hashspan for OpenTelemetry engineers](https://hashspan.dev/docs/otel-engineers): the spans, the schema status, the Collector setup and seven words you'll meet. Not on TypeScript? The same page lists what you can reuse from any language. ## What you get - A `send {chainId}` span when the hash comes back, and a linked `confirm {chainId}` span when the receipt arrives. - Status (`success`, `reverted`, `replaced`; a wait that gives up ends with `error.type` `timeout`, and a transaction a chain reorganisation removed ends with `not_on_chain`), block, gas used and the fee, including the L1 data fee on OP-stack chains. - The decoded revert reason for mined reverts. - For x402 and MPP, a `payment {chainId}` span per paid request: what was paid, to whom and whether it settled. With a reader, a linked `confirm` span adds block, gas and fees of the settling transaction, and `blockchain.payment.verified` says whether that transaction carries the payment. - The agent's identity (`gen_ai.agent.id`, `gen_ai.agent.name`) on every transaction span, when there is an agent. - Optionally, a span per JSON-RPC request with `traceTransport()`, and send, confirmation and fee histograms when an OpenTelemetry metrics SDK is set up. The spans nest under whatever span is active when the transaction is sent, usually your agent framework's `execute_tool` span, or the request or job span of a service. They go to the tracing backend you already use: Jaeger, Grafana Tempo, Langfuse, Honeycomb or any OTLP endpoint. ## Packages | Package | Use it for | |---|---| | [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem) | Agents and services that send with viem wallet clients ([what the trace looks like](https://hashspan.dev/docs/viem-opentelemetry)) | | [`@hashspan/cdp`](https://hashspan.dev/docs/adapters/cdp) | Agents and services that send with Coinbase CDP server accounts | | [`@hashspan/x402`](https://hashspan.dev/docs/adapters/x402) | Agents that pay for API calls with x402 | | [`@hashspan/mpp`](https://hashspan.dev/docs/adapters/mpp) | Agents that pay for API calls with MPP, through mppx (0.x, follows mppx's minor lines) | | [`@hashspan/core`](https://hashspan.dev/docs/adapters/core) | Any other send path; the tracker every adapter uses | ## What it doesn't do hashspan never signs or broadcasts transactions, and it doesn't alert. It records what happened so you can read it next to the agent's decisions. It doesn't slow your calls much either: on a local chain it added about 0.25 ms per transaction at p50 and one JSON-RPC request, none when you pass the nonce ([tracing overhead benchmark](https://hashspan.dev/docs/benchmarks/tracing-overhead)). ## New to OpenTelemetry? The [Learn](https://hashspan.dev/learn) pages explain spans, attributes, context and sampling in a few minutes each, every one on a real agent run. [Your agent's trace stops at the tool call](https://hashspan.dev/blog/agent-trace-stops-at-the-tool-call) shows the problem hashspan solves, end to end. **Next:** Trace your first transaction in the [Quickstart](https://hashspan.dev/docs/quickstart), or run the [local lab](https://hashspan.dev/docs/local-lab) without an API key. --- # Coinbase CDP Source: https://hashspan.dev/docs/adapters/cdp Synced from GitHub This page mirrors packages/cdp/README.md at @hashspan/cdp@1.3.1, the version on npm. Trace the transactions your AI agents, or any of your services, send with [Coinbase CDP](https://docs.cdp.coinbase.com) server accounts, and the user operations of CDP smart accounts, using OpenTelemetry. CDP signs and broadcasts transactions through its API, so no RPC client of yours sees them. This adapter wraps a `CdpClient`: each transaction becomes a `send {chainId}` span inside the active trace and, with a reader, a linked `confirm {chainId}` span with status, gas, fees and revert reason, like [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem). ## Install ```sh npm install @hashspan/cdp @coinbase/cdp-sdk @opentelemetry/api viem ``` Requires Node.js 22.3 or later. From a CommonJS module it needs Node.js 22.12, because the CDP SDK's CommonJS build loads an ES module; ES modules work from 22.3. Bring your own [OpenTelemetry SDK and exporter](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/). ## Usage ```ts import { CdpClient } from '@coinbase/cdp-sdk'; import { withHashspan } from '@hashspan/cdp'; import { createPublicClient, http } from 'viem'; import { baseSepolia } from 'viem/chains'; const cdp = new CdpClient(); // Wraps the client in place; call it once, right after creating the client. const hashspan = withHashspan(cdp, { reader: createPublicClient({ chain: baseSepolia, transport: http() }), }); const account = await cdp.evm.getOrCreateAccount({ name: 'treasury' }); await account.sendTransaction({ network: 'base-sepolia', transaction: { to, value } }); // send + confirm spans // Before a short-lived process exits: await hashspan.flush(); ``` `reader` is a viem public client, or a function `(chainId) => client | undefined` for several chains. Confirmations run in the background through it and never delay your call. Without a reader, only send spans are recorded: the adapter never picks an RPC endpoint itself. The exceptions are the SDK's waits: `waitForTransactionReceipt` on a network-scoped account, which records a confirm span from the receipt it returns, without a revert reason, and `waitForUserOperation` ([Smart accounts](#smart-accounts)). `flush({ timeoutMs })` (default 10 000 ms) waits for every confirm span the adapter still has open, from the reader or from such a wait, and ends what is left as `timeout` if it cannot wait longer (a user operation CDP already reported `complete` ends with what is known). Call it before a short-lived process exits. ### Options `withHashspan(cdp, options)` accepts the [`@hashspan/core` options](https://hashspan.dev/docs/adapters/core#options) (address mode, agent identity, redaction hook, ...), `decodeRevertReason` and `maxBackgroundConfirmations` as in `@hashspan/viem` (the limit applies to confirmations through the reader), `tracker`, `reader`, and `confirmTimeoutMs` (default 120 000 ms; for a user operation CDP reported complete, it also bounds the poll for its bundle receipt). With `tracker`, the core options are not used: they configure the tracker the adapter would otherwise create. Call it once per client: a second call returns the first handle, ignores its options and logs a `diag` warning. Options are read from the object's own enumerable properties: options it inherits through a prototype, such as the getters of a class instance, are ignored ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/cdp@1.3.1/docs/adr/0025-untrusted-input.md)). ### With `@hashspan/viem` If your agent also waits for receipts with a viem client extended by `@hashspan/viem`, give both the same tracker, so that a transaction gets one confirm span, linked to its send span, however many parts of your code wait for it: ```ts import { createTxTracker } from '@hashspan/core'; import { withHashspan as withViemHashspan } from '@hashspan/viem'; const tracker = createTxTracker({ agent: { name: 'treasury-bot' } }); const hashspanViem = withViemHashspan({ tracker }); const reader = createPublicClient({ chain: baseSepolia, transport: http() }).extend(hashspanViem); const hashspanCdp = withHashspan(cdp, { tracker, reader }); // At shutdown, flush both: await Promise.all([hashspanCdp.flush(), hashspanViem.flush()]); ``` A reader client whose chain differs from the transaction's, given directly or returned by a reader function, is not used; a `diag` warning says so. A reader client without a chain is used for every chain. ### Smart accounts Smart accounts send ERC-4337 user operations, which CDP's bundler puts into a bundle transaction. Each operation becomes a `send {chainId}` span, ended when CDP returns its `userOpHash`, and, when your code waits for it with `waitForUserOperation`, a linked `confirm {chainId}` span: ```ts const owner = await cdp.evm.getOrCreateAccount({ name: 'owner' }); const smartAccount = await cdp.evm.getOrCreateSmartAccount({ name: 'treasury', owner }); const { userOpHash } = await smartAccount.sendUserOperation({ network: 'base-sepolia', calls: [{ to, value, data: '0x' }], }); // send span await smartAccount.waitForUserOperation({ userOpHash }); // confirm span ``` The spans carry `blockchain.user_operation.*` attributes instead of a transaction's sender, nonce and fee ([semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions)): - CDP reports `complete` with the bundle transaction's hash, or `failed` without a reason, which ends the confirm span as an error with `error.type` `failed`. A wait that gives up (the SDK's `TimeoutError`) ends it as `timeout`. - `complete` does not say whether the operation's calls succeeded: a bundle can be mined while an operation in it reverts. With a `reader` for the chain, the adapter reads the bundle's receipt and records the operation's `UserOperationEvent`, as its EntryPoint emitted it: success (a reverted operation ends with `error.type` `reverted`), gas used, cost, nonce, paymaster and EntryPoint. Without one, or if the bundle receipt is not found within `confirmTimeoutMs`, the confirm span records the bundle transaction's hash only. - The bundle transaction's status and fee are not recorded: they cover every operation in the bundle. - Without a wait, only the send span is recorded, with or without a reader. A wait names no network: it is traced when the operation was sent through the same client, or on a network-scoped smart account. - A tracker from a `@hashspan/core` without user operations records none: they are passed on untraced, with a `diag` warning. ## Traced While a traced call runs, its send span is the active span, so spans of the CDP API request that your HTTP instrumentation creates nest under it. | Call | Recorded on the send span | |---|---| | `cdp.evm.sendTransaction` | chain id, from, to, value, nonce, function selector (object or serialized transaction) | | account `sendTransaction` | the same | | account `transfer` | from; ETH: to and value; tokens: the token contract when it is given by address (not for a named token such as `'usdc'`), `transfer` and its selector (arguments with `recordFunctionArguments`) | | account `swap`, `useSpendPermission` | chain id and from | | `execute()` of a quote from `cdp.evm.createSwapQuote` or account `quoteSwap` | chain id and from (the taker) | | network-scoped accounts (`useNetwork`) | as above; on Base and Ethereum they send through the account itself, elsewhere through the SDK's own viem client, and both are traced once | | network-scoped `waitForTransactionReceipt` | without a reader: a confirm span with status, block, gas and fees, but no revert reason, and no fees for a flashblocks preconfirmation (a receipt with a zero block hash, whose fee can be another transaction's). A replacement the SDK's viem client reports through `onReplaced` (same sender and nonce) is attributed to the mined transaction; for that, the adapter adds its own `onReplaced`, which calls yours, and passes `{ transactionHash }` on as `{ hash, onReplaced }`. A receipt of another hash that viem did not report ends the confirm span with `error.type` `_OTHER`. The span records the confirmations the wait asked for (`blockchain.tx.wait.confirmations`), as viem applies them: `confirmations` when it is a positive safe integer, `1` when it is omitted, `0`, negative or `NaN`, nothing for any other value, and `1` for `{ transactionHash }`, which waits for one. With a reader, the background confirmation records it from the sealed receipt, with `1` | | smart account `sendUserOperation`, `cdp.evm.sendUserOperation`, `cdp.evm.prepareAndSendUserOperation` | user operation: chain id, hash, sender (the smart account) and the number of calls | | smart account `transfer`, `swap`, `useSpendPermission`; `execute()` of a quote for a smart account (`cdp.evm.createSwapQuote` with `smartAccount`, or smart account `quoteSwap`); `cdp.evm.createSpendPermission`, `cdp.evm.revokeSpendPermission` | user operation: chain id, hash and sender | | network-scoped smart accounts (`useNetwork`) | as above, each traced once | | `waitForUserOperation` (`cdp.evm`, smart accounts, network-scoped smart accounts) | a confirm span: the bundle transaction's hash; with a reader, the operation's outcome and cost from its `UserOperationEvent` | Accounts are traced when they come from `createAccount`, `getAccount`, `getOrCreateAccount`, `importAccount`, `updateAccount` or `listAccounts`, and smart accounts when they come from `createSmartAccount`, `getSmartAccount`, `getOrCreateSmartAccount` or `updateSmartAccount` (`listSmartAccounts` returns records without methods). Networks are mapped to chain ids with `CDP_NETWORK_CHAIN_IDS`; a call on another network, or a network-scoped account created from an RPC URL, is passed through untraced, with a `diag` warning once per network (an RPC URL is never logged, as it can contain an API key). Each call gets its own send span: retrying a call with the same `idempotencyKey` records a second send span, even when CDP returns the transaction of the first attempt; the confirm span is shared. ## Known limits - Not traced: EIP-7702 delegated accounts; Solana (`cdp.solana`); `requestFaucet`, which Coinbase sends; `signTransaction`, which does not broadcast. - Accounts turned into viem accounts with `toAccount()` are sent through your viem client: extend it with `@hashspan/viem`, whose own limits then apply ([`@hashspan/viem` known limits](https://hashspan.dev/docs/adapters/viem#known-limits)). - Networks without a chain id in `CDP_NETWORK_CHAIN_IDS`, and network-scoped accounts created from an RPC URL, are passed through untraced, as are accounts that do not come from the factories listed under [traced](#traced). - Without a `reader`, only send spans are recorded, except for the SDK's waits; a network-scoped `waitForTransactionReceipt` then records no revert reason and no OP Stack operator fee, and no fees for a flashblocks preconfirmation ([usage](#usage), [traced](#traced)). - A user operation gets a confirm span only from `waitForUserOperation`; without a reader it records only the bundle transaction's hash, and the bundle transaction's status and fee are never recorded ([smart accounts](#smart-accounts)). - A retry with the same `idempotencyKey` records a second send span ([traced](#traced)). - The limits of the core apply too ([`@hashspan/core` known limits](https://hashspan.dev/docs/adapters/core#known-limits)). ## The SDK's own telemetry The CDP SDK sends usage and error events to Coinbase by default. It is independent of hashspan; set `DISABLE_CDP_USAGE_TRACKING=true` and `DISABLE_CDP_ERROR_REPORTING=true` to turn it off. ## License Apache-2.0 Concepts Short explanations of the ideas on this page: Traces and spans: the anatomy of one agent run Transaction lifecycle: send, mempool, confirm Span status codes: Unset, Error and Ok --- # core Source: https://hashspan.dev/docs/adapters/core Synced from GitHub This page mirrors packages/core/README.md at @hashspan/core@1.3.1, the version on npm. Transaction lifecycle tracing for the on-chain actions of AI agents and other services that send transactions, built on OpenTelemetry. `@hashspan/core` records these spans inside your existing trace: - **`send {chainId}`**: from "about to send" until the transaction hash is known (or sending failed). - **`confirm {chainId}`**: waiting for the receipt: status, block, gas, fees (including the OP-stack L1 fee) and revert reason. It carries a span link to its `send` span. - **`payment {chainId}`**, instead of a `send` span: a payment that another party settles on chain, such as an x402 payment, or that a payer settles through a payment protocol, such as MPP. The core is library-agnostic and read-only: it never signs, sends or fetches anything. Adapters such as [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem) call it for you. Use the core directly to instrument any other send path. ## Install ```sh npm install @hashspan/core @opentelemetry/api ``` `@opentelemetry/api` is the only peer dependency. Bring your own [OpenTelemetry SDK and exporter](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/). Requires Node.js 22.3 or later; in other runtimes, `hashed` address mode needs a custom `hash` function and otherwise records no addresses, with a `diag` warning. ## Usage ```ts import { createTxTracker } from '@hashspan/core'; import { context } from '@opentelemetry/api'; const tracker = createTxTracker({ agent: { name: 'treasury-bot' } }); // Inside your tool, where the agent framework's span is active: const send = tracker.startSend({ chainId: 8453, from, to, value, functionName: 'transfer' }); let hash: string; try { // Run in the send span's context, so that wallet or RPC spans of the call nest under it. hash = await context.with(send.context, () => sendSomehow()); send.end({ hash }); } catch (error) { send.fail(error); throw error; } // Later, wherever you wait for the receipt (or in a background watcher): const confirm = tracker.startConfirm({ chainId: 8453, hash }); try { const receipt = await waitSomehow(hash); confirm.end({ status: receipt.status, // 'success' | 'reverted' blockNumber: receipt.blockNumber, gasUsed: receipt.gasUsed, effectiveGasPrice: receipt.effectiveGasPrice, l1Fee: receipt.l1Fee, // OP-stack chains // Hash of the mined transaction: if a replacement was mined, the receipt is attributed to it (ADR 0008). transactionHash: receipt.transactionHash, }); } catch (error) { // Withdraws this handle, so that it does not keep the confirm span open. confirm.fail(error); throw error; } ``` `startConfirm` can be called by every part of your code that waits for the receipt: calls for the same transaction share one confirm span. A receipt from any of them ends it; a timeout or failure ends it once every caller gave up. End every handle you start, since an open handle keeps the shared span open. A wait that takes a number of confirmations passes it as `confirmations`; the span records it as `blockchain.tx.wait.confirmations` from the wait that ended it. `send.fail(error, { errorType })` records a library's machine-readable error code as `error.type` instead of the error's class name, if it is a short identifier (`[A-Za-z0-9_.-]`, at most 64 characters); `exception.type` stays the class name. ## What is recorded Chain id, transaction hash, sender/recipient (per `address` mode), value, nonce, function name and selector, and, on confirmation, status, block number, gas used, effective gas price, L1 fee, total fee, OP Stack operator fee and revert reason. For user operations: their hash, smart account, EntryPoint, number of calls, success, gas used, cost, nonce and paymaster. For payments: payer, recipient, asset, amount, settled amount, status and whether the settlement was verified, for x402 the scheme and resource, and for MPP the method, intent, mode, challenge id and resource. For a replaced transaction: the replacing hash and the reason. For call batches: the batch id, sender, number of calls, outcome, status code, atomicity and transaction hashes. On every span: the agent identity. Decoded call arguments are recorded only with `recordFunctionArguments`, and error messages only with `errorMessages`. Attribute definitions: [docs/semconv.md](https://hashspan.dev/docs/reference/semantic-conventions). To look up OpenTelemetry attributes such as `error.type` by name, see the [attributes lookup](https://hashspan.dev/learn/otel-attributes). ## Metrics With an OpenTelemetry metrics SDK set up (or `meterProvider`), the tracker records three histograms: `blockchain.client.send.duration` and `blockchain.client.confirmation.duration` in seconds, and `blockchain.client.fee` in the chain's fee unit (wei of the native currency, unless a sample carries `blockchain.fee.denomination` `token`). Their attributes are the system, the chain and the outcome (and, on samples of user operations, `blockchain.operation.subject`), never an address, hash or agent identity; an `error.type` that is neither an error class name ending in `Error` nor a lower-case code of letters and underscores is recorded as `_OTHER`. The `redact` hook does not run on metrics: a fee it removes from spans is still recorded by `blockchain.client.fee`. To keep a histogram out of your backend, drop it with a View of your metrics SDK (drop aggregation). Definitions: [docs/semconv.md](https://hashspan.dev/docs/reference/semantic-conventions#metrics). ## Options | Option | Default | Description | |---|---|---| | `tracerProvider` | global provider | Tracer provider to use | | `meterProvider` | global provider | Meter provider for the [metrics](#metrics) | | `address` | `'raw'` | `'raw'`, `'hashed'`, `'off'`, or `{ mode: 'hashed', hash: (address) => string }` | | `errorMessages` | `'off'` | What failed spans record about the error: `'off'` (type only), `'sanitized'` (first line, cut to 256 characters and `...`, URLs cut to their scheme, host and port, addresses per `address` mode, calldata removed; in `hashed` and `off` mode any hex longer than an address) or `'raw'` (full message and stack trace). `'raw'` can record RPC URLs that include API keys, as some libraries put the request URL in the message; `'sanitized'` keeps only the first line (viem puts the URL on a later line) and drops the path and query of any URL in it, which is best effort. See [ADR 0006](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0006-error-privacy.md) | | `paymentResource` | `'origin'` | How much of a paid resource's URL `x402.resource` and `mpp.resource` record: `'origin'` (scheme, host and port; nothing for a resource that is not a URL), `'path'` (also the path, never the query string, fragment or user info) or `'off'`. Paths often carry user or account identifiers. At most 512 characters are recorded | | `recordFunctionArguments` | `false` | Record `functionArguments` as a JSON array in `blockchain.contract.function.arguments`: bigints as decimal strings, addresses per `address` mode (longer hex values become `` in `hashed` and `off` mode), at most 4096 characters. Reads only own enumerable data properties: `toJSON()` and getters are never called, so a `Date` records as `{}`; binary data records as `0x` hex; a Proxy's traps still run | | `agent` | none | Agent `{ id, name }`; a field set here always wins, unset fields come from the Baggage entries `gen_ai.agent.id` / `gen_ai.agent.name` | | `agentFromBaggage` | `true` | Read agent identity fields that `agent` leaves unset from Baggage; set to `false` in services that accept requests from outside their trust boundary | | `redact` | none | `(attributes) => attributes`, runs last on every span attribute set, including exception event attributes, but not on [metrics](#metrics); if it throws or returns something other than an attributes object, only non-sensitive identifiers are kept | | `linkTtlMs` | `600000` | How long a sent transaction or user operation can be linked from its confirmation, and how long after a receipt further waits for it add no confirm span | | `maxTrackedTransactions` | `10000` | Upper bound on transactions, and separately on user operations, kept for linking and confirm deduplication | ## More of the API `tracker.startPayment({ chainId, protocol, payer, recipient, asset, amount })` records a payment that another party settles on chain, such as an x402 facilitator, as a `payment {chainId}` span; end it with `end({ status, hash })`, `fail(error)` or `timeout()`, and call `link(hash)` to link a settling transaction's confirm span before the payment ends. A settlement with a hash links the transaction's confirm span to the payment span ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0013-x402-payments.md)). For a payment whose settlement transaction the payer sends or signs itself, as with MPP, pass `sender: 'payer'`, so its fee is not labelled a facilitator's, and the transaction's `hash` when the payer knows it before the settlement is reported: it is recorded from the span's start, a timeout included, and when the tracker sent that transaction itself, the payment span starts with a link to its send span. `tracker.paymentSenders` tells an adapter that the tracker reads these fields ([ADR 0029](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0029-mpp-payments.md)). `tracker.startUserOperationSend({ chainId, sender, entryPoint, callCount })` records a user operation of an ERC-4337 smart account handed to a bundler as a `send {chainId}` span; end it with `end({ userOpHash })` or `fail(error)`. `tracker.startUserOperationConfirm({ chainId, userOpHash })` joins its confirm span, as `startConfirm` does for a transaction, and `end(receipt)` takes the operation's receipt: `success`, `actualGasCost`, `actualGasUsed`, `sender`, `nonce`, `paymaster`, `entryPoint`, `revertReason`, and the bundle transaction's `transactionHash` and `blockNumber`. A receipt with `success: false` ends the span with `error.type` `reverted`; the bundle transaction's status and fee are not recorded, since they cover every operation in the bundle ([ADR 0021](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0021-user-operations.md)). `tracker.startCallBatchSend({ chainId, sender, callCount })` records a batch of calls handed to a wallet with EIP-5792 `wallet_sendCalls` as a `send {chainId}` span; end it with `end({ id })`, the batch id the wallet returned, or `fail(error)`. `end({ id, transactionHashes })` also links the transactions an account sent itself for the batch. `tracker.startCallBatchConfirm({ chainId, id })` joins its confirm span, and `end(status)` takes the batch status: `statusCode`, `atomic`, and `receipts` with their `transactionHash` and `blockNumber`. The status code sets the outcome, and no fee is recorded; see the call batch rows of the [semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions) and [ADR 0022](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0022-call-batches.md). All of these calls accept an explicit parent `Context` as a second argument. An integration that learns about a call only after it started can record it after the fact: pass `startTime` in the input and `endTime` in the options of the handle method, e.g. `send.end({ hash }, { endTime })` ([ADR 0009](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0009-telemetry-off-the-call-path.md)). Every method is safe to call: failures inside the instrumentation are reported through `diag` and never thrown into your code. 1.0 removed the positional forms of earlier releases, such as `send.end(hash, endTime)` ([migrating to 1.0](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/migrating-to-1.0.md)). ## Privacy notes - **Address modes are not anonymity.** `address: 'off'` and `'hashed'` keep addresses out of your telemetry backend. They do not hide who transacted: every span carries the transaction hash, and anyone can look up its sender, recipient, value and calldata in a block explorer. Use them to limit what your backend stores and who can query it, not to make transactions untraceable. - **A `hashed` address can be recovered from known addresses.** The default hash is unkeyed, and addresses are public, so anyone who can read the backend can hash the addresses they know and compare. To keep hashed addresses joinable within your system but not reversible by backend readers, use a keyed hash, such as an HMAC with a secret kept outside the backend: `address: { mode: 'hashed', hash: (address) => createHmac('sha256', secret).update(address).digest('hex') }` (`node:crypto`). - **Agent identity in Baggage travels.** Baggage is propagated to every downstream service your instrumented clients call when a Baggage propagator is configured (it is part of the default OpenTelemetry SDK setup), including third party APIs. Put only identifiers there that may leave your system, such as an opaque agent id. For identifiers that must stay internal, use the tracker's static `agent` option instead, which is recorded on spans but never propagated, or strip the entries before outbound calls. - **Inbound Baggage can claim an identity.** A caller can send Baggage entries with any agent id. A field set in the `agent` option cannot be overridden that way; to ignore identity from Baggage entirely, set `agentFromBaggage: false` ([ADR 0011](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0011-agent-identity-precedence.md)). Services that accept requests from outside their trust boundary should set it. - **Other instrumentation in the same trace has its own settings.** The address mode, the error message mode and the `redact` hook apply to hashspan's spans only. The AI SDK records tool call inputs and outputs, such as recipient addresses and amounts, unless `recordInputs` and `recordOutputs` are false in its `experimental_telemetry` settings. HTTP instrumentation records `url.full`, which includes an RPC provider's API key when it is part of the URL, and with `traceTransport` those HTTP spans are children of hashspan's JSON-RPC spans. Configure those instrumentations to the same policy, or remove the values in your collector. - The redaction hook (`redact`) runs last on every span attribute set and on exception attributes; use it for anything else your policy forbids. It does not run on [metrics](#metrics), which carry no address or hash. - **Your callbacks' errors go to the diagnostic logger.** If a custom `hash` function or the `redact` hook throws, its error object is logged through the OpenTelemetry `diag` logger, outside the address mode and the redaction hook. Errors of the instrumented call never are. Do not put sensitive values, such as the address being hashed, into errors your callbacks throw, or route `diag` to a sink your policy allows. ## Known limits These hold for every adapter; each adapter's README lists its own. - EVM chains only: chain ids are EIP-155 numbers and hashes 32-byte hex ([roadmap candidates](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/roadmap.md#candidates)). - `blockchain.tx.fee` and the fee histogram leave out the OP Stack operator fee. It is recorded apart, as `blockchain.tx.operator_fee`, when the adapter passes the receipt's `operatorFee`: `@hashspan/viem` reads it with one `eth_call`, and an adapter without a client to read it with, such as `@hashspan/cdp` without a `reader`, records none ([attributes](https://hashspan.dev/docs/reference/semantic-conventions#attributes)). - On chains that charge gas in a token (Celo fee currencies, Tempo), `blockchain.tx.fee` and the fee histogram are in that token's unit and are not converted. The confirm span names the token in `blockchain.tx.fee_asset`, and fee samples carry `blockchain.fee.denomination` `token`, only when the adapter passes the asset (`feeAsset` of the send or the receipt): a transaction whose asset it does not know is recorded as paid in the native currency ([fee fields](https://hashspan.dev/docs/reference/semantic-conventions#attributes)). - A confirm span is not revised after it ended: a reorganisation that removes its block leaves its status and block number ([chain reorganisations](https://hashspan.dev/docs/reference/semantic-conventions#spans)). - Values from outside are bounded: longer ones are cut or dropped, and past `maxTrackedTransactions` or `linkTtlMs` a confirm span has no link to its send ([bounds](https://hashspan.dev/docs/reference/semantic-conventions#bounds)). - The `redact` hook does not run on [metrics](#metrics). ## License Apache-2.0 Concepts Short explanations of the ideas on this page: Traces and spans: the anatomy of one agent run Span status codes: Unset, Error and Ok Baggage vs span attributes --- # MPP Source: https://hashspan.dev/docs/adapters/mpp Synced from GitHub This page mirrors packages/mpp/README.md at @hashspan/mpp@0.1.2, the version on npm. Trace the payments your AI agents make with the [Machine Payments Protocol](https://mpp.dev) (MPP) and its TypeScript SDK, [mppx](https://github.com/wevm/mppx), with OpenTelemetry. With MPP, an API answers `402 Payment Required` with a challenge, and the agent pays it on Tempo with a TIP-20 token transfer that it either sends itself (`push`) or signs for the server to broadcast (`pull`). Each payment becomes a `payment {chainId}` span inside your agent's trace, with what was paid, to whom, in which mode and whether it settled, and, with a reader, a linked `confirm {chainId}` span with block, gas and fees, and whether the transaction carries this payment. ## Install ```sh npm install @hashspan/mpp @hashspan/core @hashspan/viem mppx @opentelemetry/api viem ``` `@hashspan/core` and `@hashspan/viem` are needed for the shared tracker of the example below. Requires Node.js 22.3 or later, mppx 0.13.5 or a later 0.13 release, and viem 2.57.1 or later (mppx's own floor). Bring your own [OpenTelemetry SDK and exporter](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/). ## Usage ```ts import { createTxTracker } from '@hashspan/core'; import { withHashspan } from '@hashspan/mpp'; import { withHashspan as withViemHashspan } from '@hashspan/viem'; import { Mppx, tempo } from 'mppx/client'; import { createClient, createPublicClient, http, publicActions, walletActions } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; import { tempo as tempoChain } from 'viem/chains'; // One tracker for both adapters, so that a payment span links to the payer's own send span and each // transaction gets one confirm span. const tracker = createTxTracker(); const account = privateKeyToAccount(privateKey); const wallet = createClient({ account, chain: tempoChain, transport: http() }) .extend(publicActions) .extend(walletActions) .extend(withViemHashspan({ tracker })); const mppx = Mppx.create({ methods: [tempo({ account, getClient: () => wallet })], polyfill: false, }); // Registers observers on the client; call it once per client. const hashspan = withHashspan(mppx, { tracker, reader: createPublicClient({ chain: tempoChain, transport: http() }), }); await mppx.fetch('https://api.example.com/weather'); // payment span, then confirm span // Before a short-lived process exits: await hashspan.flush(); ``` A test runs this example as written, against an mppx server and Anvil in Tempo mode, with only its URLs redirected ([`quick-start.int.test.ts`](https://github.com/selimaytac/hashspan/blob/@hashspan/mpp@0.1.2/packages/mpp/test/quick-start.int.test.ts)). Pass the client `Mppx.create()` returns. Every payment made through its `fetch`, or through the global `fetch` it polyfills (the default), reaches the adapter. The adapter only registers observers with `on()`: it replaces no method, and its observers never throw, never return a value and never change a payment. ### Push and pull With a private key, mppx pays in `pull` mode: it signs the transaction, and the server broadcasts it, so there is no send span, and the confirm span comes from the reader. With `mode: 'push'`, or a wallet (JSON-RPC) account, the payer sends the transaction itself, through the client `getClient` returns: extend that client with `@hashspan/viem`'s `withHashspan()`, as in the example, to record the send span. Give both adapters the same `tracker`: the payment span then starts with a link to that send span, and the transaction gets one confirm span. With two trackers, a push payment gets two confirm spans and two fee samples for one transaction, and no link between the payment and its send. ### Options `withHashspan(mppx, options)` accepts the [`@hashspan/core` options](https://hashspan.dev/docs/adapters/core#options) (address mode, agent identity, redaction hook, ...), `decodeRevertReason` as in `@hashspan/viem` but off by default, `maxBackgroundConfirmations` as in `@hashspan/viem` (it limits the confirmations through the reader; a payment whose confirmation is not started gets no `verified`), `tracker`, `reader` (a viem public client, or a function returning one for a chain id), `confirmTimeoutMs` (default 120 000 ms) and `observeFailures` (default `false`, see [failures](#failures)). Without a reader, only payment spans are recorded, with the settlement as the server reported it. With `tracker`, the core options are not used; it must come from `@hashspan/core` 1.2 or later, which records who sent the settlement, and with an older one nothing is traced, with a `diag` warning. Options are read once, from the object's own enumerable properties: options it inherits through a prototype, such as the getters of a class instance, are ignored ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/mpp@0.1.2/docs/adr/0025-untrusted-input.md)). ### Verified settlements The server's `Payment-Receipt` names the settlement's transaction (`reference`). For `push`, and for `pull` without a fee payer, the payer knows the transaction's hash before it is sent: the payment span records that hash from its start, and a server that reports another one does not replace it. With a reader, the payment span records `blockchain.payment.verified`: `true` when the transaction's receipt carries, from the payer's address, the transfer of the amount to the recipient with the memo that binds it to this challenge (mppx's attribution memo: its tag, the server's realm and the challenge id), and one transfer per split; `false` when it does not, or when the server reported a transaction other than the payer's own. All of it is checked from the challenge the payer signed for and the payer's own credential, not from the server's receipt. For a sponsored pull, whose hash only the server's `reference` names, `true` also needs the mined transaction to be the one the payer signed: the reader fetches it once (`eth_getTransactionByHash`, at most 10 s, off your call's path), and its sender must be the payer and its `nonceKey`, `nonce`, `validBefore` and `validAfter` those of the payer's signed transaction. The attribute is absent when no check was possible, for example: no reader, no receipt, a reverted one, a reader on another chain, a receipt funded in another currency (`fundingCurrency`), values that do not validate, or, for a sponsored pull, a credential whose transaction does not decode, a transaction that cannot be read or one whose sender or nonce fields differ. With a reader, the payment span is exported once the receipt is checked; its end time stays when the response came. ### Shutting down `flush({ timeoutMs })` (default 10 000 ms) waits for paid payments still waiting for an outcome, then for confirmations through the reader, and ends what is left as `timeout` (a payment already settled and waiting for its receipt check ends as settled, without `verified`). It does not wait for a challenge that was never paid (the client's policy refused it, or signing failed): that challenge records nothing and is dropped at its deadline. Call it before a short-lived process exits. ## Recorded | What the client sees | Payment span | |---|---| | A response with a readable `Payment-Receipt` | `blockchain.payment.status` `settled` and the transaction hash: the payer's own, or for a sponsored pull the receipt's `reference` | | A response without a `Payment-Receipt`, or with one that cannot be read | error status, `error.type` `no_settlement` or `_OTHER` | | A 5xx answer, or a paid request that failed on the network | by default, as for no outcome below; with a reader, the confirm span of the payer's hash records the transaction. With `observeFailures`: `pending` and the hash, the confirm span resolves it, or for a sponsored pull, whose hash only the receipt names, error status with `error.type` `PaymentOutcomeUnknownError` | | Creating the payment failed: the client's policy or spending limits, signing, or a push transaction that failed | by default, nothing. With `observeFailures`: error status, the error's class name as `error.type`, no hash | | A challenge the client refused before paying (no method for it, an expired one) | nothing | | A zero-amount challenge, answered with a signed proof | nothing, failures included: no value moves | | No outcome: a 3xx, a 4xx, or a final 402 after payment | error status, `error.type` `timeout`, once the challenge's `expires` (or, without one, mppx's default validity of 25 s) plus 30 s passed (at most 1 h), when more than 1000 payments are open (the oldest first), or on `flush()`; the payer's own hash stays recorded | Every payment span records the payer and the chain (from the credential the payer's client wrote), the recipient, the asset (the challenge's `currency`), the amount, `mpp.method` (`tempo`), `mpp.intent` (`charge`), `mpp.mode` (`push` or `pull`), `mpp.challenge.id` (the id the server chose, which joins your trace to the server's logs and to the transfer's memo; on spans only) and the origin of the resource URL as `mpp.resource`, e.g. `https://api.example.com`: paths of paid APIs often carry user or account identifiers. The core option `paymentResource: 'path'` records the path too (never the query string, fragment or user info), and `'off'` nothing; see [docs/semconv.md](https://hashspan.dev/docs/reference/semantic-conventions). The settlement's fee is the payer's: its fee sample carries no `blockchain.fee.payer`, or `sponsor` when the server paid it as fee payer, never `facilitator`. Count only `settled` payments as paid; a `pending` one is paid once its confirm span records the receipt. What a tool call or task spent is the `amount` of each settled payment, per `blockchain.payment.asset`: amounts of different assets do not add up. The payment span records the payment, not the paid request's status or your tool's outcome; spans are not an accounting record: a trace that a sampler dropped, or that was not exported, misses its payments. ## Failures By default, the adapter does not observe mppx's `payment.failed` event: observing it makes mppx buffer the body of a 5xx answer to a paid request (at most 64 KiB and 2 seconds), so with an observer a longer body reaches you cut, with an `Mppx-Body-Truncated: true` header, and a body whose stream fails makes `fetch` reject. Without it, the adapter changes nothing you get. A paid request answered with a 5xx, or that failed on the network, then ends as `timeout` once the challenge's deadline passed, with the payer's own hash; with a reader, the confirm span then records the transaction. A payment that failed before its credential (the client's policy or spending limits, signing, a failed push send) records nothing. `observeFailures: true` adds the `payment.failed` observer, with mppx's buffering: such payments then end at once, as `pending` with the hash, or as an error. ## Try it locally A payment can be made and traced on a local [Anvil](https://getfoundry.sh) chain in Tempo mode (`anvil --network tempo`), with no public network and no funds: Anvil funds its test accounts with pathUSD. The [charge integration test](https://github.com/selimaytac/hashspan/blob/@hashspan/mpp@0.1.2/packages/mpp/test/charge.int.test.ts) is the complete recipe, with its helpers in [`tempo-anvil.ts`](https://github.com/selimaytac/hashspan/blob/@hashspan/mpp@0.1.2/packages/mpp/test/tempo-anvil.ts): a vitest file that uses the repository's test helpers, to read rather than to copy as is. Take from it: - The chain: Anvil started with `--network tempo` and Tempo's chain id (`4217`), with one block mined before the first charge (a fresh chain's first block fails Tempo's `validAfter` check), and viem's `tempo` chain with Anvil's URL. - The paid API, `paidApi()`: an mppx server (`Mppx.create` from `mppx/server`, with `tempo({ recipient, currency, account, getClient })`) whose `charge({ amount })` answers each request, served as a function that stands in for `fetch`, so no port is opened. With `feePayer: true`, the server sponsors the fee of pull charges. - The paying client: as in [usage](#usage), with a reader on Anvil. ## Known limits - Only Tempo `charge` payments are traced, push and pull. Sessions (payment channels and their vouchers), subscriptions, the `evm` and `stripe` methods and x402 over mppx are made untraced, with a `diag` warning once per method and intent. Only the paying client is traced, not an mppx server. - Paid MCP tools are not traced: `McpClient.wrap` from `mppx/mcp/client` pays with the methods it is given, and fires none of the client's events. - Payments made with the client's manual APIs (`preparePayment`, `prepareRequest`, `createCredential`) fire no outcome event: their payment spans end as `timeout`. - A 3xx, a 4xx or a final 402 after payment fires no event: the payment ends as `timeout`, although the money may have moved; for a push payment, its send span and hash show it. - A sponsored pull's hash, and with it the confirm span, rest on the server's `reference`. Its verdict is `true` only when the mined transaction is the one the payer signed (its sender and nonce fields), so it costs one more request to the reader; when that cannot be shown, the payment has no verdict. - Auto-swap and machine-token routes, whose receipt names another `fundingCurrency`, get no verdict. - A push payment from a wallet (JSON-RPC) account is sent with `sendCallsSync`: the payment span links to the call batch's send span, as checked on Anvil, where viem falls back to `eth_sendTransaction` for a single call; a wallet that carries the batch otherwise (several transactions, a user operation) was not checked. - By default, a 5xx after payment, or a paid request that failed on the network, ends as `timeout` only once the challenge's deadline passed, and a payment that failed before its credential records nothing ([failures](#failures)). `observeFailures: true` records them at once, but lets mppx buffer and cut 5xx bodies. - The adapter cancels its copy of the paid response once the `payment.response` observers ran, unless one of them started reading it: an observer of yours that waits longer than that before it reads the copy finds it cancelled. - The limits of `@hashspan/viem`'s confirmations and of the core apply to the confirm span ([`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem#known-limits), [`@hashspan/core`](https://hashspan.dev/docs/adapters/core#known-limits)). ## Compatibility This package is 0.x and follows mppx, which is 0.x too: it supports one mppx minor line at a time (see [install](#install)), and a new line raises that range in a minor release of this package. The design is in [ADR 0029](https://github.com/selimaytac/hashspan/blob/@hashspan/mpp@0.1.2/docs/adr/0029-mpp-payments.md). ## License Apache-2.0 Concepts Short explanations of the ideas on this page: Span links: connecting spans that aren't parent and child Span attributes: putting the tx hash on the span Span status codes: Unset, Error and Ok --- # viem Source: https://hashspan.dev/docs/adapters/viem Synced from GitHub This page mirrors packages/viem/README.md at @hashspan/viem@1.3.1, the version on npm. OpenTelemetry instrumentation for viem. Trace the transactions your AI agents, or any of your services, send with [viem](https://viem.sh), using OpenTelemetry. A viem client extension that reports to [`@hashspan/core`](https://hashspan.dev/docs/adapters/core): each transaction becomes a `send {chainId}` span inside the active trace, such as your agent's tool call, a request or a job, and each receipt wait a linked `confirm {chainId}` span with status, gas and fees. See also the [viem OpenTelemetry guide](https://hashspan.dev/docs/viem-opentelemetry). ## Install ```sh npm install @hashspan/viem @opentelemetry/api viem ``` Requires Node.js 22.3 or later. Bring your own [OpenTelemetry SDK and exporter](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/). ## Usage ```ts import { createPublicClient, createWalletClient, http } from 'viem'; import { baseSepolia } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: baseSepolia, transport: http() }).extend(hashspan); const reader = createPublicClient({ chain: baseSepolia, transport: http() }).extend(hashspan); // Inside an agent tool, while the framework's span is active: const hash = await wallet.sendTransaction({ to, value }); // send span await reader.waitForTransactionReceipt({ hash }); // confirm span, linked to the send span ``` Reuse the same `withHashspan()` result for every client of one agent: the clients then share one tracker, so confirmations are linked to their sends even when they happen on a different client. Its place among other extensions matters ([apply it last](#apply-it-last)), and a short-lived process flushes it before it exits ([shutting down](#shutting-down)). ## Traced actions | Action | Span | Recorded | |---|---|---| | `sendTransaction` | `send` | chain id, from, to, value, nonce (when the call passes one), function selector, hash, and the EIP-7702 authorizations of a type 4 transaction (count, delegated addresses, chain ids; never signatures) | | `writeContract` | `send` | as above, plus the function name, and the call arguments with `recordFunctionArguments: true` | | `sendTransactionSync`, `writeContractSync` | `send` and `confirm` | as `sendTransaction` or `writeContract` and `waitForTransactionReceipt`; both spans cover the call, since viem returns the hash only with the receipt; a pending receipt is no outcome ([Tempo transactions](https://hashspan.dev/docs/adapters/viem-reference#tempo-transactions)); no `blockchain.tx.wait.confirmations`, as the call takes no count | | `sendRawTransaction` | `send` | on a wallet or a public client, from the signed transaction: chain id (the client's when it has none), to, value, nonce, function selector, hash and EIP-7702 authorizations; no sender, which only the signature gives. A transaction that viem cannot parse, or longer than 128 KiB, records the chain id and hash only | | `sendRawTransactionSync` | `send` and `confirm` | as `sendRawTransaction` and `waitForTransactionReceipt`, over the call like the other sync forms | | `waitForTransactionReceipt` | `confirm` | status, block, gas used, effective gas price, L1 fee (OP-stack) and total fee from the sealed receipt ([preconfirmed receipts](https://hashspan.dev/docs/adapters/viem-reference#preconfirmed-receipts-flashblocks)), the OP Stack operator fee ([operator fee](https://hashspan.dev/docs/adapters/viem-reference#op-stack-operator-fee)), the token a fee was paid in ([fees paid in a token](https://hashspan.dev/docs/adapters/viem-reference#fees-paid-in-a-token)), revert reason ([revert reasons](https://hashspan.dev/docs/adapters/viem-reference#revert-reasons)), replacement ([replaced transactions](https://hashspan.dev/docs/adapters/viem-reference#replaced-transactions)), and the confirmations the wait asked for (`blockchain.tx.wait.confirmations`): your `confirmations` when it is a positive safe integer, `1` when it is omitted, `0`, negative or `NaN`, as viem then waits for one, and nothing for any other value | | `sendUserOperation` | `send` | chain id, smart account, EntryPoint, number of calls, user operation hash; see [Smart accounts](#smart-accounts-erc-4337) | | `waitForUserOperationReceipt` | `confirm` | success, gas used, cost, nonce, paymaster, revert reason, bundle transaction hash and block | | `sendCalls` | `send` | chain id, account, number of calls, batch id; see [Call batches](#call-batches-eip-5792) | | `waitForCallsStatus` | `confirm` | outcome, status code, atomicity, transaction hashes, highest block | | `sendCallsSync` | `send` and `confirm` | as `sendCalls` and `waitForCallsStatus` | While `sendTransaction`, `writeContract`, `sendRawTransaction` (or their sync forms), `sendUserOperation` or `sendCalls` runs, its send span is the active span, so spans that your RPC or HTTP instrumentation creates for the request nest under it; the code after the call stays in your own context. Failed sends, reverted receipts and receipt timeouts set error status; the original error is always rethrown unchanged. Spans record only the error type unless `errorMessages` allows more, because viem error messages include the request arguments. A failed send's `error.type` is the error viem classified the failure as, under the error it throws: for example `NonceTooLowError`, `InsufficientFundsError` or `IntrinsicGasTooLowError` under a `TransactionExecutionError`, or the bundler's error under a `UserOperationExecutionError`; `exception.type` stays the class it threw. For the actions that are not traced, see [known limits](#known-limits). ## Options `withHashspan(options)` accepts all [`@hashspan/core` options](https://hashspan.dev/docs/adapters/core#options) (address mode, error messages, agent identity, redaction hook) plus: | Option | Default | Description | |---|---|---| | `tracker` | new tracker | A tracker from `createTxTracker()`, to share one between adapters; the core options then configure nothing, since they are the tracker's. A tracker from an older `@hashspan/core` records only what that core supports: user operations and call batches are then passed on untraced | | `confirm` | none | `{ mode: 'background', timeoutMs? }` confirms every sent transaction without an explicit wait | | `decodeRevertReason` | `true` | Replay reverted transactions to record their revert reason; `{ timeoutMs }` bounds the replay (default 10 000 ms); see [revert reasons](https://hashspan.dev/docs/adapters/viem-reference#revert-reasons) | | `maxBackgroundConfirmations` | `256` | Most background confirmations (background mode, `watch()` and `watchUserOperation()`) polling at once; see [Background confirmation](#background-confirmation) | | `followMultisigOperations` | off | `true` or `{ timeoutMs? }` (default 120 000 ms): after a pending receipt of a Tempo multisig operation, wait for the transaction submitted for it; see [Tempo transactions](#tempo-transactions) | Options are read once, from the object's own enumerable properties: options it inherits through a prototype, such as the getters of a class instance, are ignored ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0025-untrusted-input.md)). ## Shutting down Some spans end after the traced call returned: background confirmations, confirmations of reverted transactions, which wait for the revert reason, confirmations of [preconfirmed receipts](https://hashspan.dev/docs/adapters/viem-reference#preconfirmed-receipts-flashblocks), which wait for the sealed receipt, and waits with `confirmations` above 1, whose receipt is read again once the wait resolved. In scripts, CLI agents and serverless functions, flush them before shutting the OpenTelemetry SDK down, or they are lost: ```ts await hashspan.flush(); // at most 10 s by default: hashspan.flush({ timeoutMs }) await provider.shutdown(); ``` `flush()` waits only for the work of its own `withHashspan()` result, even when results share one `tracker`: an app with several results, or with other hashspan adapters, flushes each of them. What a timeout ends, and how: [shutting down](https://hashspan.dev/docs/adapters/viem-reference#shutting-down). ## Background confirmation Some agent frameworks wait for receipts through their own client, or never wait at all. With `confirm: { mode: 'background' }`, every transaction sent through the extended client gets a confirm span anyway: the adapter polls for the receipt through the sending client and records the result. The send call is not delayed. ```ts const wallet = createWalletClient({ account, chain, transport: http() }).extend( withHashspan({ confirm: { mode: 'background', timeoutMs: 60_000 } }), ); ``` The requests it adds, the process it keeps alive, how it shares a span with your own waits and its limit: [background confirmation](https://hashspan.dev/docs/adapters/viem-reference#background-confirmation). ## Transactions sent elsewhere When a transaction is sent by something other than an extended client, such as a wallet API or another library, `watch()` confirms it through a viem client in the background. The confirm span carries the receipt, revert reason and fees like any other, and links to a send span recorded by the same tracker: ```ts const hashspan = withHashspan(); const reader = createPublicClient({ chain: baseSepolia, transport: http() }); const { transactionHash } = await walletApi.send(tx); // not traced by hashspan hashspan.watch(reader, { hash: transactionHash }); ``` To get a send span as well, record the call that sends with the core's tracker and give `withHashspan()` the same tracker (install `@hashspan/core` too, at the minor of `@hashspan/viem`): the send span then covers the call, including the service's queue and approval steps, and the confirm span links to it. ```ts const tracker = createTxTracker(); const hashspan = withHashspan({ tracker }); const reader = createPublicClient({ chain: baseSepolia, transport: http() }); const send = tracker.startSend({ chainId: baseSepolia.id, from, to, value }); let hash: `0x${string}`; try { // The API call runs in the send span's context, so its HTTP span nests under the send span. ({ transactionHash: hash } = await context.with(send.context, () => walletApi.send(tx))); send.end({ hash }); } catch (error) { send.fail(error); throw error; } hashspan.watch(reader, { hash }); // linked to the send span ``` The options of `watch()` (`chainId`, `timeoutMs`, `abi`, `onReceipt`) and the parent of its confirm span: [transactions sent elsewhere](https://hashspan.dev/docs/adapters/viem-reference#transactions-sent-elsewhere). ## Apply it last `client.extend()` lets later extensions replace earlier actions. Extensions such as `publicActions` define `waitForTransactionReceipt` again and hide the traced version, so apply `withHashspan()` after them: ```ts // Traced walletClient.extend(publicActions).extend(withHashspan()); // Not traced: publicActions replaces waitForTransactionReceipt walletClient.extend(withHashspan()).extend(publicActions); ``` When a framework extends the client you pass in, check whether confirm spans appear; send spans are unaffected. Extensions that add their own actions are the other way round. The actions of Tempo's `tempoActions()` (`viem/tempo`), such as `token.transfer`, call viem's actions with the client they extend: they are traced only when `withHashspan()` was applied before `tempoActions()`, and then through the action they reach (`token.transfer` as a `sendTransaction` to the token). Applied after it, they are not traced; their results and requests are the same either way. The same holds for the `sendTransaction` and `writeContract` of a smart account SDK: apply `withHashspan()` before its actions to trace their user operations ([smart account SDKs](#smart-account-sdks)). ## Many transactions In a worker or a bot that sends many transactions, these limits decide what is recorded: - **Background confirmations at once.** Past `maxBackgroundConfirmations`, a transaction gets no background confirm span and so no confirmation or fee sample ([background confirmation](#background-confirmation)). Size it to the transactions you send per second times how long one takes to confirm, or wait for receipts yourself: your own waits are not counted. - **Requests to your provider.** Each background confirmation polls for its receipt through the sending client, at the client's `pollingInterval` ([JSON-RPC requests hashspan adds](https://hashspan.dev/docs/architecture#json-rpc-requests-hashspan-adds)). - **Links to the send.** The tracker keeps a send for `linkTtlMs` after it ended, and at most `maxTrackedTransactions` sends ([core options](https://hashspan.dev/docs/adapters/core#options)). A confirmation that starts later, or after that many newer sends, has no link, and without an active span starts a trace of its own. Background confirmation starts at the send and is not affected; a wait or a `watch()` you start later is, so raise both when you confirm in batches long after sending. - **Sampling.** The send, confirmation and fee [metrics](https://hashspan.dev/docs/reference/semantic-conventions#metrics) record every transaction whatever the sampler decides, so their percentiles stay complete while you sample traces. With a parent-based sampler, a send and its confirm span are kept or dropped together ([troubleshooting](https://hashspan.dev/docs/troubleshooting#a-send-span-but-no-confirm-span)). ## Smart accounts (ERC-4337) A smart account sends user operations, not transactions: a bundler includes them in a bundle transaction that the bundler sends. On a bundler client from viem's `createBundlerClient`, `withHashspan()` traces `sendUserOperation` and `waitForUserOperationReceipt` as `send` and `confirm` spans identified by the user operation hash: ```ts import { createPublicClient, http } from 'viem'; import { createBundlerClient } from 'viem/account-abstraction'; import { baseSepolia } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; const client = createPublicClient({ chain: baseSepolia, transport: http() }); const bundler = createBundlerClient({ account, client, transport: http(bundlerUrl) }).extend( withHashspan(), ); const hash = await bundler.sendUserOperation({ calls: [{ to, value }] }); // send span await bundler.waitForUserOperationReceipt({ hash }); // confirm span, linked to the send span ``` What each span covers and records, and which bundler actions are traced: [smart accounts](https://hashspan.dev/docs/adapters/viem-reference#smart-accounts-erc-4337). ### Smart account SDKs The smart account clients of permissionless.js and ZeroDev send a user operation from `sendTransaction` and `writeContract` (and, in permissionless, `sendCalls`), and return the bundle transaction's hash. On the finished client, those calls are not traced, with a `diag` warning; for user operation spans from them, apply `withHashspan()` before the SDK's actions. Details: [smart account SDKs](https://hashspan.dev/docs/adapters/viem-reference#smart-account-sdks). ### User operations sent elsewhere When a smart account SDK or a wallet sends a user operation for you and returns only a hash, `watchUserOperation()` records the operation's confirm span from it, in the background: its success, gas used and cost, nonce and paymaster, and the bundle transaction's hash and block, as for a wait, linked to a send span recorded by the same tracker. `watch()` on the same hash would record the bundle transaction instead. ```ts import { createPublicClient, http } from 'viem'; import { withHashspan } from '@hashspan/viem'; const hashspan = withHashspan(); const reader = createPublicClient({ chain, transport: http(rpcUrl) }); // The bundle transaction's hash a wallet or an SDK returned, and the smart account that sent the operation: hashspan.watchUserOperation(reader, { bundleHash: hash, sender: smartAccountAddress }); // Or the user operation hash, through a bundler client (createBundlerClient): hashspan.watchUserOperation(bundler, { userOpHash }); // Before a short-lived process exits: await hashspan.flush(); ``` How the operation is found in the bundle, which client each form needs, and its options: [user operations sent elsewhere](https://hashspan.dev/docs/adapters/viem-reference#user-operations-sent-elsewhere). ## Call batches (EIP-5792) A wallet that supports EIP-5792 takes a batch of calls with `wallet_sendCalls` and returns a batch id; the wallet decides how the calls reach the chain. `withHashspan()` traces `sendCalls`, `waitForCallsStatus` and `sendCallsSync` as `send` and `confirm` spans identified by the batch id: ```ts import { createWalletClient, custom } from 'viem'; import { base } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; const wallet = createWalletClient({ account, chain: base, transport: custom(provider) }).extend( withHashspan(), ); const { id } = await wallet.sendCalls({ calls: [{ to, value }] }); // send span await wallet.waitForCallsStatus({ id }); // confirm span, linked to the send span ``` What each span records, the `experimental_fallback` transactions and the viem releases `sendCallsSync` needs: [call batches](https://hashspan.dev/docs/adapters/viem-reference#call-batches-eip-5792). ## JSON-RPC requests `traceTransport()` wraps a viem transport so that each request it sends becomes a client span named after its JSON-RPC method, such as `eth_sendRawTransaction`. With `withHashspan()`, the requests of a transaction nest under its `send` span, so a slow or failing provider call shows up there: ```ts const wallet = createWalletClient({ account, chain, transport: traceTransport(http(), { methods: (method) => method !== 'eth_getTransactionReceipt', }), }).extend(withHashspan()); ``` The spans record the method and the host and port of the URL, never parameters, results or the URL's path. Their attributes and options: [JSON-RPC requests](https://hashspan.dev/docs/adapters/viem-reference#json-rpc-requests). ### Trace context for the provider The JSON-RPC spans send nothing to the provider. To let an RPC gateway that reads W3C trace context, such as eRPC, record its spans in your trace, pass `propagateTraceContext()` as the `onFetchRequest` hook of viem's `http()` transport. Each request then carries the `traceparent` header of its JSON-RPC span: ```ts const wallet = createWalletClient({ account, chain, transport: traceTransport( http('https://rpc.example.com', { onFetchRequest: propagateTraceContext() }), ), }).extend(withHashspan()); ``` What it sends, and why not together with an HTTP instrumentation of `fetch`: [trace context for the provider](https://hashspan.dev/docs/adapters/viem-reference#trace-context-for-the-provider). ## Tempo transactions A sponsored fee on Tempo is not counted as the sender's, and a sync send that a multisig relay answers with a pending receipt gets a confirm span without an outcome, unless `followMultisigOperations` waits for the transaction submitted for it. Details: [Tempo transactions](https://hashspan.dev/docs/adapters/viem-reference#tempo-transactions). ## More in the reference The [`@hashspan/viem` reference](https://hashspan.dev/docs/adapters/viem-reference) has the details of these features. ### Revert reasons A reverted transaction is replayed with `eth_call` to record its decoded revert reason. Details: [revert reasons](https://hashspan.dev/docs/adapters/viem-reference#revert-reasons). ### Clients without a chain For a client without a `chain`, the adapter asks the node for its chain id alongside each traced call. Details: [clients without a chain](https://hashspan.dev/docs/adapters/viem-reference#clients-without-a-chain). ### Replaced transactions A sped-up or cancelled transaction is recorded with the receipt of the transaction that replaced it. Details: [replaced transactions](https://hashspan.dev/docs/adapters/viem-reference#replaced-transactions). ### Preconfirmed receipts (flashblocks) A receipt answered before its block is sealed is completed from the sealed receipt. Details: [preconfirmed receipts (flashblocks)](https://hashspan.dev/docs/adapters/viem-reference#preconfirmed-receipts-flashblocks). ### OP Stack operator fee On OP Stack chains that charge an operator fee, it is recorded apart from the execution and L1 fees. Details: [OP Stack operator fee](https://hashspan.dev/docs/adapters/viem-reference#op-stack-operator-fee). ### Fees paid in a token On Celo and Tempo, the token a fee was paid in is recorded with the fee. Details: [fees paid in a token](https://hashspan.dev/docs/adapters/viem-reference#fees-paid-in-a-token). ## Known limits - Not traced: `deployContract` ([#36](https://github.com/selimaytac/hashspan/issues/36)). - `sendTransactionSync` and `writeContractSync` (viem 2.38.0) send and wait in one call: their send span ends when the receipt arrives, and a wait that times out inside the call is recorded as a failed send without a hash and no confirm span. - Only actions called as methods of an extended client are traced as such: a library that calls viem's actions as functions bypasses the extension for that action, and one that creates its own client bypasses it entirely. A function is recorded only through the client actions viem calls inside it, with less detail ([libraries that take a viem client](https://hashspan.dev/docs/integrations#libraries-that-take-a-viem-client)), and an extension applied after `withHashspan()` can hide the traced actions ([apply it last](#apply-it-last)). - A transaction sent by a wallet API or a wallet provider that creates its own client gets a confirm span through `watch()`, and a send span only when you record the call that sends with the same tracker ([transactions sent elsewhere](#transactions-sent-elsewhere)). - Only waits are traced: polling `getTransactionReceipt` or `getCallsStatus` yourself records nothing ([call batches](#call-batches-eip-5792)). - Background confirmation and `watch()` cover transactions only: a user operation or a call batch gets a confirm span only from your own wait ([smart accounts](#smart-accounts-erc-4337), [call batches](#call-batches-eip-5792)), or for a user operation from `watchUserOperation()`. `watch()` on a bundle transaction's hash records the bundle transaction, not the operations in it; `watchUserOperation()` records one operation of it, and nothing when the sender has several there without a nonce given ([user operations sent elsewhere](#user-operations-sent-elsewhere)). - A smart account SDK's `sendTransaction`, `writeContract` and `sendCalls` are not traced on a client extended after the SDK's actions ([smart account SDKs](#smart-account-sdks)). - `sendCallsSync` is traced from viem 2.45.2, and call batches record no fee ([call batches](#call-batches-eip-5792)). - A transaction sent while `maxBackgroundConfirmations` confirmations are polling gets no background confirm span, and in serverless runtimes that freeze after the response, background confirmations may not complete ([background confirmation](#background-confirmation)). - Revert reasons are best effort: a provider without historical state cannot replay the transaction, and earlier transactions in the same block can change the result ([revert reasons](https://hashspan.dev/docs/adapters/viem-reference#revert-reasons)). - Replaced transactions, Tempo, preconfirmed receipts, Celo fee currencies, clients without a chain and older viem releases have limits of their own ([reference](https://hashspan.dev/docs/adapters/viem-reference#known-limits)). - The limits of the core apply too ([`@hashspan/core` known limits](https://hashspan.dev/docs/adapters/core#known-limits)). ## License Apache-2.0 Concepts Short explanations of the ideas on this page: Traces and spans: the anatomy of one agent run Context propagation in async Node.js Span links: connecting spans that aren't parent and child --- # viem adapter reference Source: https://hashspan.dev/docs/adapters/viem-reference Synced from GitHub This page mirrors docs/viem.md at @hashspan/viem@1.3.1, the version on npm. The details of [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem) that its README links to: how each feature behaves at its edges, what it costs in requests, and the limits of single features. Start with the README; each section here has the name of the README section it continues. ## Transactions sent elsewhere Options of `watch()`: `chainId` (defaults to the client's chain; without either, when either is not a positive safe integer, or when it contradicts the client's chain, nothing is recorded and a `diag` message says why; a client without a chain is asked for its chain id with `eth_chainId` first), `timeoutMs` (default 120 000 ms), `abi`, to decode custom errors in the revert reason, and `onReceipt`, called once when the watch ends with the receipt of the mined transaction, or with `undefined` when none was retrieved; it never affects the confirm span. A `hash` that is not a 32-byte hex hash records nothing and sends no request: `onReceipt` is called with `undefined`. `watch()` never throws or waits; `flush()` awaits the confirmation, not the callback. The confirm span's parent is the span active when `watch()` is called. Without one, it is the send span's parent when the same tracker sent the transaction; otherwise, as for a hash sent by another tracker or by something untraced, the confirm span is in a trace of its own, with no send span and no link ([confirm span parent](https://hashspan.dev/docs/reference/semantic-conventions#spans)). ## Shutting down `flush()` resolves `true` when all pending work finished and `false` on timeout; it never rejects, and it keeps the process alive while it waits. On timeout, confirm spans still waiting are ended and exported: with the receipt if only the revert reason or the OP Stack operator fee was still pending (without them), without fees if only the sealed receipt was, otherwise as an error with `error.type` `timeout`. Background confirmations keep polling until their own `timeoutMs`, so short-lived processes should keep that short. The timeout also ends the spans of your own `waitForTransactionReceipt` calls that are still waiting, and a receipt they return later is not recorded: call `flush()` only when the process is shutting down. Long-running services do not need `flush()`. ## Background confirmation - Each transaction gets one confirm span per tracker. If the caller also waits for the receipt on a client extended with the same tracker, both waits share that span: a receipt from either ends it, and a background timeout does not end it while the caller is still waiting. A wait without a timeout (`timeout: 0`) for a transaction that is never mined therefore keeps the span open. - Polling adds RPC requests to your provider (one receipt request per polling interval until the receipt arrives). It polls independently of your own `waitForTransactionReceipt` calls, so your `timeout`, `confirmations` and other options always apply to your wait; while both run, receipt requests are made for each. - Background confirmation and `watch()` wait for one confirmation and record `blockchain.tx.wait.confirmations` `1`. The span records the count of the wait that ended it, and the background wait usually gets the receipt first: your own wait with `confirmations: 12` for the same transaction then adds nothing to it ([one confirm span per transaction](https://hashspan.dev/docs/reference/semantic-conventions#spans)). - A pending confirmation keeps the Node.js process alive until the receipt arrives or `timeoutMs` (default 120 000 ms) passes; the span then ends as an error with `error.type` `timeout`. - viem's `waitForTransactionReceipt` fails on the first failed request of its poll (a rate limit, a JSON-RPC error, a request timeout, a connection reset), and with `TransactionReceiptNotFoundError` when a node returns a mined transaction before its receipt. Background confirmation and `watch()` then wait one polling interval and poll again, until `timeoutMs`: the span ends as `timeout` only if no receipt came, and a provider that keeps failing holds the confirmation (and its place under `maxBackgroundConfirmations`) until then. Your own waits are passed on unchanged, including these errors. - At most `maxBackgroundConfirmations` (default 256) background confirmations, including those of `watch()` and `watchUserOperation()`, poll at once. A transaction sent while that many are polling gets no background confirm span, and a `diag` warning is logged; `0` turns background confirmation off. Your own waits are not counted. See [ADR 0018](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0018-background-confirmation-limit.md). - In serverless runtimes that freeze after the response, background confirmations may not complete. ## Revert reasons A receipt only says that a transaction reverted. For reverted transactions the adapter replays the transaction with `eth_call` on the previous block's state (and, if that does not revert, as when the contract was created earlier in the same block, once more on its own block) and records the decoded reason as `blockchain.tx.revert.reason`: `Error(string)` messages, `Panic` codes, and custom errors when the ABI is known (transactions sent with `writeContract`), otherwise the error selector. To look up an error message by hand, use the [error decoder](https://hashspan.dev/errors/decode). - Two extra RPC requests per reverted transaction (three when the second replay is needed); none for successful ones. - Best effort: the replay can differ when earlier transactions in the same block changed the state, and providers without historical state cannot replay. The reason is then missing. - `waitForTransactionReceipt` returns as soon as the receipt is available; the confirm span ends once the reason has been fetched. - The replay is bounded: if the provider has not answered within 10 s (`decodeRevertReason: { timeoutMs }`), the receipt is recorded without a reason and the confirm span ends. - Turn it off with `withHashspan({ decodeRevertReason: false })`. ## Clients without a chain Prefer clients with a `chain`. For a client without one, the adapter asks the node for its chain id alongside each traced call, never before it, and records the span once the answer arrives, with the call's start and end time: - The call is never delayed or failed by that request. If it fails, or has not answered 30 s after the call ended, that call is not traced. - The chain id is asked for on every call, so spans follow a wallet that switches networks (one extra `eth_chainId` per traced call). - These spans have millisecond precision and are exported shortly after the call ends. - RPC or HTTP spans of the call do not nest under the send span, since it does not exist yet while the call runs. ## Replaced transactions A pending transaction can be sped up or cancelled by sending another one with the same nonce. viem then resolves `waitForTransactionReceipt` with the receipt of the replacing transaction. hashspan records that receipt on the confirm span of the transaction that was mined, and ends the confirm span of the awaited hash with `blockchain.tx.status = replaced`, `blockchain.tx.replacement.hash` and `blockchain.tx.replacement.reason` (`repriced`, `cancelled` or `replaced`, as viem classifies it). Fees and status therefore always belong to the hash that paid them. - Your `onReplaced` callback is called unchanged, and your wait still resolves with viem's result. If your callback throws, the wait rejects with its error as in plain viem; the replacement is recorded anyway. - viem detects a replacement only after its wait has read the original transaction. A replacement sent right after the original, before the wait read it, takes it out of the node's pool unseen, and viem's wait times out ([wevm/viem#5242](https://github.com/wevm/viem/issues/5242)). hashspan then looks for the replacement itself, for a transaction sent through the same `withHashspan()` result: it records it as above, without `blockchain.tx.replacement.reason`, as viem reported none. It knows the sender and nonce from your call, or, when you pass no `nonce` or send a signed transaction, from one `eth_getTransactionByHash` it sends through the sending client right after the send, off your call's path; not through an EIP-1193 provider (`custom()`), such as a wallet service's, which hashspan sends nothing. It searches the last 128 blocks, with at most 13 requests after the timeout; your wait still rejects with viem's `WaitForTransactionReceiptTimeoutError`, and both confirm spans end when it did. - `checkReplacement: false` turns off viem's detection, and with it this attribution. - A replaced transaction is not an error: whether a cancellation is a failure is up to your application. ## Preconfirmed receipts (flashblocks) Some RPC nodes, such as Base's, answer `eth_getTransactionReceipt` before the block is sealed, with a receipt whose block hash is zero. Its status, block and gas are those of the sealed receipt, but its L1 fee can be another transaction's. hashspan therefore records fees from the sealed receipt: - A receipt with a zero or null block hash is treated as a preconfirmation. The adapter reads the receipt again, off your call's path, once per polling interval (1 s without one), for at most 30 s and never past a background confirmation's `timeoutMs`, and records the sealed receipt. The span keeps the time the preconfirmation arrived as its end time. - If no sealed receipt comes in time, or `flush()` cannot wait for it, the span records the preconfirmation without `effective_gas_price`, `l1_fee` and `fee`, since a fee without its L1 part would look valid and be too low. - Your `waitForTransactionReceipt` still returns what the node returned, and `watch()`'s `onReceipt` gets the receipt viem resolved with. - See [ADR 0024](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0024-sealed-receipt-fees.md). ## OP Stack operator fee OP Stack chains after the Isthmus upgrade can charge an operator fee on top of the execution and L1 fees. A node adds `operatorFeeScalar` and `operatorFeeConstant` to the receipt when the chain charges one; viem passes them on unformatted. Only for such a receipt, and only from the sealed receipt, the adapter makes one `eth_call`, off your call's path, to the GasPriceOracle's `getOperatorFee(gasUsed)` at the receipt's block, through the client that read the receipt, and records the result as `blockchain.tx.operator_fee`. `blockchain.tx.fee` and the fee histogram do not include it. A receipt without the fields, as on Base and OP Mainnet while their operator fee is zero, costs no request. If the call fails, answers with something other than one `uint256`, or takes longer than 10 s, the receipt is recorded without it. The span keeps the time the receipt arrived as its end time. ## Fees paid in a token On Celo, a transaction with a `feeCurrency` pays gas in that token, and on Tempo every transaction pays in a fee token, a plain EIP-1559 one as well as one of Tempo's type `0x76`. The fee attributes are recorded as the receipt gives them, never converted, and the confirm span names the token in `blockchain.tx.fee_asset` (under the address mode), while the fee sample carries `blockchain.fee.denomination` `token`. The adapter reads Celo's `feeCurrency` from the arguments of `sendTransaction`, `writeContract` and their sync forms, and Tempo's `feeToken` from a receipt whose type is `0x76`, or of another type that also names a `feePayer`, as Tempo's receipts all do; it imports no chain module and makes no request for either. A transaction sent with `sendRawTransaction`, or sent elsewhere and confirmed with `watch()`, records no Celo fee currency, since only the transaction names it ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0028-fee-asset.md)). ## Tempo transactions - **Sponsored fees.** Only a Tempo transaction of type `0x76` can have its fee paid by another account, which its receipt names in `feePayer`. When it is a valid address other than the receipt's `from`, the fee sample carries `blockchain.fee.payer` `sponsor`, so it is not counted as the sender's fee; no address is recorded on the sample or the span ([ADR 0020](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0020-metrics.md)). - **Pending receipts.** A Tempo multisig relay (`Relay.multisig` of `viem/tempo`) answers a sync send whose approvals are below quorum with a receipt whose `status` is `pending` (or none, without viem's Tempo formatter) and whose hash is the multisig operation's. The send span ends when the call returns, as usual. A pending receipt is no outcome: it withdraws the wait, as a timeout does and as a pending call batch does, so the confirm span ends without an outcome, error or metric sample when no other wait of the hash is running, and no request is made. A wait that resolves with one does the same. - **Following a multisig operation.** With `followMultisigOperations`, the confirm span of such a sync send keeps waiting off your call's path: the adapter asks for the operation's receipt with `eth_getTransactionReceipt`, spread over the option's `timeoutMs` (default 120 000 ms), at most 60 times and never more often than the client's polling interval. Through a remote relay (`withRelay(http(), http(relayUrl))`), each poll that finds nothing costs 3 JSON-RPC requests: the receipt from the default node, `multisig_getOperation` and the receipt from the relay. When the relay has submitted the transaction, its receipt names the operation under `multisig` and is recorded on the span, which keeps the operation's hash; otherwise the span ends as `timeout`. The follow counts towards `maxBackgroundConfirmations` and `flush()` ends it; when the limit is reached, or is `0`, nothing is followed and the span ends without an outcome, as without the option ([ADR 0018](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0018-background-confirmation-limit.md)). With the option, your own `waitForTransactionReceipt` of the operation's hash records the submitted transaction's receipt the same way. ## Smart accounts (ERC-4337) - The send span covers preparing, signing and handing the operation to the bundler, and is the active span while that runs, so spans that your RPC or HTTP instrumentation creates for the bundler and paymaster requests nest under it. It records the smart account, EntryPoint, number of calls and user operation hash, and no `blockchain.tx.*` attribute. - The confirm span records the operation's success, gas used and cost (`actualGasCost`), nonce (a decimal string) and paymaster, and the bundle transaction's hash and block. An operation whose calls reverted ends with `error.type` `reverted` and its decoded revert reason, even though the bundle transaction succeeded. The bundle transaction's status and fee are not recorded: they cover every operation in the bundle. The fee histogram records the operation's cost ([ADR 0021](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0021-user-operations.md)). - The chain id is the bundler client's, which `createBundlerClient` takes from its `client`; without one, the spans are recorded once the bundler answered `eth_chainId`, as for [clients without a chain](#clients-without-a-chain). - Only these two bundler actions are traced; viem calls the others from inside them. Background confirmation and `watch()` cover transactions only: a user operation gets a confirm span when you wait for its receipt, or from [`watchUserOperation()`](#user-operations-sent-elsewhere). ### Smart account SDKs The smart account clients of permissionless.js (`createSmartAccountClient`) and ZeroDev (`createKernelAccountClient`) are bundler clients with `sendTransaction`, `writeContract` and, in permissionless, `sendCalls` on top. These send a user operation through the client the SDK added them to, wait for its receipt and return the bundle transaction's hash, so `withHashspan()` applied to the finished client does not see the operation: - On such a client, `sendUserOperation` and `waitForUserOperationReceipt` are traced as above. - A `sendTransaction`, `writeContract` or `sendCalls` whose account (the call's, else the client's) is a smart account is not traced, with a `diag` warning: it sends no transaction and no call batch, and the hash it returns is the bundle's, whose status and fee belong to the bundler. Their sync forms are not traced either. - For user operation spans from those actions, apply `withHashspan()` before the SDK's actions: `createBundlerClient(...).extend(withHashspan()).extend(smartAccountActions)` with permissionless ([setup](https://hashspan.dev/docs/integrations#smart-account-sdks)). The returned hash is the bundle transaction's: a wait for it on a traced client records the bundle transaction, not the operation. `watchUserOperation()` with that hash and the smart account's address records the operation ([below](#user-operations-sent-elsewhere)). ### User operations sent elsewhere - **With `bundleHash` and `sender`**, a client of the chain's node, such as a public client, reads the bundle receipt (`eth_getTransactionReceipt`; a bundler client would ask its bundler), and the operation is its one `UserOperationEvent` with that `sender` topic, emitted by the contract the bundle transaction called or by a canonical EntryPoint (0.6 to 0.9); events of other contracts do not count. Give `nonce` when the sender can have more than one operation in a bundle: with several matches and no nonce, nothing is recorded. Give `userOpHash` too when you know it, and the event must carry it. No match, a receipt of more than 10 000 logs, or no receipt by `timeoutMs` record nothing, with a `diag` message: the operation's hash, which keys its span, is only known from the event. A user operation hash given as `bundleHash` has no transaction receipt: it is polled until `timeoutMs`, in a slot of `maxBackgroundConfirmations`, and records nothing. - **With `userOpHash` alone**, the client must be a bundler client (one with viem's bundler actions, such as `createBundlerClient`'s or a smart account SDK's); it is polled with `eth_getUserOperationReceipt`. Any other client records nothing and gets no request. No receipt by `timeoutMs` ends the span as `timeout`. - Options as for `watch()`: `chainId` (defaults to the client's chain, or for a bundler client to that of its `client`) and `timeoutMs` (default 120 000 ms). It counts as one of the `maxBackgroundConfirmations`, sends its requests with the client's `request`, not its actions, so a client extended by `withHashspan()` records no confirm span of the bundle transaction, never throws or waits, and `flush()` awaits it. - There is no send span unless you record the call that sends with the same tracker. Revert reasons are recorded only from a bundler's receipt ([ADR 0031](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0031-watch-user-operation.md)). ## Call batches (EIP-5792) - The send span covers handing the batch to the wallet, and records the account, number of calls and batch id. - The confirm span records the outcome (`blockchain.call_batch.status`: `success` for 200, `reverted` for 500, `partially_reverted` for 600), the status code, whether the batch ran atomically, the hashes of the transactions that carried it, and the highest block among its receipts. A status 400 ends with `error.type` `failed`, any other code with `_OTHER`; a wait that accepts a pending status ends without an outcome. No fee is recorded: wallet receipts lack the L1 fee and can be a bundle transaction shared with others. Batch ids must be `0x`-prefixed hex; others are not recorded. - With `experimental_fallback`, viem sends the calls as plain transactions when the wallet lacks `wallet_sendCalls`. Each is confirmed as a transaction linked to the batch's send span, as `watch()` does, so its receipt and fee are recorded whether or not background confirmation is on. Each of these confirmations polls until `confirm.timeoutMs` (120 000 ms by default, also without background mode) and counts toward `maxBackgroundConfirmations`. At most 64 transactions of a batch are confirmed this way ([bounds](https://hashspan.dev/docs/reference/semantic-conventions#bounds)). - `sendCallsSync` records one send span and one confirm span: viem's own `sendCallsSync` runs with the traced `sendCalls` and `waitForCallsStatus`, so an extension applied before this one that replaces `sendCallsSync` itself is not called ([apply it last](https://hashspan.dev/docs/adapters/viem#apply-it-last)). This needs viem 2.45.2 or later: from 2.38.0 to 2.45.1, viem's `sendCallsSync` calls the two actions directly, so the batch is sent and waited for untraced, with its result unchanged. `getCallsStatus` is not traced: polling it yourself records nothing, as with `getTransactionReceipt`. Background confirmation and `watch()` do not cover batches. - See [ADR 0022](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0022-call-batches.md). ## JSON-RPC requests - Spans follow the OpenTelemetry RPC conventions: `rpc.system.name` `jsonrpc`, `rpc.method`, `jsonrpc.protocol.version`, and `server.address` and `server.port` from the transport's URL, plus `blockchain.chain.id`. No parameters or results are recorded, and of the URL only the host and port, since the path often holds an API key. The host is recorded as it is: with a provider that gives each endpoint its own subdomain, it identifies your endpoint. These spans do not pass through the tracker's `redact` hook; to leave the host out, drop `server.address` in a span processor or a collector rule. - A failed request ends with error status and `error.type`: its JSON-RPC error code (also `rpc.response.status_code`) or the error's class name, never the message. - `methods` chooses which methods get a span (default: all); the README's example leaves out receipt polling. `tracerProvider` replaces the global tracer provider. - See [ADR 0019](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0019-json-rpc-spans.md). ### Trace context for the provider - Off unless you pass it. It sends `traceparent`, and `tracestate` when the trace has one, never `baggage`; it does not use the global propagator. The trace id, span id and sampled flag reach your provider. - Use it or an HTTP instrumentation of `fetch` (such as `@opentelemetry/instrumentation-undici`), never both: the instrumentation adds a second `traceparent` header that the hook cannot see. A request that already has a `traceparent` header keeps it. - A request gets a header only inside its JSON-RPC span: wrap the transport in `traceTransport()`, and a method that `methods` leaves out gets none. With `batch: true`, an HTTP request of several JSON-RPC requests gets none, since they can belong to different traces. - Needs viem 2.21.25 or later; with an older viem it sends nothing and logs a `diag` warning. To keep a hook of your own, call this one from it and return its result. - An eRPC setup: [RPC gateways](https://hashspan.dev/docs/integrations#rpc-gateways-erpc). See [ADR 0030](https://github.com/selimaytac/hashspan/blob/@hashspan/viem@1.3.1/docs/adr/0030-trace-context-on-json-rpc-requests.md). ## Known limits The limits of single features; the README lists the [general ones](https://hashspan.dev/docs/adapters/viem#known-limits). - A transaction replaced before viem's wait read it still ends its confirm span with `error.type` `timeout` when the replacement was mined before the last 128 blocks, when it was sent without a `nonce` and the replacement reached the node before hashspan's read of the sent transaction (the node then no longer knows it), when that read fails, and when it was sent elsewhere and confirmed with `watch()` ([replaced transactions](#replaced-transactions)). - Through an EIP-1193 provider (`custom()`, such as a wallet service's), a send without a `nonce` is not read, so a replacement sent right after it ends its confirm span with `error.type` `timeout`; pass `nonce` to have it found. - A send without a `nonce`, and a raw send, cost one `eth_getTransactionByHash` through the sending client, for the replacement search above, unless the client sends through an EIP-1193 provider; pass `nonce` to avoid it. - A wait that resolves with the receipt of another transaction that viem did not report as a replacement, as after a mixed-up RPC response, ends the confirm span with `error.type` `_OTHER` and records none of that receipt's data; the transaction's own outcome is not recorded ([#355](https://github.com/selimaytac/hashspan/issues/355)). - A Tempo multisig transaction sent with a sync form below quorum gets a confirm span without an outcome, unless `followMultisigOperations` is on; its send and confirm spans carry the multisig operation's hash, not the hash of the transaction the relay submits later, and a follow costs up to 60 polls, 3 requests each through a remote relay ([Tempo transactions](#tempo-transactions)). - A Tempo transaction (type `0x76`) with a `calls` list has no top-level `to`: its send span records no recipient. - A preconfirmed receipt whose sealed receipt does not come in time is recorded without fees ([preconfirmed receipts](#preconfirmed-receipts-flashblocks)). - A Celo transaction sent with `sendRawTransaction`, or confirmed with `watch()` without a traced send, records its fee in its fee currency without `blockchain.tx.fee_asset` ([fees paid in a token](#fees-paid-in-a-token)). - With viem older than 2.57.0, concurrent waits on one hash with different counts share viem's first poll, so the recorded count may be the other wait's: a wait of 1 that joins a wait of 5 on the same client resolves with it at depth 5 and records `blockchain.tx.wait.confirmations` `1` ([wevm/viem#5142](https://github.com/wevm/viem/pull/5142)). - With viem older than 2.33.0, viem's `waitForTransactionReceipt` reads the transaction before its receipt even when the transaction is already mined, so background confirmation and `watch()` also send an `eth_getTransactionByHash` for a transaction mined before they start. With viem older than 2.21.58, a user operation that the bundler refuses because its call reverts, for a call with `data` and no `abi`, records `error.type` `ContractFunctionRevertedError`, as viem classifies it there, where newer viem gives `ExecutionRevertedError`. - On a client without a chain, a call is not traced when its `eth_chainId` request fails or has not answered 30 s after the call ended ([clients without a chain](#clients-without-a-chain)). Concepts Short explanations of the ideas on this page: Flashblocks: a receipt before the block exists ERC-4337 user operations: when the transaction isn't yours EIP-5792 call batches: one request, several calls --- # x402 Source: https://hashspan.dev/docs/adapters/x402 Synced from GitHub This page mirrors packages/x402/README.md at @hashspan/x402@1.3.1, the version on npm. Trace the payments your AI agents make over [x402](https://www.x402.org) with OpenTelemetry. With x402, an agent pays for an API call by signing an authorization, and the API's facilitator sends the settling transaction from its own account. Each payment becomes a `payment {chainId}` span inside your agent's trace, with what was paid, to whom, for which resource and whether it settled, and, with a reader, a linked `confirm {chainId}` span with block, gas and fees. ## Install ```sh npm install @hashspan/x402 @x402/core @x402/evm @x402/fetch @opentelemetry/api viem ``` `@x402/evm` and `@x402/fetch` are the payment scheme and HTTP client of the example below; install the ones your setup uses. Requires Node.js 22.3 or later and `@x402/core` 2.13 or later. Bring your own [OpenTelemetry SDK and exporter](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/). ## Usage ```ts import { withHashspan } from '@hashspan/x402'; import { x402Client } from '@x402/core/client'; import { registerExactEvmScheme } from '@x402/evm/exact/client'; import { wrapFetchWithPayment } from '@x402/fetch'; import { createPublicClient, http } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; import { baseSepolia } from 'viem/chains'; const client = new x402Client(); registerExactEvmScheme(client, { signer: privateKeyToAccount(privateKey) }); // Registers hooks on the client; call it once per client. const hashspan = withHashspan(client, { reader: createPublicClient({ chain: baseSepolia, transport: http() }), }); const fetchWithPayment = wrapFetchWithPayment(fetch, client); await fetchWithPayment('https://api.example.com/weather'); // payment span, then confirm span // Before a short-lived process exits: await hashspan.flush(); ``` Pass the `x402Client` itself: `@x402/fetch`, `@x402/axios` and `@x402/mcp` all pay through it, so one call covers them. Helpers that build their own client from a config, such as `wrapFetchWithPaymentFromConfig`, hide it: create it with `x402Client.fromConfig(config)` and pass it to both instead. An `x402MCPClient` exposes its client as `paymentClient`. The adapter only registers hooks; it replaces no method, and its hooks never throw and never change a payment. ### Hook order and spend controls Call `withHashspan` before registering hooks of your own (`onBeforePaymentCreation`, `onAfterPaymentCreation`, `onPaymentCreationFailure`, `onPaymentResponse`): the client runs them in the order they were registered, and stops at the first that recovers a failed payment or a response, so a hook registered earlier can keep hashspan from seeing the outcome, which then ends as `timeout`. Registering a scheme (`registerExactEvmScheme`), a policy or an extension is not a hook and may come first, as in the example: the hooks of schemes and extensions run after the client's own. The client's spend controls (`@x402/core` 2.23 and later) refuse a payment before it is signed, and a refused payment is not traced ([known limits](#known-limits)). They accept only the scheme's default assets, such as USDC: to pay in another token, add it with `client.setSpendControls({ allowedAssets: [{ network, asset }] })`. ### Options `withHashspan(client, options)` accepts the [`@hashspan/core` options](https://hashspan.dev/docs/adapters/core#options) (address mode, agent identity, redaction hook, ...), `decodeRevertReason` as in `@hashspan/viem` but off by default, `maxBackgroundConfirmations` as in `@hashspan/viem` (it limits the confirmations through the reader; a payment whose confirmation is not started gets no `verified`), `tracker`, `reader` (a viem public client, or a function returning one for a chain id) and `confirmTimeoutMs` (default 120 000 ms). Without a reader, only payment spans are recorded. Give it the same `tracker` as `@hashspan/viem` or `@hashspan/cdp` to share one tracker between adapters; with `tracker`, the core options are not used. Options are read once, from the object's own enumerable properties: options it inherits through a prototype, such as the getters of a class instance, are ignored ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/docs/adr/0025-untrusted-input.md)). ### Verified settlements The settlement and its transaction hash come from the server you pay. With a reader, the payment span records `blockchain.payment.verified`: `true` when the reported transaction's receipt carries your payment, `false` when it does not. For an `exact` payment authorized with EIP-3009, that means the token emitted `AuthorizationUsed` with your nonce and `Transfer` from you to `payTo` of exactly the amount. For an `exact` payment authorized with Permit2, the transaction was sent to the x402 proxy you authorized, which emitted its settlement event, and the token emitted `Transfer` from you to `payTo` of exactly the amount; for `upto`, the transaction was also sent by the facilitator your authorization names, and the transfer is of more than nothing, at most your maximum and, when the settlement reports an amount, exactly that amount. For both, the reader also fetches the mined transaction: its input must pass your Permit2 nonce and your address as the owner, so the transaction of an earlier payment is `false`, whichever client or process verified it. All of it is checked from your own payment, not from the settlement. The attribute is absent when no check was possible, for example: no reader, no receipt, a reverted one, a transaction that cannot be read, a settlement on another network, an authorization that does not match the requirements, or another scheme (such as `batch-settlement`). With a reader, the payment span is exported once the receipt is checked; its end time stays when the response came. The revert reason of a reverted settlement, which would be text from a contract the server chooses, is only recorded with `decodeRevertReason: true`. See [ADR 0017](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/docs/adr/0017-x402-payment-verification.md). ### Shutting down `flush({ timeoutMs })` (default 10 000 ms) waits for payments still waiting for their response, then for confirmations through the reader, and ends what is left as `timeout` (a payment already settled and waiting for its receipt check ends as settled, without `verified`). Call it before a short-lived process exits. ## Recorded | Outcome | Payment span | |---|---| | Settled | `blockchain.payment.status` `settled` and the transaction hash | | Settled with `batch-settlement`: the request was charged against the payer's channel | `settled` and the charge as `blockchain.payment.settled_amount`, with no hash ([known limits](#known-limits)) | | Settlement pending (`settlement_pending`) | `pending` and the hash; the confirm span resolves it | | Settlement failed | `failed`, error status, the facilitator's `errorReason` as `error.type` | | Response without a settlement: the facilitator refused the payment before settling it (e.g. the payer's balance is too low), or the API failed (e.g. answered 500) before settling; see [payment, request and task outcomes](#payment-request-and-task-outcomes) | error status, `error.type` `no_settlement` | | No response: the paid request failed on the network, `@x402/axios` got a status other than 2xx or 402, or no response came before the authorization expired | error status, `error.type` `timeout`, once the requirements' `maxTimeoutSeconds` plus 30 s passed (at least 30 s, at most 1 h; 300 s when the requirements give none), when more than 1000 payments are open (the oldest first), or on `flush()` | | Creating the payment failed (e.g. signing) | error status, the error's class name as `error.type` | Every payment span records the payer, recipient (`payTo`), asset, the amount the payer signed for and scheme, and the origin of the resource URL, e.g. `https://api.example.com`: paths of paid APIs often carry user or account identifiers. The core option `paymentResource: 'path'` records the path too (never the query string, fragment or user info), and `'off'` nothing; see [docs/semconv.md](https://hashspan.dev/docs/reference/semantic-conventions). When the settlement reports the amount it settled, it is recorded as `blockchain.payment.settled_amount`, as reported: with the `upto` scheme, it can be less than the amount signed for. With `batch-settlement`, it is the request's charge (`extra.chargedAmount`), which can also be less than the price, and never the deposit into the channel. ## Payment, request and task outcomes The payment span records the payment, not the paid request or your tool: the x402 client hooks never see the response's status. Whether the tool succeeded is on your tool span (or your framework's), the parent of the payment and confirm spans. Together they tell these cases apart, checked with `@x402/core` 2.28 on Anvil: | What happened | Payment spans under the tool span | Tool span | |---|---|---| | The API failed before settling (by default a server settles after its handler, and not when the handler failed) | one, `no_settlement` | error, if the tool fails on the status | | The API settled, then failed: a server that settles before its handler (`paymentFlow` `upfront`) answered 500 | one, `settled` and `verified` | error, if the tool fails on the status; otherwise nothing records the failure | | The request succeeded, the tool failed on the response | one, `settled` | error | | The tool retried and paid again | two, `settled`, with two settlement transactions | as the retry ended | | The tool retried after a request that was not settled | two: one `no_settlement`, one `settled` | as the retry ended | | The settlement transaction does not carry the payment | `settled`, `verified` `false` | | | The settlement's outcome is unknown | `pending`, and the linked confirm span ends with `error.type` `timeout` | | Count only `settled` payments as paid; a `pending` one is paid once its confirm span records the receipt. What a tool call or task spent is the `amount` of each settled payment (the `settled_amount` for `upto` and `batch-settlement`), per `blockchain.payment.asset`: amounts of different assets do not add up. The settlement's gas was paid by the facilitator, not by the agent. Spans are not an accounting record: a trace that a sampler dropped, or that was not exported, misses its payments. ## Try it locally A payment can be made and traced on a local [Anvil](https://getfoundry.sh) chain, with no public facilitator and no funds. The [settlement integration test](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/packages/x402/test/settlement.int.test.ts) is the complete recipe: a vitest file that uses the repository's test helpers, to read rather than to copy as is. Take from it: - The token: an EIP-3009 token that you deploy and mint to the paying account, such as the test's [`TestUsd.sol`](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/packages/x402/test/token/TestUsd.sol) (anyone can mint it; for tests only). The `beforeAll` block deploys and mints it. - The paid API, `paidApi()`: a facilitator in your process (`x402Facilitator` from `@x402/core/facilitator`, with `registerExactEvmScheme` from `@x402/evm/exact/facilitator` and `toFacilitatorEvmSigner` from `@x402/evm` for an Anvil account with ether, which sends the settling transaction) and a resource server (`x402ResourceServer` and `x402HTTPResourceServer` from `@x402/core/server`, with `registerExactEvmScheme` from `@x402/evm/exact/server`), served as a function that stands in for `fetch`, so no port is opened. The route's `price` names the token as `asset`, with its EIP-712 domain (`name` and `version`) in `extra`. - The paying client, `agentClient()`: as in [usage](#usage), with a reader on Anvil (`foundry` from `viem/chains`) and the token in its [spend controls](#hook-order-and-spend-controls). The paying account only signs and needs no ether. To run your version as a script, see the root [quick start](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/README.md#quick-start) for `"type": "module"` and tsx. ## Known limits - Only the paying client is traced: resource servers and facilitators are not ([#164](https://github.com/selimaytac/hashspan/issues/164)), and no `send` span is recorded, since the facilitator sends the transaction ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/docs/adr/0013-x402-payments.md)). - Not traced: x402 v1 payments (`registerExactEvmScheme` registers v1 networks too) and networks other than `eip155`, with a `diag` warning once per version or network; payments that client policies or spend controls refuse, which happens before any hook runs. - A hook registered before hashspan's can keep it from seeing the outcome, which then ends as `timeout`; if another `onPaymentCreationFailure` hook recovers a failed payment, its span still records the failure ([usage](#usage)). - `blockchain.payment.verified` needs a `reader` and covers `exact` payments authorized with EIP-3009 or Permit2 and `upto` with Permit2; other schemes, such as `batch-settlement`, are not checked ([usage](#usage)). - With `batch-settlement`, `settled` means the request was charged against your channel: the server claims the charges and settles them to `payTo` later, in batches, and refunds the rest of the deposit. Those transactions, and the deposit the first request makes, are sent on the server side and recorded in no payment span; a payment span has no hash, no confirm span and no `verified` ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/x402@1.3.1/docs/adr/0013-x402-payments.md)). - The payment span records the payment, not the paid request's status or your tool's outcome ([payment, request and task outcomes](#payment-request-and-task-outcomes)). - The limits of `@hashspan/viem`'s confirmations and of the core apply to the confirm span ([`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem#known-limits), [`@hashspan/core`](https://hashspan.dev/docs/adapters/core#known-limits)). ## License Apache-2.0 Concepts Short explanations of the ideas on this page: Span links: connecting spans that aren't parent and child Span attributes: putting the tx hash on the span Span status codes: Unset, Error and Ok --- # Architecture Source: https://hashspan.dev/docs/architecture Synced from GitHub This page mirrors docs/architecture.md at main. Figure: The agent's tool span, viem client, CDP SDK, x402 client and mppx client feed hashspan's viem, CDP, x402 and MPP adapters, which pass hashes and metadata to @hashspan/core; core records through @opentelemetry/api into the user's OpenTelemetry SDK, which exports over OTLP to Jaeger, Tempo, Langfuse or another backend. The viem adapter reads receipts and replays reverts with read-only RPC calls to the chain. | Component | Purpose | Path | |---|---|---| | core | Lifecycle tracker: `send`/`confirm` spans of transactions, user operations and EIP-5792 call batches, `payment` spans, links, one confirm span per transaction, replaced transactions, fees, privacy modes, and send, confirmation and fee histograms (ADR 0020). No network calls: adapters pass it receipts | `packages/core` | | viem adapter | Hooks `sendTransaction` / `writeContract` / `waitForTransactionReceipt`, `sendUserOperation` / `waitForUserOperationReceipt` of a bundler client, and `sendCalls` / `waitForCallsStatus` (ADR 0022), via `client.extend()`; background confirmation, `watch()` for transactions and `watchUserOperation()` for user operations sent elsewhere, revert reason decoding by replay, `flush()`; `traceTransport()` records JSON-RPC requests as spans | `packages/viem` | | cdp adapter | Wraps a Coinbase CDP client in place: sends of server accounts become send spans, user operations of smart accounts user operation spans (ADR 0021); confirmations through a viem reader and `@hashspan/viem`'s `watch()` | `packages/cdp` | | x402 adapter | Registers hooks on an `x402Client`: each payment becomes a `payment` span, as the facilitator, not the agent, sends the settling transaction; confirmations through a viem reader and `@hashspan/viem`'s `watch()`, which also check that the settlement carries the payment (`blockchain.payment.verified`, ADR 0017) | `packages/x402` | | mpp adapter | Registers observers on an mppx client: each Tempo charge becomes a `payment` span, as the payer sends (push) or signs (pull) the settling transaction itself; confirmations through a viem reader and `@hashspan/viem`'s `watch()`, which also check that the settlement carries the payment (ADR 0029) | `packages/mpp` | | examples | Runnable agent integrations | `examples/` | | lab | Local Jaeger (Docker) + project-local Anvil; Prometheus and Grafana with the [dashboards](https://github.com/selimaytac/hashspan/blob/main/dashboards/README.md) | `docker/`, `dashboards/`, `scripts/`, `Makefile` | Design decisions: [docs/adr](https://github.com/selimaytac/hashspan/blob/main/docs/adr/). Attribute schema: [docs/semconv.md](https://hashspan.dev/docs/reference/semantic-conventions). ## Principles - **Library, not a service.** Peer dependencies are `@opentelemetry/api` and, for an adapter, the library it instruments, per the rule in [AGENTS.md](https://github.com/selimaytac/hashspan/blob/main/AGENTS.md#conventions); users bring their own SDK and exporter. - **Never on the critical path.** Instrumentation failures are swallowed and never change the result of a transaction call. - **Off the call path.** Nothing the telemetry needs is awaited before the call it traces: no network request, no promise. What is known up front (a client with a chain) is recorded synchronously when the call starts; what is not is resolved alongside the call and the span is recorded after the fact ([ADR 0009](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0009-telemetry-off-the-call-path.md)). - **Read-only.** The library never signs or broadcasts transactions. The core makes no network calls; adapters read receipts and replay reverted transactions with read-only requests. ## JSON-RPC requests hashspan adds Agents and services often run against rate-limited endpoints. These are the requests the adapters make in addition to the traced calls, per case; the integration tests named here count them against the same calls without hashspan, so a change that adds requests fails them. How often a wait polls depends on when blocks arrive, so polling is counted by method, not by number. | Case | Requests added | Option that changes them | |---|---|---| | A send that passes `nonce` and its wait, on a client with a chain (viem; also a fee paid in a token, whose asset comes from the call or the receipt) | none | | | A send without `nonce`, or a raw send (viem), through a client whose transport is not an EIP-1193 provider (`custom()`) | one `eth_getTransactionByHash` per send through the sending client, off the call's path, for the nonce and sender a replacement search needs ([ADR 0008 amendment](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0008-replaced-transactions.md#amendment-2026-10-18-proposed-a-replacement-viem-did-not-report)) | pass `nonce` | | A wait that timed out, for a transaction hashspan sent and knows the nonce of (viem) | at most 13 requests to find a replacement viem did not report: the receipt, the block number, the sender's nonce at up to 9 blocks, one block with its transactions and the replacement's receipt | `checkReplacement: false` | | A send on a client without a chain (viem) | one `eth_chainId` per send | give the client a chain | | Background confirmation and `watch()` (viem; cdp, x402 and mpp confirm through `watch()`) | the receipt polling of one `waitForTransactionReceipt` per transaction | `confirm`, `maxBackgroundConfirmations`, `timeoutMs` | | `watchUserOperation()` (viem) | polling of `eth_getTransactionReceipt` for a bundle hash, or of `eth_getUserOperationReceipt` for a user operation hash, once per polling interval of the client until the receipt or `timeoutMs` ([ADR 0031](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0031-watch-user-operation.md)) | `maxBackgroundConfirmations`, `timeoutMs` | | A reverted transaction (viem, cdp) | one `eth_getTransactionByHash` and one `eth_call` to replay it; one more `eth_call` when the contract was created in the same block ([ADR 0005](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0005-revert-reason-replay.md)) | `decodeRevertReason: false` | | A wait with `confirmations` above 1 (viem) | one `eth_getTransactionReceipt` after the wait resolved; one `eth_getBlockByNumber` more only when that receipt is missing or in another block ([ADR 0026](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0026-receipt-after-several-confirmations.md)) | | | A receipt that carries `operatorFeeScalar` or `operatorFeeConstant` (OP Stack after Isthmus, on a chain that charges an operator fee; viem, cdp with a reader) | one `eth_call` to the GasPriceOracle's `getOperatorFee(gasUsed)` at the receipt's block, per transaction ([OP Stack operator fee](https://hashspan.dev/docs/adapters/viem#op-stack-operator-fee)); none for a receipt without the fields | | | A preconfirmed receipt (flashblocks) | `eth_getTransactionReceipt` once per polling interval until the sealed receipt, at most 30 s ([Preconfirmed receipts](https://hashspan.dev/docs/adapters/viem#preconfirmed-receipts-flashblocks)) | | | A sync send that returns a pending receipt (a Tempo multisig relay below quorum; viem) | none: the confirm span ends without an outcome | | | The same, following the multisig operation (viem) | `eth_getTransactionReceipt` of the operation's hash, spread over the option's timeout, at most 60 per pending receipt (3 requests per poll through a remote relay: the default node's receipt, `multisig_getOperation`, the relay's receipt) ([Tempo transactions](https://hashspan.dev/docs/adapters/viem-reference#tempo-transactions)) | `followMultisigOperations` (off by default), `maxBackgroundConfirmations` | | A user operation (viem bundler client) | none | | | A user operation of a CDP smart account, with a reader | `eth_getTransactionReceipt` of the bundle transaction, polled until found or `confirmTimeoutMs` ([ADR 0021](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0021-user-operations.md)) | no `reader` | | An x402 payment with a reader | the settling transaction's receipt; for Permit2 also its transaction, once | no `reader` | | An MPP payment with a reader | the settling transaction's receipt, also when the payer's traced client already confirmed it (push on a shared tracker), for the check | no `reader` | Tests: `packages/viem/test/request-count.int.test.ts`, the request-count tests of `packages/viem/test/user-operation.int.test.ts`, `packages/x402/test/settlement.int.test.ts` and `packages/x402/test/permit2-settlement.int.test.ts`. Concepts Short explanations of the ideas on this page: Traces and spans: the anatomy of one agent run Span links: connecting spans that aren't parent and child Transaction lifecycle: send, mempool, confirm --- # Tracing backends Source: https://hashspan.dev/docs/backends Synced from GitHub This page mirrors docs/backends.md at main. hashspan emits standard OpenTelemetry spans and has no exporter of its own: the spans go wherever your OpenTelemetry SDK exports to. Point the SDK's OTLP exporter at your backend with the standard variables: ```sh OTEL_EXPORTER_OTLP_ENDPOINT=https://your-backend:4318 # base URL; the HTTP exporter appends /v1/traces OTEL_EXPORTER_OTLP_HEADERS="key1=value1,key2=value2" # authentication, if the backend needs it ``` The example agent's setup ([`examples/ai-sdk-agent/src/telemetry.ts`](https://github.com/selimaytac/hashspan/blob/main/examples/ai-sdk-agent/src/telemetry.ts)) uses `@opentelemetry/exporter-trace-otlp-proto`, which reads both. Each setup below was run once with `make demo`; the hashspan and backend versions tested are listed with each backend. Keep keys out of shell history and out of the repository, for example with `read -rs KEY` before running the command. ## Console, without a backend To try hashspan without a backend, let the SDK print each span in the terminal instead of exporting it. `NodeSDK` reads the standard variable; with the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md): ```sh OTEL_TRACES_EXPORTER=console npx tsx agent.ts ``` Spans print as they end: `send 31337` and `confirm 31337`, whose `parentSpanContext.spanId` is the `id` of `pay_vendor`, which prints last; the confirm span's `links` hold the send span's id. Checked with `@opentelemetry/sdk-node` 0.222.0. Without a backend and without this variable, the exporter cannot connect and the process exits with an error ([troubleshooting](https://hashspan.dev/docs/troubleshooting#the-process-exits-with-econnrefused-on-port-4318)). ## Jaeger The [local lab](https://hashspan.dev/docs/local-lab) starts Jaeger on `http://localhost:4318`, the exporter's default endpoint, so `make demo` needs no variables. Open `http://localhost:16686`, pick the `treasury-agent` service and expand `withdraw_from_vault`. Without a checkout of this repository, the same Jaeger runs with one command, its UI and OTLP ports bound to localhost only: ```sh docker run --rm -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.22.0 ``` Tested with Jaeger 2.22.0 against hashspan 1.2.0 on 2026-10-19: every send, confirm, payment, call batch and user operation shape renders as one tree, with links shown as references. Jaeger 2.21.0 and older label a trace that holds only a confirm span (its send ran with no parent) as Incomplete, as they count its cross-trace link as a missing parent. When the caller span (for example `execute_tool`) is missing from a trace, Jaeger 2.21.0 nests each confirm span under the span it only links to (send, payment, batch send or the replaced confirm), while 2.22.0 shows them as roots. To search from a terminal, use the v3 query API: in Jaeger 2.22.0 the v1 search `/api/traces?service=` and `/api/services` answer 404 (not checked in 2.21.0), while `/api/traces/{traceId}` still answers in 2.21.0 and 2.22.0. The v3 search requires a start time range in RFC 3339: ```sh curl 'http://localhost:16686/api/v3/traces?query.service_name=my-agent&query.start_time_min=2026-10-05T00:00:00Z&query.start_time_max=2026-10-06T00:00:00Z' ``` ## Grafana Tempo Tested with Tempo 3.0.0 and Grafana 13.2.3, both in Docker, against hashspan 0.5.0 on 2026-10-02. ```sh OTEL_EXPORTER_OTLP_ENDPOINT=http://your-tempo:4318 ``` Tempo accepts OTLP only on the receivers its distributor enables: ```yaml distributor: receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 ``` In Grafana, open Explore with the Tempo data source and search for the trace. The reverted `confirm 31337` span is marked as an error, with every `blockchain.*` attribute in its span attributes. TraceQL can query them, including across the agent's tool call: ```text {span.error.type = "reverted"} | select(span.blockchain.tx.revert.reason, span.blockchain.tx.fee) {span.gen_ai.tool.name = "withdraw_from_vault"} >> {span.blockchain.operation.name = "confirm" && status = error} ``` The second query finds failed confirmations under a given tool. When you call Tempo's search API directly, pass `start` and `end`: without a time range, Tempo 3.0.0 returned no traces for these queries. Grafana Cloud takes the same variables with the OTLP endpoint and Basic authentication header shown in your stack's OpenTelemetry settings; it was not part of this test. ### grafana/otel-lgtm Tested with `grafana/otel-lgtm` 0.37.0 (Grafana 13.2.3, Tempo 3.1.0, Prometheus 3.15.0) against hashspan 1.1.0 on 2026-10-18. Both files in [dashboards/](https://github.com/selimaytac/hashspan/blob/main/dashboards/README.md) import and every query runs, and the `traces.json` TraceQL panels return the expected rows on Tempo 3.1.0 (Tempo's search lags ingest by some seconds, so an immediate query can return none). On the stock image every "in range" panel of `metrics.json` shows 0 for a short-lived process, because its Prometheus runs without `created-timestamp-zero-ingestion`. Start it with the feature enabled and the panels match the sends: ```sh docker run --rm -p 127.0.0.1:3000:3000 -p 127.0.0.1:4318:4318 \ -e PROMETHEUS_EXTRA_ARGS=--enable-feature=created-timestamp-zero-ingestion grafana/otel-lgtm:0.37.0 ``` ## Langfuse Tested with Langfuse 4.49.0, self-hosted with its Docker Compose file, against hashspan 0.9.0 on 2026-10-03. Langfuse Cloud uses the same API. ```sh OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel # US: https://us.cloud.langfuse.com/api/public/otel OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $LANGFUSE_AUTH,x-langfuse-ingestion-version=4" ``` `LANGFUSE_AUTH` is `printf '%s:%s' "$PUBLIC_KEY" "$SECRET_KEY" | base64`, from a project's API keys. For a self-hosted instance the endpoint is `https://your-langfuse/api/public/otel`; plain `http://` (port 3000 by default) sends the keys unencrypted, so keep it to localhost or a private network. Langfuse accepts OTLP over HTTP only (JSON or protobuf), not gRPC. See [Langfuse's OpenTelemetry guide](https://langfuse.com/docs/opentelemetry/get-started). Langfuse maps GenAI spans to its own observation types: `invoke_agent` becomes an agent, `execute_tool` a tool named after the tool, `chat` a generation. `send`, `confirm` and `payment` spans are plain spans; the reverted confirm span has level `ERROR`, and its attributes are in the observation's metadata as `attributes.blockchain.*`. Langfuse stores an attribute string that looks like a number as a number when it can do so without losing precision. A wei value such as `blockchain.tx.fee` can therefore come back as a number on one span (`40616290500000`) and as a string on another (`"1234567890123456789"`); take that into account when filtering or exporting. ### With `LangfuseSpanProcessor` Langfuse's own JavaScript setup, `LangfuseSpanProcessor` from `@langfuse/otel` (as in Langfuse's AI SDK guide), exports only the spans its default filter `isDefaultExportSpan` keeps: Langfuse's own spans, spans with a `gen_ai.*` attribute and spans of known LLM instrumentations. It drops the others without an error (it logs them at debug level only). hashspan's `send`, `confirm`, `payment`, user operation and call batch spans carry `gen_ai.agent.*` attributes only when an agent identity is set, and its JSON-RPC spans never do. Keep every hashspan span by adding its instrumentation scopes (`@hashspan/core` and `@hashspan/viem`) to the filter: ```ts const processor = new LangfuseSpanProcessor({ shouldExportSpan: ({ otelSpan }) => isDefaultExportSpan(otelSpan) || otelSpan.instrumentationScope.name.startsWith('@hashspan/'), }); ``` Setting an agent identity (the `agent` option of `withHashspan()` or `createTxTracker()`, or the `gen_ai.agent.*` Baggage entries) also keeps the transaction and payment spans, but not the JSON-RPC spans. Tested with `@langfuse/otel` 5.11.1. ## Honeycomb Tested with Honeycomb's free plan, US region, against hashspan 0.5.0 on 2026-10-02. ```sh OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # EU: https://api.eu1.honeycomb.io OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=$HONEYCOMB_API_KEY" ``` Use an ingest key from your environment's API keys. The spans land in a dataset named after `service.name` (`treasury-agent` for the example). Attribute types are kept: wei amounts are strings, counts are integers. Query `WHERE blockchain.tx.status = reverted` and open the trace: the failed `confirm 31337` span sits under `execute_tool withdraw_from_vault`, and the link to its `send` span shows on the span. Honeycomb also recognizes the GenAI spans, so the agent span shows its model and token usage. See [Honeycomb's OpenTelemetry docs](https://docs.honeycomb.io/send-data/opentelemetry/). ## Aspire dashboard, OpenObserve, HyperDX and MLflow Each was run once against hashspan 1.1.0 on 2026-10-18: spans, parents, names, status, `exception` events and links arrive complete. The differences are in how the backend stores them: - Aspire dashboard 13.6.1: returns every attribute as a string (`blockchain.chain.id`, `blockchain.block.number`, `blockchain.tx.gas.used`, `server.port`); take that into account when filtering or exporting. - OpenObserve v1.1.1: returns every attribute as a string and flattens the names (`blockchain.tx.status` is the column `blockchain_tx_status`). Its metric names lack `_seconds` (`blockchain_client_send_duration_count`) and carry no `job` label, so [`metrics.json`](https://github.com/selimaytac/hashspan/blob/main/dashboards/metrics.json) applies only in part. - HyperDX 2.41.0 (local mode): returns every attribute as a string (a `Map(String, String)` in ClickHouse). - MLflow 3.17.1: drops the span kind on OTLP ingest ([mlflow/mlflow#26841](https://github.com/mlflow/mlflow/issues/26841)) and has no OTLP metrics endpoint (`/v1/metrics` answers 404). ## Span links A confirm span links to its send or payment span. Where no span was active at the send, the confirm span is in a trace of its own and the link is the only relation between them ([troubleshooting](https://hashspan.dev/docs/troubleshooting#a-confirm-span-with-no-send-span-next-to-it)). Checked with hashspan 0.9.0 on 2026-10-04, unless noted: | Backend | Keeps span links | How it was checked | |---|---|---| | Jaeger 2.22.0 | yes | hashspan 1.2.0, 2026-10-19: `/api/traces/{traceId}` returns each link as a `FOLLOWS_FROM` reference with the link's own trace id | | Jaeger 2.21.0 | yes | both query APIs return the link: `/api/v3/traces` as an OTLP link, `/api/traces` as a `FOLLOWS_FROM` reference | | Grafana Tempo 3.0.0 | yes | `/api/v2/traces/{traceId}` returns the link; how Grafana shows it was not checked | | Langfuse 4.49.0 | no | no link in the observations API (`/api/public/v2/observations`) or in its storage; see [langfuse/langfuse#12337](https://github.com/langfuse/langfuse/issues/12337) | | Honeycomb | yes | the link shows on the span (hashspan 0.5.0, 2026-10-02, [above](#honeycomb)) | | Aspire dashboard 13.6.1 | yes | hashspan 1.1.0, 2026-10-18: the telemetry API returns every link, including the cross-trace link of `watch()` | | Grafana Tempo 3.1.0 (in otel-lgtm) | yes | hashspan 1.1.0, 2026-10-18: same check, no link differs | | OpenObserve v1.1.1 | yes | hashspan 1.1.0, 2026-10-18: the links are a JSON string column | | HyperDX 2.41.0 | yes | hashspan 1.1.0, 2026-10-18: the `Links.*` columns in ClickHouse | | MLflow 3.17.1 | yes | hashspan 1.1.0, 2026-10-18: the link's trace id is rewritten to `tr-` and resolves | ## Span metrics and HTTP instrumentation With HTTP instrumentation next to `traceTransport()` (for example `@opentelemetry/instrumentation-undici`), each JSON-RPC span gets an HTTP client child span. A span-metrics generator such as Tempo's counts both as client calls of the service (`eth_*` and `POST`), so a RED panel that sums client calls counts each request twice. Filter on `rpc.system.name` or on the span name there. The nesting is described under [JSON-RPC spans](https://hashspan.dev/docs/reference/semantic-conventions#spans). ## Common questions Where each question is answered, with the metrics on Prometheus or the spans in a trace backend: | Question | Where | |---|---| | How long do confirmations take, per chain? | The dashboard's latency panels; in PromQL, `histogram_quantile(0.95, sum by (le, blockchain_chain_id) (rate(blockchain_client_confirmation_duration_seconds_bucket[5m])))` | | How many transactions revert, per chain? | The dashboard's outcome panel; in PromQL, `sum by (blockchain_chain_id) (rate(blockchain_client_confirmation_duration_seconds_count{blockchain_tx_status="reverted"}[5m]))` | | Why did they revert, and under which tool or job? | The TraceQL queries under [Grafana Tempo](#grafana-tempo): `blockchain.tx.revert.reason` on the confirm span, the tool or job span above it | | Did a sampler or a process exit lose spans? | Metrics are recorded for every transaction whatever the sampler decides; compare their counts with the traces ([many transactions](https://hashspan.dev/docs/adapters/viem#many-transactions)) | | Did a retried paid request pay twice, or pay without the task succeeding? | [Payment, request and task outcomes](https://hashspan.dev/docs/adapters/x402#payment-request-and-task-outcomes) | | What did a transaction cost? | `blockchain.tx.fee` on the confirm span, in wei, or in the token `blockchain.tx.fee_asset` names; see its row in [semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions#attributes) for what it includes | ## Grafana dashboard for the metrics hashspan's tracker records three histograms ([metrics](https://hashspan.dev/docs/reference/semantic-conventions#metrics)): send duration, confirmation duration and fee. [`dashboards/metrics.json`](https://github.com/selimaytac/hashspan/blob/main/dashboards/metrics.json) is a ready Grafana dashboard for them, on a Prometheus data source: confirmation and send latency percentiles per chain, fees per chain and per service (median, total, average) and their distribution, and the outcomes of sends and of the confirmations of transactions, user operations and call batches, each with its own outcome attribute ([metrics](https://hashspan.dev/docs/reference/semantic-conventions#metrics)), with service and chain id filters. The fee panels show the fees the senders paid in the native currency: samples with `blockchain.fee.payer` (a payment's facilitator, a paymaster, a sponsor) or `blockchain.fee.denomination` (a fee paid in a token) are left out. [`dashboards/traces.json`](https://github.com/selimaytac/hashspan/blob/main/dashboards/traces.json) lists the spans behind them in [Grafana Tempo](#grafana-tempo); [dashboards/](https://github.com/selimaytac/hashspan/blob/main/dashboards/README.md) describes both files and their variables. ![The hashspan dashboard in Grafana after a few runs of the example agent](https://raw.githubusercontent.com/selimaytac/hashspan/main/docs/images/grafana-dashboard.png) In the local lab, `make lab-metrics` starts Prometheus (with its OTLP receiver) and Grafana with `metrics.json` provisioned; `make demo` then sends the example agent's metrics to Prometheus, and the dashboard is on `http://localhost:3000`. Tested with Prometheus 3.15.0 and Grafana 13.2.3. For a ready stack that shows the dashboards with data from many scenarios, see [hashspan-lab](https://github.com/selimaytac/hashspan-lab). To use the dashboard elsewhere, send the metrics over OTLP to a Prometheus-compatible backend and import the file: - Prometheus needs `--web.enable-otlp-receiver` and turns `blockchain.client.send.duration` (unit `s`) into `blockchain_client_send_duration_seconds` and attributes into labels such as `blockchain_chain_id` and `error_type`, as in [`docker/prometheus.yml`](https://github.com/selimaytac/hashspan/blob/main/docker/prometheus.yml). - The counts for the selected range (sends, failures, outcomes, fees, the fee distribution) are the rise of each counter within the range, without the extrapolation of `increase()`, which turns a short run into fractional or inflated counts. They are exact when each process is a series of its own that starts at zero: give each run its own `service.instance.id`, as the example agent does, and, for a process that exports only once before it exits, let Prometheus add a zero sample at each series' start (`--enable-feature=created-timestamp-zero-ingestion`, enabled in the lab). A long-running process restarted under the same `service.instance.id` resets its counters, and the counts then miss the sends before the restart. - The dashboard's data source is a variable at its top: pick your Prometheus data source there. Concepts Short explanations of the ideas on this page: Exporters and OTLP: getting spans to Jaeger or Langfuse Sampling without losing the spans that moved money Resource attributes: who emitted this span? --- # How much does tracing slow down a viem transaction? Source: https://hashspan.dev/docs/benchmarks/tracing-overhead hashspan adds about 0.25 ms at p50 to a viem send plus its receipt wait on a local chain, and about 0.5 ms when viem polls for the receipt. It adds one JSON-RPC request per send, none if you pass the `nonce`, and three for a reverted transaction. - **Latency:** +0.25 ms at p50 (0.21 to 0.31 ms over 3 runs of 500), +0.5 ms when viem polls. The p95 gap is inside run-to-run noise. - **Requests:** +1 `eth_getTransactionByHash`, off your call's path. 0 with `nonce` passed. +3 for a revert, two of them to decode the reason. - **On Base:** a block is 2,000 ms. 0.25 to 0.5 ms is 0.01 to 0.03 % of it. Measured on 2026-10-21 with the published `@hashspan/viem` 1.3.1 and viem 2.57.7, on an Apple M3 Pro. "A transaction" here always means `sendTransaction` plus `waitForTransactionReceipt`, the time your code waits for. ## How much latency does hashspan add to a viem transaction? Figure: Bar chart of a viem send plus receipt on a local Anvil chain, 3 runs of 500. At p50: plain viem 2.48 ms, hashspan with a no-op exporter 2.75 ms (+0.28), hashspan with batch OTLP export 2.72 ms (+0.25), a revert 1.06 ms plain and 1.33 ms traced (+0.26). At p95: 4.91, 5.47, 5.54, 2.85 and 3.46 ms. The p95 gaps overlap run-to-run noise of -0.3 to +1.0 ms. Each value is the mean of the 3 per-run percentiles, in milliseconds, with the receipt already mined when the wait starts (ready mode, below). | Scenario | p50 | p95 | |---|---|---| | plain viem | 2.48 | 4.91 | | plain viem again (A/A) | 2.49 | 5.18 | | hashspan, no-op exporter | 2.75 | 5.47 | | hashspan, batch + OTLP | 2.72 | 5.54 | | plain, `nonce` passed | 2.49 | 5.22 | | hashspan, `nonce` passed | 2.66 | 5.31 | | revert, plain | 1.06 | 2.85 | | revert, hashspan | 1.33 | 3.46 | The gap to plain viem, mean and the min to max over the 3 runs: | Gap | Δ p50, ms | Δ p95, ms | |---|---|---| | A/A noise | 0.01 (−0.02 to 0.05) | 0.27 (−0.32 to 1.03) | | no-op exporter | 0.28 (0.25 to 0.30) | 0.55 (0.10 to 1.12) | | batch + OTLP | 0.24 (0.21 to 0.27) | 0.62 (−0.04 to 1.63) | | `nonce` passed | 0.17 (0.15 to 0.20) | 0.09 (−0.24 to 0.41) | | revert | 0.26 (0.21 to 0.31) | 0.61 (0.13 to 1.49) | With batch export, the p50 gap splits into +0.13 ms on the send and +0.11 ms on the wait. Passing `nonce` drops the send part to +0.06 ms. In ready mode, batch export to an OTLP endpoint cost no more at p50 than the no-op exporter: the `BatchSpanProcessor` exports off the call's path. ## Is the difference real or noise? The p50 gap is real. Every scenario ran next to a second, identical plain viem scenario, an A/A control. Their p50s differed by −0.02 to +0.05 ms, while every traced p50 gap was 0.15 ms or more, in every run. The tails are not resolved. p95 rose in 8 of the 9 runs of the no-op, batch and revert scenarios, by about 0.6 ms on average. But plain viem against itself moved by up to 1.0 ms at p95, and 2.6 ms at p99. This run can't separate a small tail cost from noise, in either direction. ## How many extra JSON-RPC requests does hashspan make? Figure: JSON-RPC requests per transaction, counted at the transport. A transfer where viem picks the nonce: 3 plain calls (eth_fillTransaction, eth_sendRawTransaction, eth_getTransactionReceipt) plus 1 eth_getTransactionByHash from tracing. A transfer with the nonce passed: no extra call. A reverted transaction: 3 extra, two eth_getTransactionByHash and one eth_call. Same counts in all 3 runs. | Case | Added | To drop it | |---|---|---| | send, viem picks the nonce | 1 | pass `nonce` | | send, `nonce` passed | 0 | | | reverted transaction | 3 | `decodeRevertReason: false` drops 2 | The `eth_getTransactionByHash` runs off your call's path. It reads the nonce and sender that hashspan needs to spot a replaced transaction. On a reverted transaction, one more lookup and an `eth_call` replay the call to decode the revert reason. The counts were the same in all 3 runs. On a rate-limited endpoint, this matters more than the milliseconds. The [architecture page](https://hashspan.dev/docs/architecture#json-rpc-requests-hashspan-adds) lists the requests added in every other case, such as `watch()` and preconfirmed receipts. ## What changes when viem polls for the receipt? In a normal app, the wait starts right after the send and viem polls. This run did that too (e2e mode, 4 runs of 300, polling every 10 ms): | Gap | Δ p50, ms | Δ p95, ms | |---|---|---| | A/A noise | 0.06 (−0.01 to 0.10) | 0.14 (−0.70 to 1.20) | | no-op exporter | 0.50 (0.44 to 0.63) | 0.65 (−0.24 to 2.04) | | batch + OTLP | 0.56 (0.44 to 0.61) | 0.68 (0.12 to 1.07) | | `nonce` passed | 0.35 (0.27 to 0.46) | 0.56 (0.18 to 1.14) | | revert | 0.51 (0.47 to 0.58) | 0.81 (0.40 to 1.11) | Plain viem took 14.48 ms at p50 here. The gap is about twice the ready-mode gap, also with `nonce` passed. Part of it is likely the extra lookup sharing one single-threaded local node with the polling. This run doesn't isolate the rest. ## How much memory does hashspan keep per transaction? | Gap | Heap kept per transaction | |---|---| | no-op exporter | 1.08 KB (0.95 to 1.31) | | batch + OTLP | 1.37 KB (1.16 to 1.56) | | revert | 2.12 KB (1.60 to 2.39) | That's what the tracker stores to link a confirm span to its send. By default it keeps each send for 10 minutes and at most 10,000 sends ([`linkTtlMs` and `maxTrackedTransactions`](https://hashspan.dev/docs/adapters/core#options)), so it is bounded. ## What does 0.25 ms mean on Base? Base builds a block every 2 seconds, and [Flashblocks](https://hashspan.dev/learn/flashblocks-preconfirmations) cut that to 200 ms preconfirmations. Base's docs: "This reduces effective block times from 2 seconds to 200 milliseconds through preconfirmations" ([transaction ordering](https://docs.base.org/base-chain/network-information/transaction-ordering)). So 0.25 to 0.5 ms is 0.01 to 0.03 % of a Base block, and 0.1 to 0.3 % of a flashblock. The +10 % you can compute from the table above is large only because a local transaction takes 2.5 ms. On a real network, the RPC round trip and the block time come on top. hashspan's own work stays in your process; only its extra request goes over the network. ## How was it measured? Every scenario sends a 1 wei transfer from a local account (viem signs, then calls `eth_sendRawTransaction`) and waits with `waitForTransactionReceipt`. Traced scenarios extend both clients with one `withHashspan({ tracerProvider, meterProvider })`. | Scenario | Telemetry | |---|---| | plain viem, twice (A/A) | none | | hashspan, no-op exporter | `SimpleSpanProcessor` to an exporter that drops spans: the instrumentation alone | | hashspan, batch + OTLP | `BatchSpanProcessor` and OTLP/HTTP traces, plus OTLP metrics every 5 s, to a sink in another process | | `nonce` passed | plain and batch + OTLP, with `nonce` set by the caller | | revert | a call to a contract that reverts with `Error("nope")`, plain and batch + OTLP | - **Order:** each round runs one operation of every scenario, in a new random order. Each scenario has its own dev account. - **Size:** 30 warm-up rounds, then 500 measured rounds in ready mode and 300 in e2e mode. 3 ready runs and 4 e2e runs, each with a fresh Anvil and a fresh Node process. - **Ready mode:** after the send, the benchmark waits, untimed, until Anvil has the receipt. The timed wait is then one receipt read plus hashspan's work, with no polling sleep in it. Anvil mines about 5 ms after the send, so without this viem's first read usually misses. - **E2E mode:** the wait starts right after the send, and viem polls every 10 ms. - **Settling:** after every operation, outside the timer, the benchmark calls `flush()` and waits for in-flight requests. One scenario's background work can't land in another's timing. - **Proof the tracing ran:** every traced run exported exactly 2 spans per operation, warm-up included (1,060 of 1,060 per ready run). Every revert's confirm span carried `blockchain.tx.revert.reason` = `nope`. - **Memory:** a separate pass, one scenario per fresh `node --expose-gc` process: 30 warm-up operations, `gc()`, 500 operations, `gc()`, and the `heapUsed` difference, 3 times. Traced minus plain. Timing uses `process.hrtime.bigint()` around the send and the wait. JSON-RPC requests are counted by a wrapper around viem's `http()` transport. ## What machine and versions? | | | |---|---| | Date | 2026-10-21 | | Machine | Apple M3 Pro, 12 cores, 18 GiB RAM, macOS 26.7.1 | | Node.js | 22.14.0 | | viem | 2.57.7 | | hashspan | `@hashspan/viem` 1.3.1, `@hashspan/core` 1.3.1, from npm | | OpenTelemetry | `@opentelemetry/api` 1.9.1, `sdk-trace-base` 2.12.0, `sdk-metrics` 2.12.0, OTLP/HTTP exporters 0.223.0 | | Chain | Anvil 1.8.5, automine, chain id 31337 | ## What are the limits of these numbers? - **A local chain.** A receipt read from Anvil takes about 0.24 ms. A real endpoint answers in tens of ms or more, so the relative overhead there is far smaller, and each extra request costs more. - **One machine.** An M3 Pro is fast. A slower machine probably shows a larger absolute gap; it wasn't measured. - **A local OTLP sink.** It answers at once. A remote collector adds network work to the exporter, but in the batch processor, off the call's path. - **One transaction at a time.** There is no concurrency here, so this says nothing about throughput under load. - **One discarded attempt.** The first e2e attempt stopped when one receipt wait hit viem's default 180 s timeout. Its data was never written. The reruns used a 30 s timeout and had no failure, nor did any ready run. ## What doesn't this benchmark measure? This run covers sends you wait for with `waitForTransactionReceipt`. It doesn't cover background confirmation, `watch()`, preconfirmed receipts, the OP Stack operator fee lookup, user operations or payments. Each makes its own requests, listed on the [architecture page](https://hashspan.dev/docs/architecture#json-rpc-requests-hashspan-adds). ## How do I run it myself? This is a reduced version of the benchmark script behind the numbers above: plain viem twice and hashspan with a no-op exporter, in a new random order every round, after 30 warm-up rounds. viem polls for the receipt, so compare its gap with the polling table. Start Anvil with `anvil`, then: ```sh npm i viem@2.57.7 @hashspan/viem@1.3.1 @opentelemetry/api@1.9.1 @opentelemetry/sdk-trace-base@2.12.0 node bench.mjs 500 ``` ```js title="bench.mjs" // Plain viem vs viem + hashspan on a local Anvil: send + receipt, p50 and p95. import { createPublicClient, createWalletClient, http } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; import { BasicTracerProvider, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; const url = 'http://127.0.0.1:8545'; const keys = [ // Anvil's default dev accounts 1-3, local chain only '0x59c6995e998f97a5a0044966f0945389dc9e86dae88c7a8412f4603b6b78690d', '0x5de4111afa1a4b94908f83103eb1f1706367c2e68ca870fc3fb9a804cdab365a', '0x7c852118294e51e653712a81e05800f419141751be58f605c371e15141b007a6', ]; const dropAll = { export: (_spans, done) => done({ code: 0 }), shutdown: async () => {} }; function scenario(name, key, traced) { let wallet = createWalletClient({ account: privateKeyToAccount(key), chain: foundry, transport: http(url) }); let reader = createPublicClient({ chain: foundry, transport: http(url), pollingInterval: 10 }); const hs = traced && withHashspan({ tracerProvider: new BasicTracerProvider({ spanProcessors: [new SimpleSpanProcessor(dropAll)] }) }); if (hs) { wallet = wallet.extend(hs); reader = reader.extend(hs); } const ms = []; return { name, ms, async op() { const t0 = performance.now(); const hash = await wallet.sendTransaction({ to: '0x70997970C51812dc3A010C7d01b50e0d17dc79C8', value: 1n }); await reader.waitForTransactionReceipt({ hash }); ms.push(performance.now() - t0); if (hs) await hs.flush(); // settle off-path work outside the timer } }; } const all = [scenario('plain', keys[0], false), scenario('plain again', keys[1], false), scenario('hashspan', keys[2], true)]; const [warmup, rounds] = [30, Number(process.argv[2] ?? 500)]; for (let r = 0; r < warmup + rounds; r++) { const order = all.map((s) => [Math.random(), s]).sort((a, b) => a[0] - b[0]); for (const [, s] of order) await s.op(); // a new random order every round if (r === warmup - 1) all.forEach((s) => (s.ms.length = 0)); } const q = (xs, p) => [...xs].sort((a, b) => a - b)[Math.floor((xs.length - 1) * p)]; for (const s of all) console.log(s.name.padEnd(12), 'p50', q(s.ms, 0.5).toFixed(2), 'ms p95', q(s.ms, 0.95).toFixed(2), 'ms'); ``` Run it a few times. If "plain" and "plain again" differ by as much as plain and hashspan do, your machine is too noisy for the comparison: close other apps and run more rounds. ## FAQ ### Does OpenTelemetry tracing slow down viem transactions? A little. On a local Anvil chain, @hashspan/viem 1.3.1 added about 0.25 ms at p50 to a send plus receipt wait when the receipt was already there, and about 0.5 ms when viem had to poll for it. A Base block is 2,000 ms, so that is 0.01 to 0.03 % of a block. ### Does hashspan add JSON-RPC requests? One eth_getTransactionByHash per send, made off your call's path, so it can link a replacement to the original. None when you pass nonce yourself. A reverted transaction gets 3 in this run: that lookup, plus 2 to replay the call and decode the revert reason, which decodeRevertReason: false turns off. ### Is the p95 or p99 latency worse with tracing? This run can't tell. The p95 gap was about +0.6 ms on average, but plain viem measured against itself moved by up to 1.0 ms at p95 between runs. The tail gaps are inside that noise, so they are neither proven nor ruled out. ### How much memory does hashspan use per transaction? About 1.1 to 1.4 KB of heap per transaction in this run, 2.1 KB for a revert. The tracker keeps each send for linking: 10 minutes and at most 10,000 sends by default (linkTtlMs and maxTrackedTransactions), so memory stays bounded. Links from the answers: [viem adapter options](https://hashspan.dev/docs/adapters/viem#options) (`decodeRevertReason`), [core options](https://hashspan.dev/docs/adapters/core#options) (`linkTtlMs`, `maxTrackedTransactions`), [tracing viem with OpenTelemetry](https://hashspan.dev/docs/viem-opentelemetry). --- # Add hashspan with a coding agent Source: https://hashspan.dev/docs/build-with-ai Your coding agent can add hashspan in one pass. Install the hashspan skill, paste the prompt below, and the agent wires up the send and confirm spans for you. - **Skill:** `npx skills add https://hashspan.dev` puts the step-by-step instructions in your project. - **Prompt:** one copy-paste prompt points the agent at the skill and the span names. - **Rules:** a short snippet in your agent's rules file stops it from inventing attributes later. ## How do I install the hashspan skill? Run this in your project root: ```sh npx skills add https://hashspan.dev ``` It offers three skills: `hashspan`, the setup steps this page is about, and two that only read, `explain-evm-error` and `otel-attribute-names`. Add `--skill hashspan` to take only the setup. The same skill also installs from its repository with `npx skills add selimaytac/hashspan`. [Agent skills](https://hashspan.dev/agents) has every skill with a ready prompt. The installer asks which agents to set up. For Cursor and GitHub Copilot it writes the skill to `.agents/skills/hashspan/SKILL.md`. For Claude Code it goes in `.claude/skills/hashspan`, as a link to that folder when you pick several agents. Commit the folder, so everyone on the project gets the same steps. The [skill file](https://github.com/selimaytac/hashspan/blob/main/skills/hashspan/SKILL.md) is plain Markdown. It walks the agent through six steps: find how the project sends, install, start the OpenTelemetry SDK first, extend every client, flush before exit, and check the traces. The hashspan repository's CI applies it to minimal versions of two of its examples, the AI SDK agent and the payout worker, and checks their spans on a local chain. Your agent doesn't read skills? Point it at the file's URL instead. The prompt below works either way. ## What prompt should I give the agent? Paste this into Claude Code, Cursor, Copilot or any agent that can edit files and run commands: ```text wrap Add hashspan to this project so every on-chain transaction it sends shows up as OpenTelemetry spans. 1. Follow the hashspan skill (in .agents/skills/hashspan/ or .claude/skills/hashspan/, or https://github.com/selimaytac/hashspan/blob/main/skills/hashspan/SKILL.md) step by step. 2. Use the hashspan adapter for each send path: @hashspan/viem for viem clients, @hashspan/cdp for the Coinbase CDP SDK, @hashspan/x402 for x402 payments, @hashspan/mpp for MPP payments made with mppx. 3. Take span and attribute names only from https://hashspan.dev/docs/reference/semantic-conventions.md. Don't hand-create send, confirm or payment spans or blockchain.* attributes: hashspan records them. 4. If the project sends through another library, such as ethers, or on a chain that is not EVM, stop and tell me. 5. Run it only against a local chain with Anvil's test accounts, or a testnet I name. Ask me before anything signs. Don't read .env files, key files or secrets. Then run the typecheck and tests. ``` When it's done, each transaction shows `send {chainId}` and `confirm {chainId}` spans under the span that was active when it was sent. An x402 payment shows `payment {chainId}` and, with a reader, `confirm {chainId}`. An MPP payment shows `payment {chainId}` and, with a reader, a linked `confirm {chainId}`. ## What should go in my agent's rules file? A one-off prompt is forgotten in the next session. A rules file isn't. Add this to the file your agent reads on every task: ```md wrap title="AGENTS.md" ## Tracing on-chain transactions (hashspan) - Transactions are traced with hashspan. Before you add or rename a span attribute, read https://hashspan.dev/docs/reference/semantic-conventions.md. - hashspan records the `send`, `confirm` and `payment` spans, and `traceTransport()` adds one span per JSON-RPC request. Don't hand-create spans with these names or `blockchain.*` attributes. - New viem clients get `.extend(hashspan)`, after other extensions. Smart account SDKs are the exception: build the client with `withHashspan()` before the SDK's actions (step 4 of https://github.com/selimaytac/hashspan/blob/main/skills/hashspan/SKILL.md). A user operation that an SDK or a hosted wallet sent and returned only a hash for is confirmed with `watchUserOperation()`, not `watch()`. CDP and x402 clients are wrapped with @hashspan/cdp and @hashspan/x402 right after they are created, and an mppx client with @hashspan/mpp's `withHashspan(mppx, { reader })` right after `Mppx.create()`. - Run only against a local chain with Anvil's test accounts, or a testnet the user names. Ask before anything signs. Never read `.env` files, key files or secrets. ``` Where that file lives depends on the agent: | Agent | Rules file | |---|---| | Claude Code | `CLAUDE.md`, or `AGENTS.md` when there is no `CLAUDE.md` (v2.1.277 or later) | | Cursor | `AGENTS.md`, or a `.mdc` file in `.cursor/rules` | | Copilot coding agent, CLI, VS Code chat | `AGENTS.md` or `.github/copilot-instructions.md` | | Other Copilot chats (GitHub.com, Visual Studio, JetBrains, Eclipse, Xcode) | `.github/copilot-instructions.md` only | One `AGENTS.md` covers Claude Code, Cursor and the Copilot agents. For the other Copilot chats, put the snippet in `.github/copilot-instructions.md` too. If you also keep a `CLAUDE.md`, Claude Code reads only that, so import `AGENTS.md` from it. ## Where are the machine-readable docs? Every page on this site has a Markdown twin for agents and LLM tools: | What | URL | |---|---| | Index of every page | [`/llms.txt`](https://hashspan.dev/llms.txt) | | All docs in one file | [`/llms-full.txt`](https://hashspan.dev/llms-full.txt) | | One page as Markdown | page URL + `.md`, e.g. [`/docs/quickstart.md`](https://hashspan.dev/docs/quickstart.md) | | Span and attribute names | [`/docs/reference/semantic-conventions.md`](https://hashspan.dev/docs/reference/semantic-conventions.md) | | Docs over the Context7 MCP server | library ID [`/selimaytac/hashspan`](https://context7.com/selimaytac/hashspan) | Each HTML page also points to its Markdown version with ``. Give the agent the `.md` URL, not the HTML page: it's smaller and has no navigation. ## FAQ ### Do I need the skill, or is the prompt enough? The prompt alone works if the agent can fetch URLs: it links the skill file. Installing the skill keeps the steps in your repository, next to your code. Agents that support skills also load it by themselves when a task matches its description. ### Which projects can the agent instrument? TypeScript projects that send EVM transactions or agent payments through viem, the Coinbase CDP SDK, x402 or MPP (mppx), on Node.js 22.3 or later. That covers agents and plain services such as payout workers. Other libraries, such as ethers, have no adapter, and the skill tells the agent to stop there. ### Why tell the agent not to invent attribute names? Backends, dashboards and queries match on exact names like blockchain.tx.status. An attribute an agent makes up looks fine in one trace and breaks every query that expects the real one. The semantic conventions reference lists every name hashspan records. ### Can I check the agent's work by hand? Yes. Compare the setup with the Quickstart, or run the local lab to see what a correct trace looks like. For a viem project, the viem OpenTelemetry page shows the spans each action should produce. When a span is missing, troubleshooting lists the usual causes. Links from the answers: [semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions), [Quickstart](https://hashspan.dev/docs/quickstart), [x402 tutorial](https://hashspan.dev/docs/x402-tutorial) (the same setup by hand, step by step), [local lab](https://hashspan.dev/docs/local-lab), [viem OpenTelemetry instrumentation](https://hashspan.dev/docs/viem-opentelemetry), [troubleshooting](https://hashspan.dev/docs/troubleshooting). --- # Integrations Source: https://hashspan.dev/docs/integrations Synced from GitHub This page mirrors docs/integrations.md at main. Agents and services rarely call viem or the CDP SDK directly: a toolkit, a framework or a wallet service sits in between. hashspan traces the client those libraries send through, so most integrations come down to handing them a traced client. This page collects the setups that were checked; each code block is compiled in CI. When a span is missing, see [troubleshooting](https://hashspan.dev/docs/troubleshooting). ## Support at a glance | Setup | Use | Extra setup | Known limits | Runs in CI | |---|---|---|---|---| | viem wallet and public clients | `@hashspan/viem` [`withHashspan()`](https://hashspan.dev/docs/adapters/viem#usage) | apply it last; one result per agent | a library that calls viem's actions as functions is not traced ([below](#libraries-that-take-a-viem-client)) | `packages/viem` tests | | viem bundler client (ERC-4337) | `@hashspan/viem` | a tracker from `@hashspan/core` 0.8 or later | no background confirmation or `watch()` for user operations; [`watchUserOperation()`](https://hashspan.dev/docs/adapters/viem#user-operations-sent-elsewhere) for one sent elsewhere ([smart accounts](https://hashspan.dev/docs/adapters/viem#smart-accounts-erc-4337)) | `packages/viem` tests, a real bundler in CI only | | permissionless.js smart account client | `@hashspan/viem` | build the client with `withHashspan()` before `smartAccountActions` | applied last, only `sendUserOperation` and `waitForUserOperationReceipt` are traced ([setup](#smart-account-sdks)) | `integrations/`, with a stand-in account | | ZeroDev Kernel account client | `@hashspan/viem` | none | only `sendUserOperation` and `waitForUserOperationReceipt` are traced; `sendTransaction` and `writeContract` are not ([setup](#smart-account-sdks)) | `integrations/`, with a stand-in account | | EIP-5792 wallet (`sendCalls`) | `@hashspan/viem` | none | no fees on the batch; `getCallsStatus` is not traced ([call batches](https://hashspan.dev/docs/adapters/viem#call-batches-eip-5792)) | `packages/viem` tests | | Transactions sent elsewhere | `@hashspan/viem` [`watch()`](https://hashspan.dev/docs/adapters/viem#transactions-sent-elsewhere) | a public client; for a send span, the call recorded with the same tracker | a send span only when you record the call | `packages/viem` tests | | Coinbase CDP SDK | `@hashspan/cdp` | a `reader` for confirm spans | CommonJS needs Node.js 22.12 ([install](https://hashspan.dev/docs/adapters/cdp#install)); see [known limits](https://hashspan.dev/docs/adapters/cdp#known-limits) | `packages/cdp` tests | | x402 client | `@hashspan/x402` | a `reader` for `verified` | x402 v1 and non-`eip155` networks are not traced ([known limits](https://hashspan.dev/docs/adapters/x402#known-limits)) | `packages/x402` tests | | MPP client (mppx) | `@hashspan/mpp` | a `reader` for confirm spans and `verified`; for push, the payer's viem client extended on the same `tracker` ([setup](#machine-payments-protocol-mppx)) | Tempo `charge` only: sessions and paid MCP tools are not traced ([known limits](https://hashspan.dev/docs/adapters/mpp#known-limits)) | `packages/mpp` tests, on Anvil in Tempo mode | | AgentKit `ViemWalletProvider` | `@hashspan/viem` | background confirmation | TypeScript needs a cast ([setup](#viemwalletprovider)) | `integrations/` | | AgentKit `CdpEvmWalletProvider`, `CdpSmartWalletProvider` | `@hashspan/cdp` on `getClient()` | the provider's public client as `reader` | `configureWithWallet()` itself is not run in CI ([setup](#cdpevmwalletprovider)) | `integrations/` | | Other AgentKit wallet providers | `@hashspan/viem` `watch()` | the returned hash | a send span only when you record the call ([setup](#other-wallet-providers)) | the pattern in `packages/viem` tests | | GOAT | `@hashspan/viem` | background confirmation | no longer maintained ([setup](#goat-sdk)) | `integrations/` | | Mastra | `@hashspan/viem` | Mastra's OpenTelemetry bridge | the bridge is experimental ([setup](#agent-frameworks)) | `integrations/` | | Genkit | `@hashspan/viem` | `disableGenkitOTelInitialization()`, and your SDK started before Genkit's first span | `generate` starts a trace of its own inside an active span unless `disableOTelRootSpanDetection()` is called; Genkit's Developer UI gets no spans ([setup](#agent-frameworks)) | `integrations/` | | LangChain JS, OpenAI Agents SDK (OpenInference) | `@hashspan/viem` | run tools in an active span of your own | tool spans are not active ([setup](#agent-frameworks)) | `integrations/` | | MCP server tools (stdio) | `@hashspan/viem` in the server | the agent's trace context in each request's `_meta`: OpenInference's MCP instrumentation (v1 SDK) or a few lines of your own (v2 SDK) | without it, the server's spans start a trace of their own; pass the `OTEL_*` settings to the server ([setup](#mcp-servers)) | `integrations/` | | ElizaOS `plugin-evm` | none | none | not traced: it creates its own wallet client ([details](#agent-frameworks)) | no | | Coinbase Agentic Wallets (`awal`, payments MCP) | `@hashspan/viem` [`watchUserOperation()`](https://hashspan.dev/docs/adapters/viem#user-operations-sent-elsewhere) on the returned hash, not tested with awal | the smart account's address and a public client of the chain; a bundler client for a user operation hash | sends run as smart account user operations in a Coinbase-hosted page inside awal's own app, so there is no send span. The returned `transactionHash` is the bundle's when awal had one, else the user operation's, and does not say which: as `bundleHash` with the account as `sender`, it records the operation whose `UserOperationEvent` a canonical EntryPoint or the called contract emitted; a user operation hash given this way finds no receipt and records nothing after `timeoutMs` (default 120 s), polling until then. As `userOpHash`, it needs a bundler. `watch()` on it records the bundle transaction | no | | Wallet services with a viem account (Turnkey, Privy) | `@hashspan/viem` | none | none ([setup](#wallet-services)) | as viem clients | | Wallet services with an EIP-1193 provider (Circle, Fireblocks) | `@hashspan/viem` on a `custom()` transport | a public client for receipts when the provider does not answer reads | Circle smart contract accounts not checked ([setup](#wallet-services)) | the pattern in `packages/viem` tests | | Wallet services that send through their API | `@hashspan/viem` `watch()` | a public client; for a send span, the API call recorded with the same tracker | none ([setup](#wallet-services)) | the pattern in `packages/viem` tests | | Hardhat 3 scripts and tests (hardhat-viem) | `@hashspan/viem` on the clients of `viem.getPublicClient()` and `viem.getWalletClients()` | pass them as `client` to `deployContract` and `getContractAt` | a deployment gets a confirm span only; Ignition deployments are not traced ([setup](#hardhat-3)) | `integrations/`, on Hardhat's in-process network | | eRPC gateway in front of the nodes | `@hashspan/viem` with `traceTransport()` | `propagateTraceContext()` on viem's `http()`, or `@opentelemetry/instrumentation-undici`, so requests carry `traceparent` | never both; with undici, two client spans per request; tracing over http needs a `host:port` endpoint ([setup](#rpc-gateways-erpc)) | `integrations/`, eRPC 0.4.0 in front of Anvil | ## Services that are not agents hashspan does not need an agent. In a TypeScript service that sends EVM transactions through viem, the Coinbase CDP SDK or x402, or through a wallet service, such as a payment or payout worker, a wallet backend or a bot, the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md) works as it is: replace the `pay_vendor` span with the span your request handler or job already has, and leave out the `agent` option. The `service.name` of your OpenTelemetry resource (the `serviceName` in the quick start) tells your services apart. Other libraries, such as ethers, have no adapter, and chains other than EVM are not supported. - **A wallet service sends for you:** [wallet services](#wallet-services) shows how each kind of service gets its send and confirm spans. - **A worker that does not wait for receipts:** [background confirmation](https://hashspan.dev/docs/adapters/viem#background-confirmation). - **Latency and fees across many transactions:** the send, confirmation and fee [histograms](https://hashspan.dev/docs/reference/semantic-conventions#metrics), and the limits that matter at volume ([many transactions](https://hashspan.dev/docs/adapters/viem#many-transactions)). - **A short-lived job:** call `flush()` before it exits ([shutting down](https://hashspan.dev/docs/adapters/viem#shutting-down)). - **An example:** [examples/payout-worker](https://github.com/selimaytac/hashspan/blob/main/examples/payout-worker), a worker that pays out from a queue, confirmed in the background, run against Anvil in CI. ## Libraries that take a viem client When a library accepts a viem wallet client, extend the client before you hand it over: - `client.extend(withHashspan())` traces `sendTransaction`, `writeContract` and `sendRawTransaction` as long as the library calls them as methods of that client. A library that calls viem's actions with the client as an argument (`sendTransaction(client, ...)` from `viem/actions`) bypasses the extension for that action; the call is recorded only through the client's own actions viem calls inside it. `writeContract` sends through the client's `sendTransaction` (a send span without the function name); with a local account, `sendTransaction` and `sendTransactionSync` sign and send through the client's `sendRawTransaction` and `sendRawTransactionSync` (without the sender). `sendTransaction` with an address-only account is not traced. - If the library waits for receipts on a client of its own, the confirm span is not recorded by that wait. Use [background confirmation](https://hashspan.dev/docs/adapters/viem#background-confirmation) so hashspan polls for the receipt itself. - Apply `withHashspan()` last ([why](https://hashspan.dev/docs/adapters/viem#apply-it-last)). ## Smart account SDKs Checked with permissionless 0.5.0 and `@zerodev/sdk` 5.5.11. Their smart account clients (`createSmartAccountClient`, `createKernelAccountClient`) are viem bundler clients with actions of the SDK on top: `sendTransaction` and `writeContract` (and permissionless's `sendCalls`) send a user operation, wait for its receipt and return the bundle transaction's hash. Those actions reach the bundler through the client they were added to, so `withHashspan()` applied to the finished client does not see the operation ([smart account SDKs](https://hashspan.dev/docs/adapters/viem#smart-account-sdks)). For user operation spans from `sendTransaction`, `writeContract` and `sendCalls`, build the client that `createSmartAccountClient` builds with viem's `createBundlerClient`, and apply `withHashspan()` before permissionless's `smartAccountActions`. On Anvil, with the OpenTelemetry SDK started as in the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md), where `account` is your permissionless smart account (for example from `toSimpleSmartAccount`) and `bundlerUrl` your bundler's: ```ts import { withHashspan } from '@hashspan/viem'; import { smartAccountActions } from 'permissionless'; import { createPublicClient, http, parseEther } from 'viem'; import { createBundlerClient } from 'viem/account-abstraction'; import { foundry } from 'viem/chains'; // Start the OpenTelemetry SDK before this, as in the quick start. const hashspan = withHashspan(); const client = createPublicClient({ chain: foundry, transport: http('http://127.0.0.1:8545') }); // Anvil // The client createSmartAccountClient({ account, client, bundlerTransport }) builds, with hashspan applied before // permissionless's actions, so that they send through the traced user operation actions. const smartAccountClient = createBundlerClient({ account, client, transport: http(bundlerUrl) }) .extend(hashspan) .extend(smartAccountActions); // A user operation send span, and its confirm span as permissionless waits for the operation's receipt. const hash = await smartAccountClient.sendTransaction({ to: '0x70997970C51812dc3A010C7d01b50e0d17dc79C8', value: parseEther('0.001'), }); // `hash` is the bundle transaction's: a wait for it on a traced client would record the bundle, not the operation. console.log(hash); // Before a short-lived process exits, then shut the SDK down: await hashspan.flush(); ``` - `createBundlerClient` takes the same `paymaster`, `paymasterContext` and `userOperation` options as `createSmartAccountClient`. With a custom `userOperation.prepareUserOperation`, `createSmartAccountClient` adds an action of its own that calls it, which this setup does not. - `createSmartAccountClient(...).extend(withHashspan())` still traces `sendUserOperation` and `waitForUserOperationReceipt` when you call them; its `sendTransaction`, `writeContract` and `sendCalls` are then not traced, with a `diag` warning. The extended client keeps the SDK's types for those actions. - ZeroDev's `createKernelAccountClient` builds its client the same way, and adds `sendTransaction` and `writeContract` with `kernelAccountClientActions()`. Extended with `withHashspan()`, its `sendUserOperation` and `waitForUserOperationReceipt` are traced and its `sendTransaction` and `writeContract` are not. `withHashspan()` applied last, as above, is the supported ZeroDev setup. Building the client by hand with `withHashspan()` before ZeroDev's actions is not a documented setup, because `createKernelAccountClient` also sets ZeroDev's fee estimate and an EIP-7702 authorization step. - For the bundle hash that `sendTransaction` returns on a client extended last, `hashspan.watchUserOperation(client, { bundleHash: hash, sender: account.address })`, with the public client above, records the operation's confirm span, without a send span ([user operations sent elsewhere](https://hashspan.dev/docs/adapters/viem#user-operations-sent-elsewhere)). The `packages/viem` tests check this with a copy of the SDKs' `sendTransaction`, not with the SDKs themselves. - In CI, both SDKs run with a stand-in smart account on a stand-in EntryPoint, since Kernel's and SimpleAccount's contracts are not available to the tests; the user operation spans themselves are also checked through a real bundler ([smart accounts](https://hashspan.dev/docs/adapters/viem#smart-accounts-erc-4337)). ## Coinbase AgentKit Checked with `@coinbase/agentkit` 0.10.4. AgentKit sends through a wallet provider; three of them can be traced without extra code. ### `ViemWalletProvider` The provider sends with the wallet client you give it, and waits for receipts with a public client it creates itself, so the setup needs background confirmation. On Anvil, with the OpenTelemetry SDK started as in the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md), in an ES module as there (`npm pkg set type=module`), since the file uses top-level await: ```ts import { ViemWalletProvider } from '@coinbase/agentkit'; import { withHashspan } from '@hashspan/viem'; import { trace } from '@opentelemetry/api'; import { createWalletClient, http, parseEther } from 'viem'; import { mnemonicToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; // Start the OpenTelemetry SDK before this, as in the quick start. const rpcUrl = 'http://127.0.0.1:8545'; // Anvil const account = mnemonicToAccount('test test test test test test test test test test test junk'); // The provider waits for receipts on a client of its own: background confirmation records them. const hashspan = withHashspan({ confirm: { mode: 'background' } }); const walletClient = createWalletClient({ account, chain: foundry, transport: http(rpcUrl) }).extend( hashspan, ); const walletProvider = new ViemWalletProvider( // AgentKit pins its own viem: see below. walletClient as unknown as ConstructorParameters[0], { rpcUrl }, ); await trace.getTracer('my-agent').startActiveSpan('pay_vendor', async (span) => { try { // The amount in wei, as a decimal string. await walletProvider.nativeTransfer( '0x70997970C51812dc3A010C7d01b50e0d17dc79C8', parseEther('0.01').toString(), ); } finally { span.end(); } }); // Before a short-lived process exits, then shut the SDK down: await hashspan.flush(); ``` `sendTransaction`, `nativeTransfer` and the action providers that send through the wallet provider (such as the ERC-20 actions) then record a `send` and a `confirm` span per transaction; without background confirmation, only the send spans are recorded. CI runs this setup against Anvil with `sendTransaction`, `nativeTransfer` and the ERC-20 `approve` action, both ways, in [`integrations/test/agentkit-viem.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/agentkit-viem.int.test.ts). Two details of the provider apply with or without hashspan: - Its public client connects to the `rpcUrl` of its second argument, else to the `RPC_URL` environment variable, else to the chain's default RPC; never through the wallet client's transport. - AgentKit 0.10.4 depends on viem 2.38.3 exactly. With another viem in the application (2.57 in the test), TypeScript rejects the wallet client in `new ViemWalletProvider(walletClient)`, while the client works at runtime. Cast it as the test does, with or without hashspan: `new ViemWalletProvider(walletClient as unknown as ConstructorParameters[0])`. With the same viem version as AgentKit, no cast is needed. tsx does not check types, so without the cast the error shows only under `tsc`. - `nativeTransfer(to, value)` takes the amount in wei as a decimal string: `'0.01'` throws, while `parseEther('0.01').toString()` sends 0.01 ETH. ### `CdpEvmWalletProvider` The provider sends through its `CdpClient`, which `getClient()` returns. Wrap it with `@hashspan/cdp`, with the provider's public client as the reader: ```ts import { withHashspan } from '@hashspan/cdp'; const walletProvider = await CdpEvmWalletProvider.configureWithWallet(config); // Wraps the provider's CdpClient in place, before its first transaction. const hashspan = withHashspan(walletProvider.getClient(), { reader: walletProvider.getPublicClient(), }); ``` The provider calls `cdp.evm.sendTransaction`, which the wrapper traces; `sendTransaction`, `nativeTransfer` and the action providers that send through the wallet provider then record a `send` and a `confirm` span per transaction. The same setup covers `CdpSmartWalletProvider`, which also has `getClient()` and `getPublicClient()`: it sends ERC-4337 user operations with `cdp.evm.sendUserOperation` and waits with `cdp.evm.waitForUserOperation`, which the wrapper records as user operation spans ([smart accounts](https://hashspan.dev/docs/adapters/cdp#smart-accounts)). CI runs both providers against a local stand-in for the CDP API and Anvil, in [`integrations/test/agentkit-cdp.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/agentkit-cdp.int.test.ts): `sendTransaction`, `nativeTransfer` and the ERC-20 `approve` action of `CdpEvmWalletProvider`, and `sendTransaction` and `waitForTransactionReceipt` of `CdpSmartWalletProvider`. `configureWithWallet()` itself is not run: it creates its `CdpClient` from the API key and wallet secret only, its config in AgentKit 0.10.4 has no field for the client's `basePath` option, and the CDP SDK reads no environment variable for the API's base URL. The test builds each provider with the constructor that `configureWithWallet()` ends with, around a `CdpClient` created with `basePath`. ### Other wallet providers - `PrivyEvmWalletProvider`, `PrivyEvmDelegatedEmbeddedWalletProvider`, `ZeroDevWalletProvider` and the legacy CDP providers build their clients internally or send through their own APIs. Record their transactions with [`watch()`](https://hashspan.dev/docs/adapters/viem#transactions-sent-elsewhere) and the hash the provider returns, and wrap the provider's send call as shown there for a send span. AgentKit reports each wallet provider's initialization and each action invocation to its analytics endpoint; this is AgentKit's own behaviour and independent of hashspan. In 0.10.4 the report is not awaited, so any failure, a request that cannot connect or a response that is not 2xx, is an unhandled rejection, and Node.js ends the process when it happens: possibly after the transaction is sent but before `hashspan.flush()` and the SDK's shutdown run, so its spans are never exported. AgentKit has no setting to turn the reports off ([coinbase/agentkit#1531](https://github.com/coinbase/agentkit/issues/1531)); running Node.js with `NODE_OPTIONS=--unhandled-rejections=warn` logs the rejection instead and lets the process finish. ## GOAT SDK GOAT is no longer maintained: since July 2026 its repository is a read-only snapshot that accepts no issues or pull requests, and `@goat-sdk/core` 0.5.0 and `@goat-sdk/wallet-viem` 0.3.0 (May 2025) are its last releases. The setup below works with those releases. Checked with `@goat-sdk/wallet-viem` 0.3.0. GOAT sends with methods of the wallet client you pass to `viem()`, and waits for receipts on a client it derives from it, so the setup needs background confirmation: ```ts // GOAT waits for receipts on a client it derives from this one: background confirmation records them. const hashspan = withHashspan({ confirm: { mode: 'background' } }); const wallet = viem(createWalletClient({ account, chain, transport: http() }).extend(hashspan)); ``` Pass `wallet` to GOAT as usual, for example `getOnChainTools({ wallet })`. A transaction sent through it then records a `send` and a `confirm` span. CI runs this setup against Anvil with GOAT's `send_token` and `approve_token_evm` tools, invoked directly, both ways, in [`integrations/test/goat-viem.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/goat-viem.int.test.ts). `@goat-sdk/wallet-viem` 0.3.0 pins viem 2.23.4 exactly. With a newer viem in the application (2.57 in the test), TypeScript rejects the wallet client in `viem(walletClient)`, while the client works at runtime. Cast it as the test does, with or without hashspan: `viem(walletClient as unknown as Parameters[0])`. With viem 2.23.4 itself, no cast is needed. GOAT's packages have no `exports` map, so Node.js loads their CommonJS build even from an ES module; the application's viem and GOAT's then come from different module copies. With viem 2.57 this does not affect the spans, because hashspan wraps the client instance rather than a module. ## Agent frameworks hashspan's spans are children of the OpenTelemetry span that is active when the transaction is sent. Whether a framework's tool-call span is that span depends on its instrumentation. A framework's own spans follow its own privacy settings, not hashspan's: see the [privacy notes](https://hashspan.dev/docs/adapters/core#privacy-notes). - **Mastra:** with its OpenTelemetry bridge (`OtelBridge` from `@mastra/otel-bridge`, set as `bridge` in an `Observability` config of `@mastra/observability`) and a registered tracer provider and context manager (the OpenTelemetry Node SDK registers both), each tool runs inside its tool span: the trace reads `invoke_agent `, then Mastra's spans of the model call and the agent's step, then `execute_tool `, whose children are `send` and `confirm`. Without Mastra observability, the send and confirm spans start traces of their own. CI runs both setups against Anvil with a scripted model, from `@mastra/core` 1.74.0 and `@mastra/otel-bridge` 1.5.13 on, in [`integrations/test/mastra.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/mastra.int.test.ts). Mastra marks the bridge as experimental. To set it up: - Install `@mastra/core @mastra/observability @mastra/otel-bridge zod @opentelemetry/sdk-node` next to the packages of the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md), and `ai` 7 for the scripted model below: it stands in for a model provider so the example needs no API key, and an agent with a real model does not need `ai`. - Start the Node SDK before the agent runs a tool, give Mastra the bridge, and send from the tool with a client extended with `withHashspan()`, as in this `agent.ts`. It runs against Anvil like the quick start, prints the agent's answer, and the trace shows in Jaeger under the service `my-agent`: `execute_tool pay_vendor`, with `send 31337` and `confirm 31337` under it. CI runs the file as written against Anvil in [`integrations/test/mastra-agent.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/mastra-agent.int.test.ts). - Flush before the process exits, as the example does: `await hashspan.flush()`, then `await sdk.shutdown()` ([shutting down](https://hashspan.dev/docs/adapters/viem#shutting-down)). - Mastra's warning that no `storage` is configured and it falls back to an in-memory store is expected: tracing needs no storage. ```ts import { Mastra } from '@mastra/core'; import { Agent } from '@mastra/core/agent'; import { createTool } from '@mastra/core/tools'; import { Observability } from '@mastra/observability'; import { OtelBridge } from '@mastra/otel-bridge'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { MockLanguageModelV4 } from 'ai/test'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { mnemonicToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; import { z } from 'zod'; import { withHashspan } from '@hashspan/viem'; // Exports over OTLP to http://localhost:4318. Start it before the agent runs a tool. const sdk = new NodeSDK({ serviceName: 'my-agent' }); sdk.start(); // Anvil's public test mnemonic: its first account is funded on every Anvil chain. const account = mnemonicToAccount('test test test test test test test test test test test junk'); const transport = http('http://127.0.0.1:8545'); const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: foundry, transport }).extend(hashspan); const reader = createPublicClient({ chain: foundry, transport }).extend(hashspan); const payVendor = createTool({ id: 'pay_vendor', description: 'Pays the vendor an amount of ETH.', inputSchema: z.object({ amountEth: z.string() }), execute: async ({ amountEth }) => { const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const hash = await wallet.sendTransaction({ to, value: parseEther(amountEth) }); const receipt = await reader.waitForTransactionReceipt({ hash }); return { hash, status: receipt.status }; }, }); // A scripted model instead of a provider, so no API key is needed: it calls the tool once, then answers. const usage = { inputTokens: { total: 50, noCache: 50, cacheRead: 0, cacheWrite: 0 }, outputTokens: { total: 20, text: 20, reasoning: 0 }, }; const model = new MockLanguageModelV4({ doGenerate: [ { content: [{ type: 'tool-call', toolCallId: 'call_1', toolName: 'pay_vendor', input: '{"amountEth":"0.01"}' }], finishReason: { unified: 'tool-calls', raw: undefined }, usage, warnings: [], }, { content: [{ type: 'text', text: 'Paid the vendor 0.01 ETH.' }], finishReason: { unified: 'stop', raw: undefined }, usage, warnings: [], }, ], }); const agent = new Agent({ id: 'treasury', name: 'treasury', instructions: 'Pay vendors when asked.', model, tools: { pay_vendor: payVendor }, }); // The bridge runs each tool inside its execute_tool span, so the send and confirm spans become its children. new Mastra({ agents: { treasury: agent }, observability: new Observability({ configs: { default: { serviceName: 'my-agent', bridge: new OtelBridge() } }, }), }); const result = await agent.generate('Pay the vendor 0.01 ETH.'); console.log(result.text); // Before the process exits: hashspan's pending spans first, then the SDK. await hashspan.flush(); await sdk.shutdown(); ``` - **Genkit:** Genkit runs each tool's function inside the tool's span, so with a registered tracer provider and context manager the trace reads `generate`, then Genkit's spans of the model calls and of the tool, `pay_vendor`, whose children are `send` and `confirm`. On its first span Genkit starts an OpenTelemetry Node SDK of its own, from the 1.x SDK line (`@opentelemetry/sdk-node` 0.52), unless `disableGenkitOTelInitialization()` from `genkit/tracing` was called first. Both SDK lines install side by side; OpenTelemetry keeps the first tracer provider registered in a process and refuses the others. CI runs the example below as written and the setups of the limits against Anvil with a scripted model, from `genkit` 1.42.0 on, in [`integrations/test/genkit.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/genkit.int.test.ts). To set it up: - Install `genkit @opentelemetry/sdk-node` next to the packages of the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md). The scripted model below stands in for a model plugin, so the example needs no API key. - Call `disableGenkitOTelInitialization()` and start the Node SDK before Genkit runs anything, and send from the tool with a client extended with `withHashspan()`, as in this `agent.ts`. It runs against Anvil like the quick start, prints the model's answer, and the trace shows in Jaeger under the service `my-agent`: `pay_vendor`, with `send 31337` and `confirm 31337` under it. - Flush before the process exits, as the example does: `await hashspan.flush()`, then `await sdk.shutdown()` ([shutting down](https://hashspan.dev/docs/adapters/viem#shutting-down)). ```ts import { withHashspan } from '@hashspan/viem'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { genkit, z } from 'genkit'; import { disableGenkitOTelInitialization } from 'genkit/tracing'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { mnemonicToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; // Genkit starts an OpenTelemetry SDK of its own on its first span unless told not to: this process has one. disableGenkitOTelInitialization(); // Exports over OTLP, to http://localhost:4318 unless OTEL_EXPORTER_OTLP_ENDPOINT says otherwise. Start it before // Genkit runs anything. const sdk = new NodeSDK({ serviceName: 'my-agent' }); sdk.start(); // Anvil's public test mnemonic: its first account is funded on every Anvil chain. const account = mnemonicToAccount('test test test test test test test test test test test junk'); const transport = http(process.env.RPC_URL ?? 'http://127.0.0.1:8545'); // Anvil const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: foundry, transport }).extend(hashspan); const reader = createPublicClient({ chain: foundry, transport }).extend(hashspan); const ai = genkit({}); // Genkit runs the tool's function inside the tool's span, so the send and confirm spans become its children. const payVendor = ai.defineTool( { name: 'pay_vendor', description: 'Pays the vendor an amount of ETH.', inputSchema: z.object({ amountEth: z.string() }), outputSchema: z.object({ hash: z.string(), status: z.string() }), }, async ({ amountEth }) => { const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const hash = await wallet.sendTransaction({ to, value: parseEther(amountEth) }); const receipt = await reader.waitForTransactionReceipt({ hash }); return { hash, status: receipt.status }; }, ); // A scripted model instead of a provider plugin, so no API key is needed: it asks for the tool once, then answers. ai.defineModel({ name: 'scripted' }, async (request) => { const paid = request.messages.some((message) => message.content.some((part) => part.toolResponse)); return { message: { role: 'model', content: paid ? [{ text: 'Paid the vendor 0.01 ETH.' }] : [{ toolRequest: { name: 'pay_vendor', ref: 'call_1', input: { amountEth: '0.01' } } }], }, finishReason: 'stop', }; }); const { text } = await ai.generate({ model: 'scripted', prompt: 'Pay the vendor 0.01 ETH.', tools: [payVendor], }); console.log(text); // Before the process exits: hashspan's pending spans first, then the SDK. await hashspan.flush(); await sdk.shutdown(); ``` Known limits with Genkit 1.42; CI checks the first two, with and without the calls they name: - **Start your SDK first:** without `disableGenkitOTelInitialization()`, when a Genkit span starts before your SDK does, Genkit's own SDK registers the tracer provider, your SDK's registration fails, and none of the process's spans reach your exporters. With your SDK started first, the trace is recorded as above, and Genkit's SDK fails to register and stays idle; with a diag logger set, OpenTelemetry reports `Attempted duplicate registration of API: trace` for it. - **An active span of your own:** `generate` starts a trace of its own even when a span is active, such as an HTTP server's span. Call `disableOTelRootSpanDetection()` from `genkit/tracing` at startup to make it a child of the active span. - **Genkit's Developer UI:** Genkit sends traces to its Developer UI (`GENKIT_TELEMETRY_SERVER`) only through a span processor of its own SDK, so with this setup the Developer UI receives no spans. - **LangChain JS and the OpenAI Agents SDK:** OpenInference's instrumentations (`@arizeai/openinference-instrumentation-langchain` 4.1.4, `@arizeai/openinference-instrumentation-openai-agents` 0.3.2) record tool spans but do not make them active, so hashspan's spans attach to whatever span was active before the agent ran (upstream: [Arize-ai/openinference#3925](https://github.com/Arize-ai/openinference/issues/3925) for the OpenAI Agents SDK, [#1103](https://github.com/Arize-ai/openinference/issues/1103) for LangChain JS). When the agent runs inside an active span of your own, hashspan's spans and OpenInference's spans are in that span's trace. CI runs both instrumentations against Anvil with a scripted model (`langchain` 1.5.15 and `@openai/agents` 0.18.0), with and without the workaround below, in [`integrations/test/openinference-langchain.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/openinference-langchain.int.test.ts) and [`integrations/test/openinference-openai-agents.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/openinference-openai-agents.int.test.ts). The OpenAI Agents SDK turns its tracing off when `NODE_ENV` is `test`, so OpenInference records nothing there until `setTracingDisabled(false)` is called. To group a tool's transactions, run the tool's function in an active span of your own; that span is a sibling of OpenInference's tool span, not its child: ```ts const tracer = trace.getTracer('treasury-agent'); // The tool's function: its send and confirm spans become children of `pay_vendor`. async function payVendor(): Promise<`0x${string}`> { return tracer.startActiveSpan('pay_vendor', async (span) => { try { const hash = await wallet.sendTransaction({ to, value }); await wallet.waitForTransactionReceipt({ hash }); return hash; } finally { span.end(); } }); } ``` - **ElizaOS:** `@elizaos/plugin-evm` 1.0.13 creates a new wallet client from the `EVM_PRIVATE_KEY` setting for each action and offers no way to pass a client of your own, so its transactions are not traced today. ## MCP servers When an agent calls a tool of an MCP server and that tool sends the transaction, hashspan runs in the server's process, and its spans are children of the span active there. Nothing in the MCP SDKs carries the agent's trace context to the server, so the server's `send` and `confirm` spans start a trace of their own, apart from the agent's tool call. MCP reserves the `traceparent`, `tracestate` and `baggage` keys of a request's `_meta` for W3C trace context; the v2 SDK exports their names (`TRACEPARENT_META_KEY`, `TRACESTATE_META_KEY`, `BAGGAGE_META_KEY`), but neither SDK fills or reads them. Checked over stdio with `@modelcontextprotocol/sdk` 1.32.1 (v1) and `@modelcontextprotocol/client` and `@modelcontextprotocol/server` 2.3.1 (v2). With the v2 SDK, inject the active context into the `_meta` of each tool call, and run each tool handler in the context it extracts from `ctx.mcpReq._meta`. The server, `mcp-server.ts`, sends with a client extended with `withHashspan()`: ```ts import { withHashspan } from '@hashspan/viem'; import { McpServer, type ServerContext } from '@modelcontextprotocol/server'; import { serveStdio } from '@modelcontextprotocol/server/stdio'; import { context, propagation } from '@opentelemetry/api'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { mnemonicToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; import { z } from 'zod'; // Exports over OTLP, to http://localhost:4318 unless OTEL_EXPORTER_OTLP_ENDPOINT says otherwise. Never log to // stdout: it carries the MCP messages. const sdk = new NodeSDK({ serviceName: 'wallet-mcp' }); sdk.start(); const account = mnemonicToAccount('test test test test test test test test test test test junk'); const transport = http(process.env.RPC_URL ?? 'http://127.0.0.1:8545'); // Anvil const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: foundry, transport }).extend(hashspan); const reader = createPublicClient({ chain: foundry, transport }).extend(hashspan); /** Runs `fn` in the trace context the client sent in the request's `_meta`. */ function inClientTrace(ctx: ServerContext, fn: () => Promise): Promise { return context.with(propagation.extract(context.active(), ctx.mcpReq._meta ?? {}), fn); } function createServer(): McpServer { const server = new McpServer({ name: 'wallet', version: '1.0.0' }); server.registerTool( 'pay_vendor', { description: 'Pays the vendor an amount of ETH.', inputSchema: z.object({ amountEth: z.string() }) }, ({ amountEth }, ctx) => inClientTrace(ctx, async () => { const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const hash = await wallet.sendTransaction({ to, value: parseEther(amountEth) }); const receipt = await reader.waitForTransactionReceipt({ hash }); return { content: [{ type: 'text', text: `${hash} ${receipt.status}` }] }; }), ); return server; } serveStdio(createServer); // The client ends stdin to stop the server: export the pending spans before the process exits. process.stdin.once('end', async () => { await hashspan.flush(); await sdk.shutdown(); process.exit(0); }); ``` The agent, `mcp-client.ts`, starts the server as a child process and calls the tool inside an active span of its own. With Anvil running, run it with `node --import tsx mcp-client.ts` after installing `@modelcontextprotocol/client @modelcontextprotocol/server @opentelemetry/sdk-node zod tsx` and the packages of the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md). The trace shows `pay_vendor` of the service `my-agent`, with `send 31337` and `confirm 31337` of the service `wallet-mcp` under it: ```ts import { Client } from '@modelcontextprotocol/client'; import { getDefaultEnvironment, StdioClientTransport } from '@modelcontextprotocol/client/stdio'; import { context, propagation, trace } from '@opentelemetry/api'; import { NodeSDK } from '@opentelemetry/sdk-node'; const sdk = new NodeSDK({ serviceName: 'my-agent' }); sdk.start(); /** The active trace context as `_meta` keys (`traceparent`, `tracestate`, `baggage`) for one request. */ function traceMeta(): Record { const meta: Record = {}; propagation.inject(context.active(), meta); return meta; } // A stdio server gets only a few variables of this environment, such as PATH and HOME: pass on its // OpenTelemetry and RPC settings. It runs mcp-server.ts next to this file, with this process's Node.js flags (tsx's). const settings = Object.entries(process.env).filter( (entry): entry is [string, string] => (entry[0].startsWith('OTEL_') || entry[0] === 'RPC_URL') && entry[1] !== undefined, ); const transport = new StdioClientTransport({ command: process.execPath, args: [...process.execArgv, new URL('./mcp-server.ts', import.meta.url).pathname], env: { ...getDefaultEnvironment(), ...Object.fromEntries(settings) }, }); const client = new Client({ name: 'my-agent', version: '1.0.0' }); await client.connect(transport); // The agent's tool call: the server's send and confirm spans become children of this span. await trace.getTracer('my-agent').startActiveSpan('pay_vendor', async (span) => { try { const result = await client.callTool({ name: 'pay_vendor', arguments: { amountEth: '0.01' }, _meta: traceMeta(), }); console.log(result.content); } finally { span.end(); } }); // Closing ends the server's stdin, so it exports its spans and exits; then this process's spans. await client.close(); await sdk.shutdown(); ``` With the v1 SDK, `@arizeai/openinference-instrumentation-mcp` 0.2.36 does both steps for every request, and creates no spans of its own. Install it, and run this in the agent's process and in the server's: ```ts import { MCPInstrumentation } from '@arizeai/openinference-instrumentation-mcp'; import * as clientStdioModule from '@modelcontextprotocol/sdk/client/stdio.js'; import * as serverStdioModule from '@modelcontextprotocol/sdk/server/stdio.js'; // In the agent's process and in the server's, after the OpenTelemetry SDK starts and before the transport // connects: the client's requests carry the active trace context in `_meta`, and the server handles each request // in that context. new MCPInstrumentation().manuallyInstrument({ clientStdioModule, serverStdioModule }); ``` CI runs all three setups against Anvil in [`integrations/test/mcp.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/mcp.int.test.ts), with the agent and the server as separate processes: the v2 files above as written, and a v1 agent and server with and without the instrumentation. Without it, the server's `send` span is the root of a trace of its own; with it, and with the v2 setup, `send` and `confirm` are children of the agent's `pay_vendor` span. Known limits: - **The server's environment:** both SDKs' stdio transports start the server with only a few variables of the agent's environment (`getDefaultEnvironment()`), so the server does not see `OTEL_*` settings unless the agent passes them in `env`, as above. - **stdout:** it carries the MCP messages, so the server cannot use a console exporter or log to stdout. - **Shutting down:** the v1 server transport does not close when stdin ends, and both clients' `close()` waits 2 seconds for the server to exit before it sends SIGTERM. Flush on the end of stdin, as the server above does, so the last spans are exported in time. - **OpenInference with v2:** the instrumentation's types and its automatic hooks cover only the modules of `@modelcontextprotocol/sdk`; use the hand-written lines with the v2 packages. - **Other transports:** only stdio is checked. The `_meta` lines do not depend on the transport. - **Which span is the parent:** the tool span must be active in the agent when it calls `callTool`; see [agent frameworks](#agent-frameworks) for frameworks whose tool spans are not active. The server takes the client's trace context as given, as an HTTP server does with a `traceparent` header. ## Wallet services How a wallet service sends decides what hashspan records: - **Services that provide a viem account** sign while viem sends: create a wallet client with that account and extend it as in the [viem quick start](https://hashspan.dev/docs/adapters/viem#usage). Every transaction is traced like one signed locally. - **Services that provide an EIP-1193 provider** send when the provider receives `eth_sendTransaction` and return the hash: create a wallet client with `custom(provider)`, the wallet's address as `account` and the chain, and extend it with `withHashspan()`. viem sends `eth_sendTransaction` for an address-only account, so the send span covers the service's call; wait for receipts through an extended public client on an RPC endpoint when the provider does not answer reads, or confirm with `watch()` on one. The provider needs to answer only `eth_chainId`, which viem asks before it sends, and `eth_sendTransaction`; hashspan sends it nothing else. Background confirmation polls through the sending client, so it needs a provider that answers reads. - **Services that send through their own API** return a transaction hash, sometimes only after polling: record the API call as a send span with the core's tracker and the transaction with `watch()` on the same tracker, as in [transactions sent elsewhere](https://hashspan.dev/docs/adapters/viem#transactions-sent-elsewhere). The send span then covers the API call, the confirm span the wait, linked to it. Without the recorded call, there is only the confirm span. These services were checked against the source of their published packages, on the versions named; none of them runs in CI. Their patterns do, on Anvil: a provider that sends after a delay and answers only `eth_chainId`, `eth_accounts` and `eth_sendTransaction` (`packages/viem/test/eip1193-provider.int.test.ts`), and an API call recorded as a send (`packages/viem/test/api-wallet.test.ts`). | Service | How it sends | Use | Spans | |---|---|---|---| | Turnkey (`@turnkey/viem` 0.14) | `createAccount()` returns a viem account | `withHashspan()` on the wallet client | send, confirm | | Privy (`@privy-io/node` 0.35) | `createViemAccount()` from `@privy-io/node/viem` returns a viem account | `withHashspan()` on the wallet client | send, confirm | | Privy API | `wallets().ethereum().sendTransaction()` returns the `hash` | the call recorded as a send, then `watch()` | send, confirm | | Privy x402 (`createX402Client()` from `@privy-io/node/x402`, experimental) | returns an `x402Client` | [`@hashspan/x402`](https://hashspan.dev/docs/adapters/x402#usage) on that client, with `@x402/core` 2.13 or later | payment, confirm | | Circle developer-controlled wallets (`@circle-fin/developer-controlled-wallets` 10.8) | `createEIP1193Provider()` from `/evm`: `eth_sendTransaction` creates the transaction and resolves with its hash once Circle has sent it; other reads go to its `fallback` provider, if one is set | `custom(provider)`; a public client on your RPC endpoint for receipts | send, confirm | | Circle API | `createTransaction()` returns an id; `getTransaction({ id, waitForState: 'SENT' })` returns the `txHash` | both calls recorded as one send, then `watch()` | send, confirm | | Fireblocks (`@fireblocks/fireblocks-web3-provider` 1.5) | an EIP-1193 provider: `eth_sendTransaction` creates a Fireblocks transaction and returns its hash; other requests go to its `rpcUrl` | `custom(provider)` with the vault account's address | send, confirm | - **Send duration:** with an account or a provider from a service, the send span also covers the service's queue and its policy or approval steps, so it lasts longer than a local signer's. - **Privy:** `@privy-io/server-auth` is deprecated in favour of `@privy-io/node`; its `createViemAccount()` works the same way. - **Circle smart contract accounts (ERC-4337):** Circle reports their hash only once the transaction is confirmed; this path was not checked. ## Machine Payments Protocol (mppx) [mppx](https://github.com/wevm/mppx) pays an API's `402 Payment Required` challenge with a Tempo transaction. [`@hashspan/mpp`](https://hashspan.dev/docs/adapters/mpp#usage) records each payment as a payment span, with the settlement's confirm span through a reader; it supersedes the setup of a traced `getClient` client and a manual `watch()` on the receipt's `reference`. What a trace shows depends on who broadcasts the transaction: | Charge mode | Who broadcasts | Spans, with `@hashspan/mpp` and a reader | |---|---|---| | `pull` (the default for a private key) | the server, the transaction the payer signed | payment; confirm, linked to the payment span, from the reader | | `push` (`mode: 'push'`, or a wallet account) | the payer, through the client mppx's `getClient` returns | payment; send and confirm of the payer's client when it is extended with `@hashspan/viem`'s `withHashspan()`, on the same `tracker`, with the payment span linked to the send span | 1. Extend the payer's viem client with `@hashspan/viem`'s `withHashspan({ tracker })` and pass it to mppx's `tempo({ account, getClient })`, so that a push transaction gets its send span. 2. Call `withHashspan(mppx, { tracker, reader })` from `@hashspan/mpp` on the client `Mppx.create()` returns, with the same tracker: with two, a push payment gets two confirm spans and two fee samples. 3. Call `flush()` of both before a short-lived process exits. The payment's fee is the payer's, and is labelled `sponsor` when the server sponsors it as fee payer, never `facilitator`. The README's quick start runs in CI as written, against an mppx server and Anvil in Tempo mode (`packages/mpp/test/quick-start.int.test.ts`); the real charges, both modes, sponsored fees and splits, are in `packages/mpp/test/charge.int.test.ts`. On Anvil in Tempo mode, mine one block before the first charge: a fresh chain's first block fails Tempo's `validAfter` check in the server's simulation. ## Hardhat 3 Checked with `hardhat` 3.20.0 and, of the plugins of Hardhat's viem project template (`npx hardhat --init`), `@nomicfoundation/hardhat-viem` 3.0.10, `@nomicfoundation/hardhat-ignition` 3.1.9, `@nomicfoundation/hardhat-ignition-viem` 3.1.6 and `@nomicfoundation/hardhat-node-test-runner` 3.1.1, on Hardhat's in-process network. The `viem` helpers of a network connection create new viem clients on every call, and `deployContract` and `getContractAt` send with clients of their own unless they get a `client` option. Extend the public client and a wallet client with `withHashspan()`, and pass both as `client` to every helper that sends. A script, `scripts/counter.ts`, after installing the packages of the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md) in the project, run with `npx hardhat run scripts/counter.ts`: ```ts import { withHashspan } from '@hashspan/viem'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { network } from 'hardhat'; // Exports over OTLP, to http://localhost:4318 unless OTEL_EXPORTER_OTLP_ENDPOINT says otherwise. const sdk = new NodeSDK({ serviceName: 'counter-script' }); sdk.start(); const { viem } = await network.create(); const hashspan = withHashspan(); // hardhat-viem creates new clients for each helper: extend them, and pass them to every helper that sends. const publicClient = (await viem.getPublicClient()).extend(hashspan); const [sender] = await viem.getWalletClients(); if (sender === undefined) throw new Error('The network has no accounts'); const client = { public: publicClient, wallet: sender.extend(hashspan) }; // The deployment records a confirm span only: deployContract is not traced yet. const counter = await viem.deployContract('Counter', [], { client }); const hash = await counter.write.incBy([5n]); const receipt = await publicClient.waitForTransactionReceipt({ hash }); console.log(`${hash} ${receipt.status}`); // Before the script ends: await hashspan.flush(); await sdk.shutdown(); ``` The `incBy` call records a `send 31337` and a `confirm 31337` span; the deployment records a `confirm` span only, since viem's `deployContract` is not traced yet ([#36](https://github.com/selimaytac/hashspan/issues/36)). The in-process network mines each transaction when it is sent (automine), so a confirm span covers only the wait for a receipt that is already there. In a test file of Hardhat's node:test runner, run with `npx hardhat test nodejs`, the same clients are set up once per suite. The runner runs the test files in Hardhat's own process, so the SDK starts once and shuts down after the last test: ```ts import assert from 'node:assert/strict'; import { after, describe, it } from 'node:test'; import { withHashspan } from '@hashspan/viem'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { network } from 'hardhat'; // Hardhat runs the test files in its own process: start the SDK once, before the first test. const sdk = new NodeSDK({ serviceName: 'counter-tests' }); sdk.start(); describe('Counter', async () => { const { viem } = await network.create(); const hashspan = withHashspan(); const publicClient = (await viem.getPublicClient()).extend(hashspan); const [sender] = await viem.getWalletClients(); if (sender === undefined) throw new Error('The network has no accounts'); const client = { public: publicClient, wallet: sender.extend(hashspan) }; after(async () => { await hashspan.flush(); await sdk.shutdown(); }); it('adds the increment', async () => { const counter = await viem.deployContract('Counter', [], { client }); const hash = await counter.write.incBy([5n]); await publicClient.waitForTransactionReceipt({ hash }); assert.equal(await counter.read.x(), 5n); }); }); ``` Ignition sends a deployment's transactions through the network connection's provider, not through a viem client, so `ignition.deploy()` records no spans; `npx hardhat ignition deploy` sends the same way. The contracts it returns send with clients hardhat-viem creates; bind them to the extended clients with `getContractAt` to trace their calls: ```ts import { withHashspan } from '@hashspan/viem'; import { buildModule } from '@nomicfoundation/hardhat-ignition/modules'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { network } from 'hardhat'; const sdk = new NodeSDK({ serviceName: 'counter-ignition' }); sdk.start(); const CounterModule = buildModule('CounterModule', (m) => { const counter = m.contract('Counter'); m.call(counter, 'incBy', [5n]); return { counter }; }); const { viem, ignition } = await network.create(); const hashspan = withHashspan(); const publicClient = (await viem.getPublicClient()).extend(hashspan); const [sender] = await viem.getWalletClients(); if (sender === undefined) throw new Error('The network has no accounts'); const client = { public: publicClient, wallet: sender.extend(hashspan) }; // Ignition sends through the network connection's provider, not through a viem client: no spans. const { counter } = await ignition.deploy(CounterModule); // The contract Ignition returns sends with clients of its own: bind it to the extended ones. const traced = await viem.getContractAt('Counter', counter.address, { client }); const hash = await traced.write.incBy([1n]); const receipt = await publicClient.waitForTransactionReceipt({ hash }); console.log(`${hash} ${receipt.status}`); await hashspan.flush(); await sdk.shutdown(); ``` CI runs the three files above as written, with `hardhat run` and `hardhat test nodejs` in a Hardhat project, in [`integrations/test/hardhat.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/hardhat.int.test.ts), and a script without the `client` option that records no spans. Known limits: - **Deployments:** with `client`, `deployContract` records only the confirm span of its own wait; without it, nothing. - **Ignition:** the transactions of a deployment are not traced. Calls after the deployment are, through a contract bound with `getContractAt` as above. CI runs `ignition.deploy()` from a script; the CLI task `hardhat ignition deploy` is not run. - **Several test files:** the file above starts and shuts down an SDK of its own. Two such files in one run exported only the first file's spans, since Hardhat runs them in one process. Only one test file per run is checked. - **Networks:** only the in-process network (`edr-simulated`, `chainType: 'l1'`) is checked; `http` networks and the `op` chain type are not. ## RPC gateways (eRPC) [eRPC](https://github.com/erpc/erpc) is a JSON-RPC gateway between an agent and its nodes, with caching, failover and its own OpenTelemetry spans. It continues the trace of a request that carries a W3C `traceparent` header. hashspan's JSON-RPC spans send no header unless asked to, so without HTTP instrumentation each request the agent sends starts a trace of its own in eRPC. With `@opentelemetry/instrumentation-undici`, which traces Node.js's `fetch` that viem's `http()` transport uses, or with `propagateTraceContext()` from `@hashspan/viem` ([below](#without-http-instrumentation)), every request carries the header and the trace runs from the agent's `send` span through eRPC to the node. Checked with eRPC 0.4.0 and `@opentelemetry/instrumentation-undici` 0.32.0. An eRPC config, `erpc.yaml`, for Anvil on 127.0.0.1:8545 and an OTLP/HTTP collector on 127.0.0.1:4318, started with `erpc --require-config --config erpc.yaml` (without a config, `--require-config` stops eRPC from starting with a default project of public endpoints): ```yaml logLevel: info server: listenV4: true httpHostV4: 127.0.0.1 httpPortV4: 4000 listenV6: false # eRPC's metrics server listens on every interface whatever metrics.hostV4 says. metrics: enabled: false tracing: enabled: true protocol: http # host:port, without a scheme: see the known limits below. endpoint: 127.0.0.1:4318 # Below 1, eRPC follows the agent's sampling decision. sampleRate: 0.1 serviceName: erpc projects: - id: main upstreams: - id: anvil endpoint: http://127.0.0.1:8545 evm: chainId: 31337 networks: - architecture: evm evm: chainId: 31337 ``` The agent, `agent.ts`, after installing `@opentelemetry/sdk-node @opentelemetry/instrumentation-undici` and the packages of the [quick start](https://github.com/selimaytac/hashspan/blob/main/docs/quick-start.md), run with `node --import tsx agent.ts`: ```ts import { traceTransport, withHashspan } from '@hashspan/viem'; import { trace } from '@opentelemetry/api'; import { UndiciInstrumentation } from '@opentelemetry/instrumentation-undici'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { mnemonicToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; // Exports over OTLP, to http://localhost:4318 unless OTEL_EXPORTER_OTLP_ENDPOINT says otherwise. The undici // instrumentation traces each request of Node.js's fetch, which viem's http() transport uses, and sends the // `traceparent` header that puts eRPC's spans in this trace. const sdk = new NodeSDK({ serviceName: 'my-agent', instrumentations: [new UndiciInstrumentation()] }); sdk.start(); const account = mnemonicToAccount('test test test test test test test test test test test junk'); // eRPC's project `main` and its network `evm` 31337 in erpc.yaml. const transport = traceTransport(http(process.env.RPC_URL ?? 'http://127.0.0.1:4000/main/evm/31337')); const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: foundry, transport }).extend(hashspan); const reader = createPublicClient({ chain: foundry, transport }).extend(hashspan); await trace.getTracer('my-agent').startActiveSpan('pay_vendor', async (span) => { try { const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const hash = await wallet.sendTransaction({ to, value: parseEther('0.01') }); const receipt = await reader.waitForTransactionReceipt({ hash }); console.log(`${hash} ${receipt.status}`); } finally { span.end(); } }); await hashspan.flush(); await sdk.shutdown(); ``` The trace shows `pay_vendor` > `send 31337` > `eth_sendRawTransaction` (hashspan) > `POST` (undici) > `Http.ReceivedRequest` (eRPC) > ... > `HttpJsonRpcClient.sendSingleRequest` (eRPC's call to the node), and the same for each other request. ### Without HTTP instrumentation `propagateTraceContext()`, a hook for viem's `http()` transport, sends the header from each JSON-RPC span itself (viem 2.21.25 or later; see the [viem README](https://hashspan.dev/docs/adapters/viem#trace-context-for-the-provider)). Drop the undici instrumentation, import `propagateTraceContext` from `@hashspan/viem`, and change the SDK and the transport of `agent.ts` to: ```ts // No HTTP instrumentation: the hook sends the `traceparent` header of each JSON-RPC span. const sdk = new NodeSDK({ serviceName: 'my-agent' }); sdk.start(); const transport = traceTransport( http(process.env.RPC_URL ?? 'http://127.0.0.1:4000/main/evm/31337', { onFetchRequest: propagateTraceContext(), }), ); ``` The trace then shows `eth_sendRawTransaction` (hashspan) > `Http.ReceivedRequest` (eRPC), with no `POST` span between them. The hook sends `traceparent` and `tracestate`, never `baggage`. CI runs both agents as written against eRPC in front of Anvil, with this config, in [`integrations/test/erpc.int.test.ts`](https://github.com/selimaytac/hashspan/blob/main/integrations/test/erpc.int.test.ts), and a control with neither, whose eRPC spans are in traces of their own. Known limits: - **Never both:** use `propagateTraceContext()` or the undici instrumentation, not both. The undici instrumentation adds a second `traceparent` line rather than replacing the first, and a proxy that joins the two into one value makes eRPC start a new trace for every request. - **Two client spans per request, with undici:** each JSON-RPC span of `traceTransport()` gets the undici `POST` span as a child, and eRPC's server span sits under that one. A span-metrics generator counts both ([span metrics](https://hashspan.dev/docs/backends#span-metrics-and-http-instrumentation)). With `propagateTraceContext()` there is one. - **Batches:** with `http(url, { batch: true })`, an HTTP request of several JSON-RPC requests gets no header from `propagateTraceContext()`, since its requests can belong to different traces; eRPC traces it on its own. - **Tracing endpoint over http:** in eRPC 0.4.0, `tracing.endpoint` with `protocol: http` must be `host:port`. A URL with a scheme, and the default endpoint when none is set, export nothing, and the error is logged only at the `trace` level ([erpc/erpc#1276](https://github.com/erpc/erpc/issues/1276)). - **Sampling:** at `sampleRate: 1`, eRPC's default, eRPC records every request whatever the agent decided, so with a sampler in the agent, eRPC's spans of the traces the agent dropped show up without their root. Below 1, eRPC follows the sampled flag of the header and samples requests without one at that rate. Set a small positive value: eRPC reads `0` as unset and uses 1. - **Multiplexing:** eRPC answers identical requests that arrive together with one call to the node, on by default. The trace of a request that waits for another one's call has eRPC's `Network.WaitForMultiplexResult` span and no span of the node call, which is only in the other request's trace, with no link to it. `multiplexing: false` on the network gives each request its own call. A cache hit has no node call either. - **Baggage:** eRPC passes `tracestate` on to the node, but not `baggage`. - **Other transports:** only HTTP is checked; eRPC does not continue a trace over gRPC, and viem does not send to it over gRPC. Concepts Short explanations of the ideas on this page: Traces and spans: the anatomy of one agent run Context propagation in async Node.js Transaction lifecycle: send, mempool, confirm --- # Local lab Source: https://hashspan.dev/docs/local-lab The repository ships an example agent built with the Vercel AI SDK. It pays a vendor, then tries a withdrawal that reverts, and both transactions show up in its trace. It runs against a local Anvil chain and a scripted model, so it needs no API key. ## Run it You need Node.js 22.18 or later, pnpm and Docker. With nvm, `nvm install` reads the repository's `.nvmrc` (Node.js 24), installs it if needed and switches to it. The packages themselves only need Node.js 22.3; the higher version is for the repository's build tools. ```sh git clone https://github.com/selimaytac/hashspan && cd hashspan nvm install && corepack enable pnpm && pnpm install make lab-up # Jaeger UI on http://localhost:16686, OTLP on :4317/:4318 make demo # runs the agent against a fresh local chain ``` For the metrics, `make lab-metrics` also starts Prometheus and Grafana with a dashboard for the send, confirmation and fee histograms, on http://localhost:3000; run `make demo` after it. The [backends page](https://hashspan.dev/docs/backends#grafana-dashboard-for-the-metrics) explains the dashboard and how to use it elsewhere. Open Jaeger, pick the `treasury-agent` service and expand `withdraw_from_vault`. The second `confirm 31337` span is marked as an error and carries `blockchain.tx.revert.reason`. [This post](https://hashspan.dev/blog/agent-trace-stops-at-the-tool-call) walks through the same trace, span by span. ## Run it on a testnet The same agent also runs on Base Sepolia with a test account of your own: `make demo-base-sepolia`. Its spans are named `send 84532` and `confirm 84532`, and each transaction gets a Basescan link. How to set up and fund the account, and keep its key out of the repository, is in the [example's README](https://github.com/selimaytac/hashspan/tree/main/examples/ai-sdk-agent#run-it-on-base-sepolia). ## A real run on Base Sepolia This is one run of `make demo-base-sepolia` with hashspan 0.8.0, on 2026-10-02. The values come from the confirm spans and match the receipts on chain: | Transaction | Status | Gas used | Execution fee | L1 data fee | Total fee (`blockchain.tx.fee`) | |---|---|---|---|---|---| | [Payment](https://sepolia.basescan.org/tx/0x39ee471a316220689a37d7c1eb41a09add1d24a14c1d49d2eedeadfabcb55820) | success | 21,000 | 126,000,000,000 wei | 7,351,065,046 wei | 133,351,065,046 wei | | [Withdrawal](https://sepolia.basescan.org/tx/0x580fcb5b0a2493aecc5217226677ae44c50abecabfcb677abb4c63b1782cf68a) | reverted | 21,630 | 129,780,000,000 wei | 7,351,065,046 wei | 137,131,065,046 wei | The withdrawal's confirm span carries `blockchain.tx.revert.reason` `WithdrawalLimitExceeded(10000000000000, 20000000000000)`. On the local chain there is no L1 data fee; on Base Sepolia it is about 5.5% of the payment's total fee. Every setup run so far is listed on [Tested setups](https://hashspan.dev/docs/tested). ## Clean up ```sh make lab-pause # stop, keep state make lab-nuke # remove containers, images, tools and build output ``` The example's code is in [`examples/ai-sdk-agent`](https://github.com/selimaytac/hashspan/tree/main/examples/ai-sdk-agent). --- # For OpenTelemetry engineers Source: https://hashspan.dev/docs/otel-engineers You know OpenTelemetry. This page gives you hashspan in OTel terms, and the few blockchain words its spans use. ## What does it add to my traces? A library, not a service: it uses your SDK, your exporter and your backend. Inside whatever span is active, such as a tool call, it records: | Span | Kind | Ends when | |---|---|---| | `send {blockchain.chain.id}` | CLIENT | the node returns a transaction hash, or the send fails | | `confirm {blockchain.chain.id}` | CLIENT | the receipt arrives, or a timeout or error; linked to `send` or `payment` | | `payment {blockchain.chain.id}` | CLIENT | a paid request settles or fails (x402, MPP) | It also records histograms for send duration, confirmation duration and fees, for every transaction, whatever a sampler decides. Span names stay low-cardinality: the chain id, never a hash. ## Is the schema stable? Not yet. Every attribute is marked development, in schema `0.5.0-dev` ([semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions)). OpenTelemetry has no blockchain conventions, and had no open proposal when hashspan started ([ADR 0003](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0003-attribute-namespace.md)). So hashspan uses its own `blockchain.*` namespace, shaped like `db.*` and `rpc.*`. Where upstream names exist it reuses them: `error.type`, `rpc.*`, `jsonrpc.*` and `gen_ai.agent.*`. The plan is to propose the schema upstream once it has real users. Renames follow a written change policy. The one rename so far records both names, and drops the old one at 1.0. ## What does it cost? On a local chain, a traced send plus receipt took 0.25 to 0.5 ms more at p50 than plain viem. Tracing can add JSON-RPC requests: one for a transfer where viem picks the nonce, three for a reverted transaction. The [tracing overhead benchmark](https://hashspan.dev/docs/benchmarks/tracing-overhead) has the method and every number. ## How do I run it in production? - **Sampling:** keep head sampling at 100% and decide in a Collector. A reverted transaction only shows on the `confirm` span, which can end after the Collector decides. A policy on `blockchain.*` keeps those traces: [Collector recipe](https://hashspan.dev/docs/recipes/collector), tested. - **Privacy:** addresses are recorded by default, in lower case. Hash or drop them in the SDK or in the Collector. Calldata arguments and error messages are opt-in ([privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy)). - **Backends:** any OTLP backend. [Backends](https://hashspan.dev/docs/backends) has dated notes for Jaeger, Grafana, Langfuse, Honeycomb and others. - **Security and stability:** keys, defaults, vulnerability reports and what 1.0 keeps stable are on [one page](https://hashspan.dev/docs/security). - **Short-lived processes:** call `flush()` before exit, or confirmations still open end as `timeout`. ## Which languages? TypeScript on Node.js 22.3 or later, with viem, the Coinbase CDP SDK, x402 or MPP. There is no Go, Python or Java version. Not on TypeScript? Much of it still applies, because it is plain OpenTelemetry: - **The names:** the [semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions) define the spans, attributes and metrics. You can record the same names from any OpenTelemetry SDK, such as Python's around web3.py. - **The Collector:** the [Collector recipe](https://hashspan.dev/docs/recipes/collector) works on any spans that carry `blockchain.operation.name`. - **Queries and dashboards:** the TraceQL queries on [Backends](https://hashspan.dev/docs/backends) and the [Grafana dashboards](https://github.com/selimaytac/hashspan/tree/main/dashboards) match on those names. ## Seven blockchain words you'll meet | Word | Meaning | |---|---| | Transaction hash | The id of a sent transaction. Like a trace id: known at send, used to look it up later | | Nonce | A per-account counter. Each transaction uses the next one, once | | Gas | The work a transaction does, in units. Units times a price per unit is the fee | | Receipt | The node's record that a transaction ran: block, gas used, success or reverted | | Revert | The contract stopped on purpose and undid its changes. The fee is still paid | | Wei | The smallest unit of ETH, 10^-18. Too big for int64, so hashspan records it as a decimal string | | L1 data fee | On chains built on top of Ethereum, such as Base, a second fee for posting the transaction to Ethereum itself | [Transaction lifecycle](https://hashspan.dev/learn/tx-lifecycle) walks a transaction from send to receipt. The [architecture](https://hashspan.dev/docs/architecture) page shows the packages and every extra request. ## Who maintains it? An open-source project under Apache-2.0, maintained on [GitHub](https://github.com/selimaytac/hashspan). `@hashspan/viem` has had 23 releases since its first, on 2026-09-29. Releases are listed in the [changelog](https://hashspan.dev/changelog). --- # Quickstart Source: https://hashspan.dev/docs/quickstart This guide adds hashspan to an agent that sends transactions with viem. You need Node.js 22.3 or later. - **Adding it to your own agent?** Follow the steps below. - **Starting from scratch?** Jump to the [complete example](#a-complete-example-on-a-local-chain): one file, a local chain, no API key. - **Using a coding agent?** Install the [hashspan skill](https://github.com/selimaytac/hashspan/blob/main/skills/hashspan/SKILL.md) with `npx skills add https://hashspan.dev --skill hashspan`, then ask the agent to add hashspan to your project. [Build with AI](https://hashspan.dev/docs/build-with-ai) has a ready prompt and a rules snippet. - **New to wallets and crypto?** The [x402 tutorial](https://hashspan.dev/docs/x402-tutorial) starts from zero: a wallet, a first payment and an x402 payment on a local chain, each with its trace. - **Just want to see a trace?** The [local lab](https://hashspan.dev/docs/local-lab) runs the repository's example agent. The [x402 example](https://github.com/selimaytac/hashspan/tree/main/examples/x402-agent) adds a payment to a paid API, offline too. - **Not an agent?** In a TypeScript service that sends EVM transactions through viem, the Coinbase CDP SDK, x402 or MPP (mppx), or through a [wallet service](https://hashspan.dev/docs/integrations#wallet-services), such as a payment or payout worker, a wallet backend or a bot, follow the same steps: leave out the `agent` option and send inside the span your request handler or job already has. The `service.name` of your OpenTelemetry resource, set in [Set up OpenTelemetry](#set-up-opentelemetry), tells your services apart. Other libraries, such as ethers, have no adapter, and chains other than EVM are not supported. At volume, see [many transactions](https://hashspan.dev/docs/adapters/viem#many-transactions); [examples/payout-worker](https://github.com/selimaytac/hashspan/tree/main/examples/payout-worker) is a worked example. - **New to traces and spans?** [Traces and spans](https://hashspan.dev/learn/otel-traces-and-spans) explains them on one agent run, in three minutes. ## Add hashspan to your agent 1. ### Install the packages npm: ```sh npm install @hashspan/viem @opentelemetry/api viem ``` pnpm: ```sh pnpm add @hashspan/viem @opentelemetry/api viem ``` yarn: ```sh yarn add @hashspan/viem @opentelemetry/api viem ``` 2. ### Set up OpenTelemetry hashspan only needs `@opentelemetry/api`. Any SDK setup works. If you don't have one yet, this exports spans over OTLP: ```sh npm install \ @opentelemetry/sdk-trace-node \ @opentelemetry/sdk-trace-base \ @opentelemetry/exporter-trace-otlp-proto \ @opentelemetry/resources \ @opentelemetry/semantic-conventions ``` ```ts title="telemetry.ts" import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto'; import { resourceFromAttributes } from '@opentelemetry/resources'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'; import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'; export const provider = new NodeTracerProvider({ resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'my-agent', }), // OTLPTraceExporter honours OTEL_EXPORTER_OTLP_ENDPOINT spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())], }); provider.register(); ``` Import it first in your entry file, before anything that sends transactions: `import { provider } from './telemetry.js';`. Already use `@opentelemetry/sdk-node`? A `NodeSDK` setup works the same; the [agent skill](https://hashspan.dev/.well-known/agent-skills/hashspan/SKILL.md) uses one. This page builds the provider by hand so each part is visible. No tracing backend yet? Start Jaeger now, so the first run already lands somewhere: ```sh wrap docker run --rm -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.22.0 ``` 3. ### Extend your viem clients ```ts import { createPublicClient, createWalletClient, http } from 'viem'; import { baseSepolia } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; const hashspan = withHashspan(); const chain = baseSepolia; // account: any viem Account, e.g. privateKeyToAccount(process.env.PRIVATE_KEY) const wallet = createWalletClient({ account, chain, transport: http() }) .extend(hashspan); const reader = createPublicClient({ chain, transport: http() }) .extend(hashspan); // Inside an agent tool, while the framework's span is active: const hash = await wallet.sendTransaction({ to, value }); // send span await reader.waitForTransactionReceipt({ hash }); // confirm span ``` The confirm span links back to the send span. Reuse the same `withHashspan()` result for every client of one agent, so confirmations link to their sends. 4. ### Flush before the process exits A confirm span can end after your code moves on, for example while a revert reason is decoded. Short-lived scripts and serverless functions should flush first: ```ts await hashspan.flush(); await provider.shutdown(); ``` 5. ### Open your traces Find your service in your tracing backend, [localhost:16686](http://localhost:16686) for the Jaeger from step 2, and expand a tool span. ## A complete example on a local chain One file that sends a transaction on a local [Anvil](https://www.getfoundry.sh/anvil) chain and traces it. Anvil comes with Foundry: if `anvil --version` fails, install it with `curl -L https://foundry.paradigm.xyz | bash`, then run `foundryup` in a new terminal ([PATH setup](https://hashspan.dev/docs/x402-tutorial#start-a-local-chain-and-jaeger)). Start from an empty project: ```sh wrap mkdir my-agent && cd my-agent npm init -y && npm pkg set type=module npm install @hashspan/viem @opentelemetry/api viem @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources @opentelemetry/semantic-conventions tsx ``` Add `telemetry.ts` from step 2, then this file next to it: ```ts title="agent.ts" import { provider } from './telemetry.js'; import { trace } from '@opentelemetry/api'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; import { withHashspan } from '@hashspan/viem'; // Anvil's first test account. Its key is public: never use it on a real chain. const account = privateKeyToAccount( '0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80', ); const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const hashspan = withHashspan({ agent: { name: 'my-agent' } }); const wallet = createWalletClient({ account, chain: foundry, transport: http() }) .extend(hashspan); const reader = createPublicClient({ chain: foundry, transport: http() }) .extend(hashspan); // Stands in for your agent framework's tool span. await trace.getTracer('my-agent').startActiveSpan('execute_tool pay_vendor', async (tool) => { const hash = await wallet.sendTransaction({ to, value: parseEther('0.01') }); await reader.waitForTransactionReceipt({ hash }); tool.end(); }); await hashspan.flush(); await provider.shutdown(); ``` Start a chain and Jaeger, then run it: ```sh anvil # terminal 1: local chain on :8545 docker run --rm -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.22.0 # terminal 2 npx tsx agent.ts # terminal 3 ``` In Jaeger, pick `my-agent`: `execute_tool pay_vendor` has a `send 31337` and a `confirm 31337` span with `blockchain.tx.status = success`. ## What you should see Each transaction shows up as a `send` and a `confirm` span under the tool that sent it. A reverted transaction is marked as an error and carries the decoded reason: Figure: Jaeger showing a confirm 31337 span marked as an error, with blockchain.tx.status reverted, the fee, gas used and the decoded revert reason WithdrawalLimitExceeded A reverted confirm span in Jaeger, from the example agent in the [local lab](https://hashspan.dev/docs/local-lab). ## Next - Run the full example agent with no API key: [Local lab](https://hashspan.dev/docs/local-lab). - Building with the Vercel AI SDK: [Trace the transactions of an AI SDK agent](https://hashspan.dev/docs/recipes/ai-sdk). - What each viem action records, with the spans from a real run: [viem OpenTelemetry instrumentation](https://hashspan.dev/docs/viem-opentelemetry). - Options, background confirmation and replaced transactions: [viem adapter](https://hashspan.dev/docs/adapters/viem). - Every span and attribute: [Semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions). - What tracing costs per transaction, measured: [tracing overhead benchmark](https://hashspan.dev/docs/benchmarks/tracing-overhead). --- # Trace the transactions of an AI SDK agent Source: https://hashspan.dev/docs/recipes/ai-sdk The AI SDK's OpenTelemetry integration records every tool call. In the run below, the withdrawal reverted on chain, and `execute_tool withdraw_from_vault` still has no error on it. The tool returned normally: a reverted receipt is a value, not an exception. hashspan adds what happened to the transaction, as child spans of the tool that sent it: ```text invoke_agent ├─ step 1 │ ├─ chat │ └─ execute_tool pay_vendor │ ├─ send 31337 │ └─ confirm 31337 │ blockchain.tx.status=success └─ step 2 ├─ chat └─ execute_tool withdraw_from_vault (no error) ├─ send 31337 └─ confirm 31337 ✕ error blockchain.tx.status=reverted blockchain.tx.revert.reason=WithdrawalLimitExceeded(…) ``` The withdrawal tool sets its own gas limit, so the transaction is mined and reverts on chain. Without one, viem's gas estimate fails before sending: the `send` span records the error and the tool span is marked as an error too. This page sets that up with AI SDK 7 (`ai` 7, `@ai-sdk/otel` 1) and viem. You need Node.js 22.3 or later. ## Set it up 1. ### Install the packages ```sh npm install ai @ai-sdk/otel @hashspan/viem viem zod \ @opentelemetry/api \ @opentelemetry/sdk-trace-node \ @opentelemetry/sdk-trace-base \ @opentelemetry/exporter-trace-otlp-proto \ @opentelemetry/resources \ @opentelemetry/semantic-conventions ``` Add the package for your model provider too, such as `@ai-sdk/openai`. 2. ### Register one tracer provider for both The AI SDK and hashspan both use the global OpenTelemetry API, so one provider receives the agent's spans and the transaction spans: ```ts title="telemetry.ts" import { OpenTelemetry } from '@ai-sdk/otel'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto'; import { resourceFromAttributes } from '@opentelemetry/resources'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'; import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'; import { registerTelemetry } from 'ai'; export const provider = new NodeTracerProvider({ resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'treasury-agent', }), // OTLPTraceExporter honours OTEL_EXPORTER_OTLP_ENDPOINT spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())], }); provider.register(); // The AI SDK's spans (invoke_agent, execute_tool, ...) // go through the same provider. registerTelemetry(new OpenTelemetry()); ``` 3. ### Send from extended viem clients in your tools Extend the clients your tools send with. The tools need no tracing code: the AI SDK runs each tool inside its `execute_tool` span, and hashspan starts its spans in whatever span is active. ```ts title="tools.ts" import { withHashspan } from '@hashspan/viem'; import { tool } from 'ai'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { baseSepolia } from 'viem/chains'; import { z } from 'zod'; // One result for every client of this agent, // so confirm spans link to their sends. export const hashspan = withHashspan({ agent: { name: 'treasury-agent' }, }); const chain = baseSepolia; // account: any viem Account, such as // privateKeyToAccount(key, { nonceManager }) const wallet = createWalletClient({ account, chain, transport: http() }) .extend(hashspan); const reader = createPublicClient({ chain, transport: http() }) .extend(hashspan); const vendor = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; export const tools = { pay_vendor: tool({ description: 'Pay the vendor an amount of ETH.', inputSchema: z.object({ amountEth: z.string() }), execute: async ({ amountEth }) => { const hash = await wallet.sendTransaction({ to: vendor, value: parseEther(amountEth), }); const receipt = await reader.waitForTransactionReceipt({ hash }); // Tell the model when it reverted. return { hash, status: receipt.status }; }, }), }; ``` 4. ### Run the agent, then flush Import `telemetry.ts` first. A confirm span can end after the agent returns, for example while a revert reason is decoded, so a script flushes before it exits. A long-running server doesn't need `flush()`; call it only on shutdown. ```ts title="agent.ts" import { provider } from './telemetry.js'; import { stepCountIs, ToolLoopAgent } from 'ai'; import { hashspan, tools } from './tools.js'; const agent = new ToolLoopAgent({ model, // any AI SDK model // becomes gen_ai.agent.name on the invoke_agent span telemetry: { functionId: 'treasury-agent' }, instructions: 'You manage a small treasury.', tools, stopWhen: stepCountIs(5), }); await agent.generate({ prompt: 'Pay the vendor 0.01 ETH.' }); await hashspan.flush(); await provider.shutdown(); ``` `generateText` and `streamText` with `tools` give the same tree. Open your tracing backend and expand a tool span. No backend yet? `docker run --rm -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.22.0` starts Jaeger on [localhost:16686](http://localhost:16686). Other backends are on [Backends](https://hashspan.dev/docs/backends). Figure: Jaeger showing a confirm 31337 span marked as an error, with blockchain.tx.status reverted, the fee, gas used and the decoded revert reason WithdrawalLimitExceeded The reverted confirm span from the repository's AI SDK example agent, in Jaeger. ## Two tool calls in one step can share a nonce A model can ask for several tools in one step, and the AI SDK runs them at the same time. If two of them send from the same account, viem can read the same nonce for both. On a local Anvil chain, one of the two sends failed: ```text execute_tool pay_vendor ✕ └─ send 31337 ✕ error.type=TransactionExecutionError execute_tool withdraw_from_vault ├─ send 31337 └─ confirm 31337 ``` Give the account viem's nonce manager, which hands out nonces in order within the process: ```ts import { nonceManager, privateKeyToAccount } from 'viem/accounts'; const key = process.env.PRIVATE_KEY as `0x${string}`; const account = privateKeyToAccount(key, { nonceManager }); ``` With it, both transactions in that run were mined. ## Tools that return before the receipt A tool that returns the hash right away, so the model can move on, never waits for a receipt. Background confirmation records the confirm span anyway, still under that tool's span: ```ts export const hashspan = withHashspan({ agent: { name: 'treasury-agent' }, confirm: { mode: 'background' }, }); // in the tool: execute: async ({ amountEth }) => ({ hash: await wallet.sendTransaction({ to: vendor, value: parseEther(amountEth), }), }), ``` `hashspan.flush()` before exit waits for those background confirmations too. Timeouts and polling are covered in [background confirmation](https://hashspan.dev/docs/adapters/viem#background-confirmation). ## Name the agent on the transaction spans `functionId` sets `gen_ai.agent.name` on the AI SDK's `invoke_agent` span. It does not reach hashspan's spans. Set the same name in hashspan's `agent` option, as in `tools.ts` above, and every `send` and `confirm` span carries `gen_ai.agent.name`. Your backend can then find one agent's reverted transactions with a single attribute search, without joining spans. ## Using `LegacyOpenTelemetry`? With `registerTelemetry(new LegacyOpenTelemetry())`, the AI SDK names its spans `ai.generateText` and `ai.toolCall` instead. hashspan's spans nest under `ai.toolCall` the same way. AI SDK 6 and its `experimental_telemetry` option have not been checked with hashspan. ## Next - A runnable AI SDK agent that pays a vendor and hits a reverting vault, with no API key: [examples/ai-sdk-agent on GitHub](https://github.com/selimaytac/hashspan/tree/main/examples/ai-sdk-agent), or run it in the [local lab](https://hashspan.dev/docs/local-lab). - Every option of `withHashspan()`: [viem adapter](https://hashspan.dev/docs/adapters/viem). - Sending the trace to Langfuse, Grafana Tempo or Honeycomb: [Backends](https://hashspan.dev/docs/backends). --- # Run hashspan behind an OpenTelemetry Collector Source: https://hashspan.dev/docs/recipes/collector Send everything to a Collector, then decide there what to keep. This page has a config we ran against real hashspan traces, and what it kept. The run wrote to a file exporter; the page shows `otlphttp`, so swap in your own backend. ## The config Tested with `otel/opentelemetry-collector-contrib` 0.163.1. Both processors used here come with the contrib build, not the core one. ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 processors: # Addresses are public and few: an unkeyed hash of one can be reversed by hashing candidates. Drop them here, or # use hashspan's own address modes in the SDK. transform/drop-addresses: error_mode: ignore trace_statements: - delete_matching_keys(span.attributes, "^blockchain\\.(tx\\.(from|to|fee_asset|authorization\\.addresses)|payment\\.(payer|recipient|asset)|call_batch\\.sender|user_operation\\.(sender|entry_point|paymaster))$") tail_sampling: decision_wait: 5s decision_cache: sampled_cache_size: 200000 policies: - name: failed type: status_code status_code: { status_codes: [ERROR] } - name: moved-money type: ottl_condition ottl_condition: error_mode: ignore span: - span.attributes["blockchain.operation.name"] != nil - name: everything-else type: probabilistic probabilistic: { sampling_percentage: 10 } exporters: otlphttp: endpoint: https://your-backend.example:4318 service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, transform/drop-addresses] exporters: [otlphttp] ``` In your app, keep head sampling at 100% and export everything to the Collector. Tail sampling needs every span of a trace to decide. Metrics are recorded for every transaction, whatever a sampler decides. ## What did it keep? We sent 200 agent runs through it: 20 that sent a transaction, 10 that failed without one, and 170 plain ones. | Runs | Sent | Kept | |---|---|---| | With a transaction | 20 | 20, each with its send and confirm span | | Failed, no transaction | 10 | 10 | | Plain | 170 | 13, close to the 10% sample | No address attribute was left in the output. ## Why a policy on `blockchain.*`, not only on errors? Because a failed transaction isn't an error yet when the receipt takes longer than `decision_wait`. hashspan's `confirm` span ends when the receipt arrives, often after the rest of the trace. A transaction that reverts on chain only shows as an error on that span. The tool span that sent it ended fine. We tested this with 12-second blocks and a 5-second `decision_wait`. Each run sent one transaction that reverted. The confirm span ended about 12 seconds after the send, 7 seconds after the decision: | Policies | Reverted payments kept, of 20 | |---|---| | `failed` + 10% of the rest | 3 | | `failed` + `moved-money` + 10% of the rest | 20, whole, with the revert reason | The `send` span carries `blockchain.operation.name` from its start, so the `moved-money` policy keeps the trace before the confirm arrives. In a run of 10 successful transactions, all 10 confirm spans arrived after the decision and were kept with their traces. ## Notes - **`decision_wait`** is how long the Collector waits after a trace's first span. Longer waits hold more traces in memory. With the `moved-money` policy, a short wait still keeps transaction traces. - **`decision_cache`** keeps the decision for spans that arrive after the trace left the processor's buffer (`num_traces`, 50,000 by default). The processor's [README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor) asks for a cache much larger than `num_traces`. - **Addresses:** the regex drops the address attributes. Addresses can also sit inside other values, such as a revert reason or `error.type`; only hashspan's address mode `off` covers those ([address modes](https://hashspan.dev/docs/reference/semantic-conventions#privacy)). An unkeyed hash of a public address can be reversed by hashing candidates. - **Background confirmations** need the app to run until the receipt arrives. A short-lived process should call `flush()` first, or its open confirm spans end as `timeout`. [Sampling](https://hashspan.dev/learn/otel-sampling) explains head and tail sampling with a simulation. --- # Agent fees and confirmation times in Grafana Source: https://hashspan.dev/docs/recipes/fees-in-grafana How much did the agent spend on gas today, and how long did its transactions take to land? hashspan answers both with three histograms, recorded next to its spans: send duration, confirmation duration and fee. This page puts them in Grafana, first in the local lab, then from your own agent. ## What gets recorded - **Send duration:** `blockchain.client.send.duration`, in Prometheus `blockchain_client_send_duration_seconds`. Labels: the chain, and `error.type` when the send failed. - **Confirmation duration:** `blockchain.client.confirmation.duration`, in Prometheus `blockchain_client_confirmation_duration_seconds`. Labels: the chain and the outcome. - **Fee:** `blockchain.client.fee`, in Prometheus `blockchain_client_fee`. Labels: the chain, `blockchain.tx.status` and, since 0.11, `blockchain.fee.payer` on samples whose fee the sender did not pay (`facilitator` for a payment's settlement, `paymaster` for a user operation). The labels are low-cardinality on purpose: no address, hash or agent identity. Prometheus turns `blockchain.chain.id` into `blockchain_chain_id` and `service.name` into the `job` label. Every field is described on the [metrics reference](https://hashspan.dev/docs/reference/attributes/metrics). ## Try it in the local lab 1. Clone the repository and install it, as in the [local lab](https://hashspan.dev/docs/local-lab). 2. Start Jaeger, Prometheus and Grafana with the dashboard: ```sh make lab-metrics ``` 3. Run the example agent a few times. When Prometheus is up, `make demo` sends the metrics to it too. ```sh make demo ``` 4. Open http://localhost:3000. The hashspan dashboard is the home page: sends, confirmation p95, median fee, latency and fee per chain, and confirmation outcomes. ## From your own agent hashspan records the histograms through the global meter provider, so the only setup is an OpenTelemetry metrics SDK with an OTLP exporter: ```sh npm install @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-proto @opentelemetry/resources ``` Import this file first in your agent's entry point, before anything sends: ```ts title="metrics.ts" import { metrics } from '@opentelemetry/api'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto'; import { resourceFromAttributes } from '@opentelemetry/resources'; import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; // The exporter reads OTEL_EXPORTER_OTLP_METRICS_ENDPOINT. export const meterProvider = new MeterProvider({ resource: resourceFromAttributes({ 'service.name': 'my-agent' }), readers: [new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() })], }); metrics.setGlobalMeterProvider(meterProvider); ``` Point it at Prometheus's OTLP receiver, which needs `--web.enable-otlp-receiver`: ```sh wrap OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:9090/api/v1/otlp/v1/metrics node agent.js ``` A short-lived script must call `await meterProvider.shutdown()` before it exits, or its last samples are lost. If it exports only once, Prometheus also needs `--enable-feature=created-timestamp-zero-ingestion` for `increase()` and `rate()` to see it, as the lab sets. Then import the [dashboard file](https://github.com/selimaytac/hashspan/blob/main/docker/grafana/dashboards/hashspan.json) into Grafana, as the [backends page](https://hashspan.dev/docs/backends#grafana-dashboard-for-the-metrics) explains. ## Queries to start from What the agent spent per chain over the last day, in wei. Samples with a `blockchain_fee_payer` label are fees someone else paid, so the query leaves them out: ```text wrap sum by (blockchain_chain_id) (increase(blockchain_client_fee_sum{blockchain_fee_payer=""}[1d])) ``` How many transactions reverted: ```text wrap sum by (blockchain_chain_id) (increase(blockchain_client_fee_count{blockchain_tx_status="reverted"}[1d])) ``` Confirmation time at the 95th percentile: ```text wrap histogram_quantile(0.95, sum by (le, blockchain_chain_id) (rate(blockchain_client_confirmation_duration_seconds_bucket[1h]))) ``` These are estimates, not counts. `increase()` extrapolates to the edges of the range, and a short-lived process starts its counters again on every run: in a local test, three reverted transactions from three runs of the demo showed as 3.3 over 15 minutes and 7.6 over a day. For an exact count, count the confirm spans in your trace backend. ## What it doesn't do The fee histogram has no agent label, so per-agent spend comes from the spans: each confirm span carries `blockchain.tx.fee` and `gen_ai.agent.name`. Alerting is your backend's job; hashspan only records. --- # Find your agent's reverted transactions Source: https://hashspan.dev/docs/recipes/find-reverted-transactions Your agent ran for a day. Which of its transactions reverted, and why? Each confirm span hashspan records carries the outcome as `blockchain.tx.status`, so a reverted transaction is one attribute search away. ## Search by status In the Jaeger UI, pick the agent's service and type this into **Attributes**: ```text blockchain.tx.status=reverted ``` Every trace in the results has at least one reverted transaction. Open one: the `confirm` span with the error mark is the reverted one, under the tool call that sent it. ## Narrow it to one agent Several agents in one service? Add the agent's name, separated by a space: ```text wrap blockchain.tx.status=reverted gen_ai.agent.name=treasury-agent ``` hashspan puts `gen_ai.agent.name` on its spans from the tracker's `agent` option or from Baggage ([agent identity](https://hashspan.dev/docs/reference/semantic-conventions#attributes)). ## Read why it reverted The reverted confirm span records, in this example from the local lab: - `blockchain.tx.status`: `reverted` - `error.type`: `reverted` - `blockchain.tx.revert.reason`: `WithdrawalLimitExceeded(100000000000000000, 1000000000000000000)` The revert reason is decoded by replaying the transaction, so custom errors show their name and arguments when the ABI is known ([how](https://hashspan.dev/docs/reference/attributes/transactions#blockchaintxrevertreason)). ## The same search over the API Jaeger's API takes the attributes as a JSON object. This returns the traces from the last hour: ```sh curl -sG http://localhost:16686/api/v3/traces \ --data-urlencode 'query.service_name=treasury-agent' \ --data-urlencode "query.start_time_min=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" \ --data-urlencode "query.start_time_max=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --data-urlencode 'query.attributes={"blockchain.tx.status":"reverted"}' ``` The `date -v` flag is macOS's; on Linux use `date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ`. ## Other values worth a search - `blockchain.tx.status=replaced`: the agent's transaction was replaced by another with the same nonce; `blockchain.tx.replacement.hash` names the one that was mined. - `error.type=timeout`: the wait gave up before a receipt came. No `blockchain.tx.status` is recorded then, since the outcome on chain is not known. Every attribute is listed in the [attribute reference](https://hashspan.dev/docs/reference/attributes/transactions). --- # Call batch attributes Source: https://hashspan.dev/docs/reference/attributes/call-batches Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. Attributes of the send and confirm spans of EIP-5792 call batches: `blockchain.call_batch.*`. *Default* is what is recorded without configuration; addresses follow the address mode described under [privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy). ## `blockchain.call_batch.id` | Type | Spans | Default | |---|---|---| | string | send, confirm | on | the batch id the wallet returned for EIP-5792 `wallet_sendCalls` (`0x`-prefixed hex, at most 8194 characters), truncated after 256 characters; with the chain id, it identifies the batch; hashed like an address in `hashed` address mode and not recorded in `off` mode, see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `blockchain.call_batch.sender` | Type | Spans | Default | |---|---|---| | string | send | raw | address of the account the calls are sent from, subject to address mode ## `blockchain.call_batch.call_count` | Type | Spans | Default | |---|---|---| | int | send | on | number of calls in the batch ## `blockchain.call_batch.status` | Type | Spans | Default | |---|---|---| | string | confirm | on | the batch's outcome from chain data: `success` | `reverted` | `partially_reverted`; absent for other outcomes, which `error.type` describes ## `blockchain.call_batch.status_code` | Type | Spans | Default | |---|---|---| | int | confirm | on | the EIP-5792 status code the wallet reported, e.g. `200` confirmed, `500` reverted; spans only, never a metric attribute ## `blockchain.call_batch.atomic` | Type | Spans | Default | |---|---|---| | boolean | confirm | on | whether the wallet ran the calls atomically ## `blockchain.call_batch.transaction_hashes` | Type | Spans | Default | |---|---|---| | string[] | confirm | on | hashes of the transactions whose receipts the wallet reported for the batch, de-duplicated, at most 64 --- # Common attributes Source: https://hashspan.dev/docs/reference/attributes/common Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. Attributes shared by every span or metric: the chain, the operation and `error.type`. *Default* is what is recorded without configuration; addresses follow the address mode described under [privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy). ## `blockchain.system.name` | Type | Spans | Default | |---|---|---| | string | send, confirm, payment | on | `evm` ## `blockchain.chain.id` | Type | Spans | Default | |---|---|---| | int | all | on | EIP-155 chain id, e.g. `8453` ## `blockchain.operation.name` | Type | Spans | Default | |---|---|---| | string | send, confirm, payment | on | `send` | `confirm` | `payment` ## `blockchain.operation.subject` | Type | Spans | Default | |---|---|---| | string | none (metrics only) | on | on [metrics](https://hashspan.dev/docs/reference/semantic-conventions#metrics) of user operations: `user_operation`; of call batches: `call_batch`; absent on those of transactions ## `blockchain.fee.payer` | Type | Spans | Default | |---|---|---| | string | none (metrics only) | on | on `blockchain.client.fee` samples whose fee the sender of the traced transaction or operation did not pay: `facilitator` for a payment's settlement transaction that another party sent (x402), `paymaster` for a user operation a paymaster paid for, `sponsor` for a transaction whose receipt names another account as its fee payer (a Tempo `feePayer`); a payment's settlement sent by another party keeps `facilitator`, also when its fee was sponsored, while one the payer sent (MPP) is `sponsor` when the server paid its fee; absent when the sender paid ([ADR 0020](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0020-metrics.md)) ## `blockchain.fee.denomination` | Type | Spans | Default | |---|---|---| | string | none (metrics only) | on | on `blockchain.client.fee` samples whose fee was paid in a token rather than the chain's native currency: `token`, a constant, never an address or a value read from the chain; set whenever `blockchain.tx.fee_asset` is known, in every address mode; absent for the native currency, Arc's USDC included ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md)) ## `error.type` | Type | Spans | Default | |---|---|---| | string | all | on | see *Span status* and [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses); reused from OpenTelemetry general conventions --- # Metrics Source: https://hashspan.dev/docs/reference/attributes/metrics Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. The histograms the tracker records. Bucket boundaries, and how `error.type` is kept on metrics, are under [metrics](https://hashspan.dev/docs/reference/semantic-conventions#metrics). ## `blockchain.client.send.duration` | Instrument | Unit | |---|---| | histogram | `s` | **Attributes:** chain; `blockchain.operation.subject` for a user operation or call batch; `error.type` if the send failed **Recorded when:** a send span ends with a hash or id, or fails: from the start of the sending call until then ## `blockchain.client.confirmation.duration` | Instrument | Unit | |---|---| | histogram | `s` | **Attributes:** chain; `blockchain.operation.subject` for a user operation or call batch; the outcome from chain data, `blockchain.tx.status` (for a user operation, `blockchain.user_operation.success`; for a call batch, `blockchain.call_batch.status`), else `error.type` (`timeout`, `not_on_chain`, an adapter's error type, an error class name, or `_OTHER`) **Recorded when:** a confirm span ends: from the start of the wait until the receipt or batch status, a replacement, a timeout or a failure; not for a call batch that ended while still pending, nor for a transaction whose receipt was pending ## `blockchain.client.fee` | Instrument | Unit | |---|---| | histogram | `{wei}` | **Attributes:** chain; `blockchain.operation.subject` for a user operation; `blockchain.tx.status` (for a user operation, `blockchain.user_operation.success`); `blockchain.fee.payer` when someone other than the sender paid; `blockchain.fee.denomination` when the fee was paid in a token **Recorded when:** a receipt with an effective gas price is recorded: `blockchain.tx.fee` as a number, without the OP Stack operator fee, which no metric records; for a user operation, a receipt with its cost: `blockchain.user_operation.gas.cost` --- # Payment attributes Source: https://hashspan.dev/docs/reference/attributes/payments Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. Attributes of x402 and MPP payment spans: `blockchain.payment.*`, `x402.*` and `mpp.*`. *Default* is what is recorded without configuration; addresses follow the address mode described under [privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy). ## `blockchain.payment.protocol` | Type | Spans | Default | |---|---|---| | string | payment | on | `x402` | `mpp`; see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `blockchain.payment.payer` | Type | Spans | Default | |---|---|---| | string | payment | raw | address that pays, subject to address mode; the settlement's payer only when the payer knew none ## `blockchain.payment.recipient` | Type | Spans | Default | |---|---|---| | string | payment | raw | address that is paid, subject to address mode ## `blockchain.payment.asset` | Type | Spans | Default | |---|---|---| | string | payment | raw | contract address of the token paid with, subject to address mode ## `blockchain.payment.amount` | Type | Spans | Default | |---|---|---| | string | payment | on | amount in the asset's smallest unit, decimal string; the settlement's amount only when the payer knew none ## `blockchain.payment.settled_amount` | Type | Spans | Default | |---|---|---| | string | payment | on | amount the settling party reports it settled, decimal string, e.g. less than the authorized maximum with x402 `upto`; for x402 `batch-settlement`, the request's charge against the channel (`extra.chargedAmount`), never the deposit; as reported, not checked ## `blockchain.payment.verified` | Type | Spans | Default | |---|---|---| | boolean | payment | on | whether the settlement transaction carries this payment (its receipt and, for Permit2, its input; for an MPP sponsored pull, also its sender and the nonce the payer signed), checked by the adapter from the payer's own data; absent when no check was possible ([ADR 0017](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0017-x402-payment-verification.md)) ## `blockchain.payment.status` | Type | Spans | Default | |---|---|---| | string | payment | on | `settled` | `pending` | `failed` ## `x402.scheme` | Type | Spans | Default | |---|---|---| | string | payment | on | x402 payment scheme, e.g. `exact`; see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `x402.resource` | Type | Spans | Default | |---|---|---| | string | payment | origin | the resource paid for, per the tracker's `paymentResource` mode: `origin` (default) records scheme, host and port only, `path` the URL without query string, fragment or user info, `off` nothing; at most 512 characters, and nothing for text whose user info contains `?` or `#`; see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `mpp.method` | Type | Spans | Default | |---|---|---| | string | payment | on | MPP payment method, e.g. `tempo`; see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `mpp.intent` | Type | Spans | Default | |---|---|---| | string | payment | on | MPP intent, e.g. `charge`; see [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses) ## `mpp.mode` | Type | Spans | Default | |---|---|---| | string | payment | on | how an MPP charge is settled: `push` (the payer broadcasts) | `pull` (the server broadcasts what the payer signed) ## `mpp.challenge.id` | Type | Spans | Default | |---|---|---| | string | payment | on | id of the MPP challenge paid, chosen by the server: 1 to 128 characters of `A-Z a-z 0-9 - . _ ~`, else not recorded (see also [values a remote party chooses](https://hashspan.dev/docs/reference/semantic-conventions#values-a-remote-party-chooses)); spans only, never a metric attribute ([ADR 0029](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0029-mpp-payments.md)) ## `mpp.resource` | Type | Spans | Default | |---|---|---| | string | payment | origin | the resource paid for, as `x402.resource` records it --- # Transaction attributes Source: https://hashspan.dev/docs/reference/attributes/transactions Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. Attributes of the send and confirm spans of transactions: `blockchain.tx.*`, `blockchain.contract.*` and `blockchain.block.number`. *Default* is what is recorded without configuration; addresses follow the address mode described under [privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy). ## `blockchain.tx.hash` | Type | Spans | Default | |---|---|---| | string | send, confirm, payment | on | `0x`-prefixed tx hash; on a payment span, the settling transaction's, when reported, or from the span's start when the payer knows it (MPP), and never for x402 `batch-settlement`, whose reported transaction is the channel's deposit ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0013-x402-payments.md)); on a user operation's confirm span, the bundle transaction's; absent on a user operation's send span ## `blockchain.tx.from` | Type | Spans | Default | |---|---|---| | string | send | raw | sender address, subject to address mode ## `blockchain.tx.to` | Type | Spans | Default | |---|---|---| | string | send | raw | recipient / contract address, subject to address mode ## `blockchain.tx.value` | Type | Spans | Default | |---|---|---| | string | send | on | value in wei, decimal string ## `blockchain.tx.nonce` | Type | Spans | Default | |---|---|---| | int | send | on | sender nonce, when the sending call passes one (a nonce the wallet or viem picks is not known to the adapter) ## `blockchain.tx.authorization.count` | Type | Spans | Default | |---|---|---| | int | send | on | number of EIP-7702 authorizations a type 4 transaction carries ## `blockchain.tx.authorization.addresses` | Type | Spans | Default | |---|---|---| | string[] | send | raw | delegated contract address of each well-formed authorization, subject to address mode, at most 64; `0x000...0` clears a delegation ## `blockchain.tx.authorization.chain_ids` | Type | Spans | Default | |---|---|---| | int[] | send | on | chain id of each well-formed authorization, in the order of the addresses, at most 64; `0` means valid on every chain ## `blockchain.contract.function.name` | Type | Spans | Default | |---|---|---| | string | send | on | decoded function name when an ABI is known ## `blockchain.contract.function.selector` | Type | Spans | Default | |---|---|---| | string | send | on | 4-byte selector, e.g. `0xa9059cbb` ## `blockchain.contract.function.arguments` | Type | Spans | Default | |---|---|---| | string | send | off (opt-in) | decoded call arguments as a JSON array, e.g. `["0x2222...2222","1000000"]`: bigints as decimal strings, addresses per address mode, truncated after 4096 characters. Only own enumerable data properties are serialized; `toJSON()` and getters are never called; binary data (typed arrays, `ArrayBuffer`) as `0x` hex ## `blockchain.tx.status` | Type | Spans | Default | |---|---|---| | string | confirm | on | from chain data: `success` | `reverted` | `replaced` ## `blockchain.block.number` | Type | Spans | Default | |---|---|---| | int | confirm | on | inclusion block; for a user operation, the bundle transaction's; for a call batch, the highest among its receipts ## `blockchain.tx.gas.used` | Type | Spans | Default | |---|---|---| | int | confirm | on | gas used ## `blockchain.tx.effective_gas_price` | Type | Spans | Default | |---|---|---| | string | confirm | on | price per gas in the chain's fee unit (see the fee table below), decimal string; see the `fee` row for when it is omitted ## `blockchain.tx.l1_fee` | Type | Spans | Default | |---|---|---| | string | confirm | on | L1 data fee on OP-stack chains, in the chain's fee unit; see the `fee` row for when it is omitted ## `blockchain.tx.fee` | Type | Spans | Default | |---|---|---| | string | confirm | on | `gas.used × effective_gas_price + l1_fee`, in the chain's fee unit: wei of the native currency, or the token `fee_asset` names (see the fee table below); omitted if the gas price is unknown. The OP Stack operator fee (Isthmus and later) is not included: it is recorded apart as `operator_fee`. Fee attributes come from the sealed receipt, never a flashblocks preconfirmation, and are omitted if only a preconfirmation was seen (see [ADR 0024](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0024-sealed-receipt-fees.md)). On the confirm span of a payment's settlement, the fee is the one the transaction's sender paid (for x402, the facilitator; for MPP, the payer), unless the receipt names a fee payer other than the sender (a Tempo `feePayer`, which every Tempo receipt names and which is often the sender itself) ## `blockchain.tx.operator_fee` | Type | Spans | Default | |---|---|---| | string | confirm | on | OP Stack operator fee (Isthmus and later), wei, decimal string; not included in `blockchain.tx.fee`. Recorded only when the receipt carries `operatorFeeScalar` or `operatorFeeConstant`, which a node adds when the chain charges the fee; the viem adapter then reads it from the GasPriceOracle's `getOperatorFee(gasUsed)` at the receipt's block, with one `eth_call`. Like the other fee attributes, it comes from the sealed receipt only, and is omitted for a deposit transaction, which pays none ## `blockchain.tx.fee_asset` | Type | Spans | Default | |---|---|---| | string | confirm | raw | contract address of the token the fee was paid in, subject to address mode, as the chain reports it (a Celo fee currency, which for a 6-decimal token is its 18-decimal adapter contract; a Tempo fee token); absent when the fee is in the native currency. Recorded with the fee attributes, when `effective_gas_price` is. Taken from the receipt when it names one, else from the send: the viem adapter reads Celo's `feeCurrency` from the sending call, never from `sendRawTransaction`, and Tempo's `feeToken` from a receipt of type `0x76`, or of another type that also names a `feePayer` ([ADR 0028 amendment](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md#amendment-2026-10-18-proposed-tempo-receipts-of-any-type)); a replacing transaction records its own, never the replaced one's ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md)) ## `blockchain.tx.revert.reason` | Type | Spans | Default | |---|---|---| | string | confirm | on | decoded revert reason when available, also of a reverted user operation: the `Error(string)` message, `Panic(0x..)`, `ErrorName(arg, ...)` for custom errors with a known ABI, else the 4-byte error selector. See [ADR 0005](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0005-revert-reason-replay.md) ## `blockchain.tx.replacement.hash` | Type | Spans | Default | |---|---|---| | string | confirm | on | on a `replaced` confirm span: hash of the mined transaction that replaced it ## `blockchain.tx.replacement.reason` | Type | Spans | Default | |---|---|---| | string | confirm | on | on a `replaced` confirm span: `repriced` | `cancelled` | `replaced`, as reported by the instrumented library; omitted when it reported none ## `blockchain.tx.wait.confirmations` | Type | Spans | Default | |---|---|---| | int | confirm | on | on a transaction's confirm span: the confirmations (blocks, counting the receipt's own) that the wait which ended the span asked for, as the library applies them: a viem wait's or a CDP network-scoped wait's `confirmations` when it is a positive safe integer; `1` when it is omitted, `0`, negative or `NaN`, as viem then waits for one, and for background confirmation and `watch()`; not recorded for any other value, nor when the wait took no count (sync actions, user operations, call batches). A count of blocks is not finality: on a rollup it counts the sequencer's blocks, and block times differ by chain, so read it with `blockchain.chain.id`. Span only, never a metric attribute (see one confirm span per transaction for which wait ends the span) --- # User operation attributes Source: https://hashspan.dev/docs/reference/attributes/user-operations Generated from the semantic conventions These rows are generated from the semantic conventions at @hashspan/core@1.3.1, the version on npm. They are a draft: names can still change. Attributes of the send and confirm spans of ERC-4337 user operations: `blockchain.user_operation.*`. *Default* is what is recorded without configuration; addresses follow the address mode described under [privacy](https://hashspan.dev/docs/reference/semantic-conventions#privacy). ## `blockchain.user_operation.hash` | Type | Spans | Default | |---|---|---| | string | send, confirm | on | `0x`-prefixed user operation hash (`userOpHash`), which identifies the operation with the chain id ## `blockchain.user_operation.sender` | Type | Spans | Default | |---|---|---| | string | send, confirm | raw | address of the smart account, subject to address mode ## `blockchain.user_operation.entry_point` | Type | Spans | Default | |---|---|---| | string | send, confirm | raw | address of the EntryPoint contract, subject to address mode ## `blockchain.user_operation.call_count` | Type | Spans | Default | |---|---|---| | int | send | on | number of calls the operation makes, when the adapter knows them ## `blockchain.user_operation.nonce` | Type | Spans | Default | |---|---|---| | string | confirm | on | the operation's nonce, decimal string (a 192-bit key and a 64-bit sequence number) ## `blockchain.user_operation.success` | Type | Spans | Default | |---|---|---| | boolean | confirm | on | whether the operation's calls succeeded; the bundle transaction can succeed while they revert ## `blockchain.user_operation.gas.used` | Type | Spans | Default | |---|---|---| | int | confirm | on | gas the operation used (`actualGasUsed`) ## `blockchain.user_operation.gas.cost` | Type | Spans | Default | |---|---|---| | string | confirm | on | what the operation paid (`actualGasCost`), wei, decimal string; its share of the bundle, not the bundle transaction's fee; paid by the paymaster when `blockchain.user_operation.paymaster` is recorded ## `blockchain.user_operation.paymaster` | Type | Spans | Default | |---|---|---| | string | confirm | raw | address of the paymaster that paid for the operation, subject to address mode; absent when none paid --- # Semantic conventions Source: https://hashspan.dev/docs/reference/semantic-conventions Synced from GitHub This page mirrors docs/semconv.md at @hashspan/core@1.3.1, the version on npm. Schema version: `0.5.0-dev` · Stability: **development** for everything below. Rationale: [ADR 0003](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0003-attribute-namespace.md). Privacy defaults: [ADR 0004](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0004-privacy-defaults.md). ## Spans | Span name | Kind | Parent | Ends when | |---|---|---|---| | `send {blockchain.chain.id}` | CLIENT | active context (e.g. `execute_tool`) | hash returned (a transaction's, or a user operation's) or send failed | | `confirm {blockchain.chain.id}` | CLIENT | see below | receipt retrieved, timeout or error; links to `send` or `payment` | | `payment {blockchain.chain.id}` | CLIENT | active context (e.g. `execute_tool`) | settlement reported or payment failed | **Send span as the active span.** While the call that sends the transaction runs, the send span is the active span, so spans that wallet, RPC or HTTP instrumentation creates for it nest under the send span. This holds for the viem adapter's clients with a chain and for the CDP adapter, not for viem clients without a chain, whose send span is recorded after the call, and not for x402's paid request ([ADR 0015](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0015-send-span-as-active-context.md)). **Confirm span parent**, in order: an explicitly passed context; otherwise the active span (whatever is waiting for the receipt); otherwise the parent of the `send` span (confirmation in the background); otherwise none. The link to the `send` span is added whenever the transaction was sent through the same tracker within the link TTL (default 10 minutes). With no parent, the confirm span is in a trace of its own, related to the send only by the link; some backends do not keep links ([troubleshooting](https://hashspan.dev/docs/troubleshooting#a-confirm-span-with-no-send-span-next-to-it)). **Replaced transactions.** A receipt is recorded on the confirm span of the transaction that was mined. When a wait for one hash ends with the receipt of another (a transaction with the same sender and nonce replaced it), the confirm span of the awaited hash ends as `replaced`, without block, gas or fee, and the receipt goes to the confirm span of the mined hash. If that span is created for this purpose, it has the same parent and start time as the replaced one and links to it and to both `send` spans when known. Dashboards counting confirmations should exclude `blockchain.tx.status = replaced`. See [ADR 0008](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0008-replaced-transactions.md). A replacement needs the same sender and nonce: where the instrumented library reports replacements (viem), a receipt of another hash that it did not report as one, such as an endpoint's answer for an unrelated transaction, is not recorded, and the confirm span ends with `error.type` `_OTHER`. A replacement viem missed because it took the transaction out of the pool before the wait read it is found by the viem adapter after the wait timed out, from the sender and nonce of its own send, and recorded the same way, without `blockchain.tx.replacement.reason` ([ADR 0008 amendment](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0008-replaced-transactions.md#amendment-2026-10-18-proposed-a-replacement-viem-did-not-report)). **Chain reorganisations.** A confirm span records the receipt its wait ended with and is not revised afterwards: when a reorganisation removes the receipt's block, the span keeps its status and block number, and a later wait for the transaction within the link TTL adds no span, as the transaction stays settled (see one confirm span per transaction, below). A wait with `confirmations` above 1 records the receipt it read last: viem returns the receipt it read first, so once the wait resolved, the confirm span reads the receipt again (viem). If that receipt is missing or in another block, it also reads the block at the height of the caller's receipt: when that block has another hash, a reorganisation during the wait moved the transaction, and the span records the receipt in the new block, or removed it, and the span ends with error status and `error.type` `not_on_chain`, without `blockchain.tx.status`. A node without that block, a block that still has the caller's hash, a failed request or an answer that cannot be read keeps the caller's receipt. The caller's result never changes, and the span ends when the wait resolved ([ADR 0026](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0026-receipt-after-several-confirmations.md)). A transaction removed before a wait, `watch()` or background confirmation read its receipt, and not included again, ends as a timeout. **Payments.** A `payment` span records a payment that the agent authorizes and another party settles on chain, such as an x402 facilitator: the agent signs, but does not send, the settling transaction, so there is no `send` span ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0013-x402-payments.md)). It carries what was paid, to whom, and the settlement. A settlement with a transaction hash makes the payment span the one a confirm span for that hash links to, and whose parent it takes for confirmation in the background, as a `send` span would. For MPP ([ADR 0029](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0029-mpp-payments.md)), the payer is the settling transaction's `from` in both of its modes: in `push` the payer sends it itself, and in `pull` it signs it and the server broadcasts it. The payer pays its fee unless the server sponsors it as fee payer. The adapter passes the transaction's hash when the payer knows it before the settlement is reported (`push`, and a `pull` without a fee payer): the payment span then records `blockchain.tx.hash` from its start, keeps it whatever the server reports, and, when the same tracker recorded the payer's own send of it, starts with a link to that send span. `blockchain.payment.status` is recorded only from the settlement the settling party reported; when the client never learned it, the payment span has none and `error.type` says why. **User operations.** A user operation of an ERC-4337 smart account has no transaction of its own: a bundler includes it in a bundle transaction that the bundler sends, together with other operations. It gets the same `send` and `confirm` spans as a transaction, identified by its chain and `blockchain.user_operation.hash` instead of `blockchain.tx.hash` ([ADR 0021](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0021-user-operations.md)). The send span covers handing the operation to the bundler, until its hash is returned, and has no `blockchain.tx.*` attribute. The confirm span records the operation's own outcome and cost (`blockchain.user_operation.*`), and the bundle transaction's `blockchain.tx.hash` and `blockchain.block.number`; it has no `blockchain.tx.status`, gas or fee, which describe the whole bundle. Parent and link rules are those of transactions, and confirmations of user operations are kept apart from those of transactions: a wait for the bundle transaction's own receipt is a separate confirm span. Replacement attribution does not apply: a bundler that resubmits a bundle keeps the operation's hash. **Call batches.** A batch of calls handed to a wallet with EIP-5792 `wallet_sendCalls` is identified by its chain and the batch id the wallet returned (`blockchain.call_batch.id`), and gets the same `send` and `confirm` spans ([ADR 0022](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0022-call-batches.md)). The send span covers handing the batch to the wallet, until the id is returned. The confirm span ends with the status a wait returned: its outcome (`blockchain.call_batch.status`) and EIP-5792 status code, whether the batch ran atomically, and the hashes of the transactions that carried it; it has no `blockchain.tx.status`, gas or fee, since wallet receipts lack the L1 fee and may be a bundle transaction shared with others. A wait that resolves while the batch is pending withdraws, as a timeout does: the span ends without an outcome only when no other wait is still running, and a later wait gets its own span. Confirmations of batches are kept apart from those of transactions and user operations. Transactions an account sends itself for a batch (viem's fallback to `eth_sendTransaction`) are linked to the batch's send span and confirmed as transactions, with their fees. **One confirm span per transaction and tracker.** Concurrent waits for the same transaction share one confirm span; its parent is determined by the first wait. A receipt from any wait ends it; a timeout or failure ends it only when it is the last wait still running, with that wait's outcome. After a receipt, further waits within the link TTL add no span; after a timeout or failure, a retry gets a new span. The same holds for each user operation and call batch. See [ADR 0007](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0007-confirmation-ownership.md). `blockchain.tx.wait.confirmations` is that of the wait which ended the span: the one whose receipt ended it, or the last to time out or fail. With background confirmation on, the background wait (`1`) usually gets the receipt first, so a caller's wait with `confirmations: 12` for the same transaction is not recorded, as it adds no span of its own. ### Span status | Situation | Span | Status | `error.type` | `blockchain.tx.status` | |---|---|---|---|---| | Transaction hash, user operation hash or call batch id returned | send | unset | none | none | | Signing, simulation, broadcast or handing to a bundler or wallet failed | send | error | the library's error code when the adapter reports one (see below), else error class name, else `_OTHER` | none | | Receipt with status success | confirm | unset | none | `success` | | Receipt with status reverted | confirm | error | `reverted` | `reverted` | | Pending receipt, of a transaction not on chain yet (a Tempo multisig relay below quorum): withdraws the wait, as a timeout does, and ends the span only when no other wait is running | confirm | unset | none | none | | Gave up waiting for the receipt (its timeout, or `flush()` gave up) | confirm | error | `timeout` | none; see below | | A reorganisation during a wait for several confirmations removed the transaction (viem; see [chain reorganisations](#spans)) | confirm | error | `not_on_chain` | none | | Replaced by another transaction (same sender and nonce) | confirm of the replaced hash | unset | none | `replaced` | | Receipt with an invalid transaction hash | confirm | error | `_OTHER` | none | | Receipt of another transaction that is not a reported replacement | confirm | error | `_OTHER` | none | | Receipt with a status other than success or reverted, or one that cannot be read | confirm | error | `_OTHER` | none | | Retrieving the receipt failed | confirm | error | error class name, else `_OTHER` | none | | User operation receipt with success | confirm | unset | none | none; `blockchain.user_operation.success` is `true` | | User operation receipt without success (its calls reverted) | confirm | error | `reverted` | none; `blockchain.user_operation.success` is `false` | | User operation receipt without a success flag (the adapter does not know the outcome) | confirm | unset | none | none | | User operation receipt whose success flag is present but not a boolean | confirm | error | `_OTHER` | none | | User operation failed without a receipt (e.g. an SDK reports `failed`) | confirm | error | the adapter's error type, else error class name, else `_OTHER` | none | | Call batch status 200 (confirmed) | confirm | unset | none | none; `blockchain.call_batch.status` is `success` | | Call batch status 500 (reverted) | confirm | error | `reverted` | none; `blockchain.call_batch.status` is `reverted` | | Call batch status 600 (partially reverted) | confirm | error | `partially_reverted` | none; `blockchain.call_batch.status` is `partially_reverted` | | Call batch status 400 (failed without inclusion) | confirm | error | `failed` | none | | Call batch status with any other code, or none, or one that cannot be read | confirm | error | `_OTHER` | none; `blockchain.call_batch.status_code` keeps an integer code | | Call batch status 100 (a wait that accepted a pending status) | confirm | unset | none | none; no metric sample is recorded | | Payment settled | payment | unset | none | none; `blockchain.payment.status` is `settled` | | Payment settlement pending: transaction known, receipt not seen | payment | unset | none | none; `blockchain.payment.status` is `pending` | | Payment settlement failed | payment | error | the settling party's reason if it is a short identifier (see below), else `_OTHER` | none; `blockchain.payment.status` is `failed` | | Creating the payment failed (e.g. signing it) | payment | error | as for a failed send | none | | Payment response without a settlement | payment | error | `no_settlement` (x402 adapter) | none | | Payment response that cannot be read | payment | error | `_OTHER` | none | | Payment outcome never learned (no response before its authorization expired, or flush gave up) | payment | error | `timeout` | none; a hash the payer knew from the start stays recorded | **Timeouts.** `blockchain.tx.status` describes the transaction as the chain recorded it. A confirm span that gave up waiting has error status and `error.type` `timeout`, and no `blockchain.tx.status`: the transaction's outcome is unknown. Up to 0.4, such spans also recorded `blockchain.tx.status` `timeout`; query `error.type` instead ([ADR 0016](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0016-timeout-is-an-observer-outcome.md)). An adapter whose library reports a stable, machine-readable error code records it as `error.type` of a failed send, if it is a short identifier (`[A-Za-z0-9_.-]`, at most 64 characters); `exception.type` stays the class name. The CDP adapter records the CDP API's error type, e.g. `insufficient_balance`. The viem adapter records the error viem classified the failure as, under the error it throws: `NonceTooLowError` or `InsufficientFundsError` under a `TransactionExecutionError` (or `ContractFunctionExecutionError`), a bundler error under a `UserOperationExecutionError`; when there is none, the thrown class. An error class name is recorded, as `error.type` and as `exception.type`, only if it is such a short identifier too; any other name is recorded as `_OTHER`. Failures with an error object add an `exception` event following the OpenTelemetry exception conventions. By default it carries only `exception.type`; `exception.message` and `exception.stacktrace` depend on the tracker's `errorMessages` mode (`off` | `sanitized` | `raw`; `sanitized` keeps the first line with URLs cut to their origin, cut to 256 characters and `...`), and the span status description is the recorded `exception.message`, if any. See [ADR 0006](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0006-error-privacy.md). **JSON-RPC spans.** With the viem adapter's `traceTransport()`, each provider request is a `CLIENT` span named after its method, or `jsonrpc`, its `rpc.system.name`, for a name that is not a method (`rpc.method` is then `_OTHER`), following the OpenTelemetry [JSON-RPC conventions](https://github.com/open-telemetry/semantic-conventions/blob/v1.44.0/docs/rpc/json-rpc.md): `rpc.system.name = "jsonrpc"`, `rpc.method`, `jsonrpc.protocol.version`, `server.address` and `server.port`, plus `blockchain.chain.id`; on failure, `error.type` and, for a JSON-RPC error, `rpc.response.status_code`. Parameters, results, the URL path and error messages are not recorded; the host is recorded as it is, and these spans do not pass through the redaction hook. Its parent is the active span, such as a `send` span ([ADR 0019](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0019-json-rpc-spans.md)). With `propagateTraceContext()` on viem's `http()` transport, each HTTP request of one JSON-RPC request also carries that span's W3C `traceparent` and `tracestate` headers, never `baggage` ([ADR 0030](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0030-trace-context-on-json-rpc-requests.md)). A request is one span however often viem's transport retries it (its `retryCount`): the attempts run inside that span, and HTTP spans of each attempt, if traced, nest under it. viem sends `eth_sendRawTransaction` without retries, so a broadcast that gets an HTTP 429 or 503 fails after one attempt, with or without hashspan. The JSON-RPC conventions (1.44) differ from these spans in two points, on purpose: - They make `rpc.method` opt-in, since a general-purpose JSON-RPC instrumentation can see any method name; hashspan records it by default. The span is named after the method anyway, so leaving the attribute out would not lower cardinality; a name is recorded only within its [bound](#bounds), and `methods` is the way to choose the methods that get a span, which the conventions ask for. - They say that an instrumentation that records `_OTHER` must also record the value it replaced as `rpc.method_original`; hashspan does not. `_OTHER` stands only for a value that is not a method name: one that is not a string, or a string outside the [bound](#bounds) of `rpc.method`. An unknown method that is a method name keeps its name. Such a value has no bound, and may not be a string at all, so it is not recorded ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0025-untrusted-input.md)). ## Attributes | Attribute | Type | Spans | Default | Description | |---|---|---|---|---| | `blockchain.system.name` | string | send, confirm, payment | on | `evm` | | `blockchain.chain.id` | int | all | on | EIP-155 chain id, e.g. `8453` | | `blockchain.operation.name` | string | send, confirm, payment | on | `send` \| `confirm` \| `payment` | | `blockchain.tx.hash` | string | send, confirm, payment | on | `0x`-prefixed tx hash; on a payment span, the settling transaction's, when reported, or from the span's start when the payer knows it (MPP), and never for x402 `batch-settlement`, whose reported transaction is the channel's deposit ([ADR 0013](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0013-x402-payments.md)); on a user operation's confirm span, the bundle transaction's; absent on a user operation's send span | | `blockchain.tx.from` | string | send | raw | sender address, subject to address mode | | `blockchain.tx.to` | string | send | raw | recipient / contract address, subject to address mode | | `blockchain.tx.value` | string | send | on | value in wei, decimal string | | `blockchain.tx.nonce` | int | send | on | sender nonce, when the sending call passes one (a nonce the wallet or viem picks is not known to the adapter) | | `blockchain.tx.authorization.count` | int | send | on | number of EIP-7702 authorizations a type 4 transaction carries | | `blockchain.tx.authorization.addresses` | string[] | send | raw | delegated contract address of each well-formed authorization, subject to address mode, at most 64; `0x000...0` clears a delegation | | `blockchain.tx.authorization.chain_ids` | int[] | send | on | chain id of each well-formed authorization, in the order of the addresses, at most 64; `0` means valid on every chain | | `blockchain.contract.function.name` | string | send | on | decoded function name when an ABI is known | | `blockchain.contract.function.selector` | string | send | on | 4-byte selector, e.g. `0xa9059cbb` | | `blockchain.contract.function.arguments` | string | send | off (opt-in) | decoded call arguments as a JSON array, e.g. `["0x2222...2222","1000000"]`: bigints as decimal strings, addresses per address mode, truncated after 4096 characters. Only own enumerable data properties are serialized; `toJSON()` and getters are never called; binary data (typed arrays, `ArrayBuffer`) as `0x` hex | | `blockchain.tx.status` | string | confirm | on | from chain data: `success` \| `reverted` \| `replaced` | | `blockchain.block.number` | int | confirm | on | inclusion block; for a user operation, the bundle transaction's; for a call batch, the highest among its receipts | | `blockchain.tx.gas.used` | int | confirm | on | gas used | | `blockchain.tx.effective_gas_price` | string | confirm | on | price per gas in the chain's fee unit (see the fee table below), decimal string; see the `fee` row for when it is omitted | | `blockchain.tx.l1_fee` | string | confirm | on | L1 data fee on OP-stack chains, in the chain's fee unit; see the `fee` row for when it is omitted | | `blockchain.tx.fee` | string | confirm | on | `gas.used × effective_gas_price + l1_fee`, in the chain's fee unit: wei of the native currency, or the token `fee_asset` names (see the fee table below); omitted if the gas price is unknown. The OP Stack operator fee (Isthmus and later) is not included: it is recorded apart as `operator_fee`. Fee attributes come from the sealed receipt, never a flashblocks preconfirmation, and are omitted if only a preconfirmation was seen (see [ADR 0024](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0024-sealed-receipt-fees.md)). On the confirm span of a payment's settlement, the fee is the one the transaction's sender paid (for x402, the facilitator; for MPP, the payer), unless the receipt names a fee payer other than the sender (a Tempo `feePayer`, which every Tempo receipt names and which is often the sender itself) | | `blockchain.tx.operator_fee` | string | confirm | on | OP Stack operator fee (Isthmus and later), wei, decimal string; not included in `blockchain.tx.fee`. Recorded only when the receipt carries `operatorFeeScalar` or `operatorFeeConstant`, which a node adds when the chain charges the fee; the viem adapter then reads it from the GasPriceOracle's `getOperatorFee(gasUsed)` at the receipt's block, with one `eth_call`. Like the other fee attributes, it comes from the sealed receipt only, and is omitted for a deposit transaction, which pays none | | `blockchain.tx.fee_asset` | string | confirm | raw | contract address of the token the fee was paid in, subject to address mode, as the chain reports it (a Celo fee currency, which for a 6-decimal token is its 18-decimal adapter contract; a Tempo fee token); absent when the fee is in the native currency. Recorded with the fee attributes, when `effective_gas_price` is. Taken from the receipt when it names one, else from the send: the viem adapter reads Celo's `feeCurrency` from the sending call, never from `sendRawTransaction`, and Tempo's `feeToken` from a receipt of type `0x76`, or of another type that also names a `feePayer` ([ADR 0028 amendment](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md#amendment-2026-10-18-proposed-tempo-receipts-of-any-type)); a replacing transaction records its own, never the replaced one's ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md)) | | `blockchain.tx.revert.reason` | string | confirm | on | decoded revert reason when available, also of a reverted user operation: the `Error(string)` message, `Panic(0x..)`, `ErrorName(arg, ...)` for custom errors with a known ABI, else the 4-byte error selector. See [ADR 0005](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0005-revert-reason-replay.md) | | `blockchain.tx.replacement.hash` | string | confirm | on | on a `replaced` confirm span: hash of the mined transaction that replaced it | | `blockchain.tx.replacement.reason` | string | confirm | on | on a `replaced` confirm span: `repriced` \| `cancelled` \| `replaced`, as reported by the instrumented library; omitted when it reported none | | `blockchain.tx.wait.confirmations` | int | confirm | on | on a transaction's confirm span: the confirmations (blocks, counting the receipt's own) that the wait which ended the span asked for, as the library applies them: a viem wait's or a CDP network-scoped wait's `confirmations` when it is a positive safe integer; `1` when it is omitted, `0`, negative or `NaN`, as viem then waits for one, and for background confirmation and `watch()`; not recorded for any other value, nor when the wait took no count (sync actions, user operations, call batches). A count of blocks is not finality: on a rollup it counts the sequencer's blocks, and block times differ by chain, so read it with `blockchain.chain.id`. Span only, never a metric attribute (see one confirm span per transaction for which wait ends the span) | | `blockchain.user_operation.hash` | string | send, confirm | on | `0x`-prefixed user operation hash (`userOpHash`), which identifies the operation with the chain id | | `blockchain.user_operation.sender` | string | send, confirm | raw | address of the smart account, subject to address mode | | `blockchain.user_operation.entry_point` | string | send, confirm | raw | address of the EntryPoint contract, subject to address mode | | `blockchain.user_operation.call_count` | int | send | on | number of calls the operation makes, when the adapter knows them | | `blockchain.user_operation.nonce` | string | confirm | on | the operation's nonce, decimal string (a 192-bit key and a 64-bit sequence number) | | `blockchain.user_operation.success` | boolean | confirm | on | whether the operation's calls succeeded; the bundle transaction can succeed while they revert | | `blockchain.user_operation.gas.used` | int | confirm | on | gas the operation used (`actualGasUsed`) | | `blockchain.user_operation.gas.cost` | string | confirm | on | what the operation paid (`actualGasCost`), wei, decimal string; its share of the bundle, not the bundle transaction's fee; paid by the paymaster when `blockchain.user_operation.paymaster` is recorded | | `blockchain.user_operation.paymaster` | string | confirm | raw | address of the paymaster that paid for the operation, subject to address mode; absent when none paid | | `blockchain.call_batch.id` | string | send, confirm | on | the batch id the wallet returned for EIP-5792 `wallet_sendCalls` (`0x`-prefixed hex, at most 8194 characters), truncated after 256 characters; with the chain id, it identifies the batch; hashed like an address in `hashed` address mode and not recorded in `off` mode, see [values a remote party chooses](#values-a-remote-party-chooses) | | `blockchain.call_batch.sender` | string | send | raw | address of the account the calls are sent from, subject to address mode | | `blockchain.call_batch.call_count` | int | send | on | number of calls in the batch | | `blockchain.call_batch.status` | string | confirm | on | the batch's outcome from chain data: `success` \| `reverted` \| `partially_reverted`; absent for other outcomes, which `error.type` describes | | `blockchain.call_batch.status_code` | int | confirm | on | the EIP-5792 status code the wallet reported, e.g. `200` confirmed, `500` reverted; spans only, never a metric attribute | | `blockchain.call_batch.atomic` | boolean | confirm | on | whether the wallet ran the calls atomically | | `blockchain.call_batch.transaction_hashes` | string[] | confirm | on | hashes of the transactions whose receipts the wallet reported for the batch, de-duplicated, at most 64 | | `blockchain.operation.subject` | string | none (metrics only) | on | on [metrics](#metrics) of user operations: `user_operation`; of call batches: `call_batch`; absent on those of transactions | | `blockchain.fee.payer` | string | none (metrics only) | on | on `blockchain.client.fee` samples whose fee the sender of the traced transaction or operation did not pay: `facilitator` for a payment's settlement transaction that another party sent (x402), `paymaster` for a user operation a paymaster paid for, `sponsor` for a transaction whose receipt names another account as its fee payer (a Tempo `feePayer`); a payment's settlement sent by another party keeps `facilitator`, also when its fee was sponsored, while one the payer sent (MPP) is `sponsor` when the server paid its fee; absent when the sender paid ([ADR 0020](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0020-metrics.md)) | | `blockchain.fee.denomination` | string | none (metrics only) | on | on `blockchain.client.fee` samples whose fee was paid in a token rather than the chain's native currency: `token`, a constant, never an address or a value read from the chain; set whenever `blockchain.tx.fee_asset` is known, in every address mode; absent for the native currency, Arc's USDC included ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md)) | | `blockchain.payment.protocol` | string | payment | on | `x402` \| `mpp`; see [values a remote party chooses](#values-a-remote-party-chooses) | | `blockchain.payment.payer` | string | payment | raw | address that pays, subject to address mode; the settlement's payer only when the payer knew none | | `blockchain.payment.recipient` | string | payment | raw | address that is paid, subject to address mode | | `blockchain.payment.asset` | string | payment | raw | contract address of the token paid with, subject to address mode | | `blockchain.payment.amount` | string | payment | on | amount in the asset's smallest unit, decimal string; the settlement's amount only when the payer knew none | | `blockchain.payment.settled_amount` | string | payment | on | amount the settling party reports it settled, decimal string, e.g. less than the authorized maximum with x402 `upto`; for x402 `batch-settlement`, the request's charge against the channel (`extra.chargedAmount`), never the deposit; as reported, not checked | | `blockchain.payment.verified` | boolean | payment | on | whether the settlement transaction carries this payment (its receipt and, for Permit2, its input; for an MPP sponsored pull, also its sender and the nonce the payer signed), checked by the adapter from the payer's own data; absent when no check was possible ([ADR 0017](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0017-x402-payment-verification.md)) | | `blockchain.payment.status` | string | payment | on | `settled` \| `pending` \| `failed` | | `x402.scheme` | string | payment | on | x402 payment scheme, e.g. `exact`; see [values a remote party chooses](#values-a-remote-party-chooses) | | `x402.resource` | string | payment | origin | the resource paid for, per the tracker's `paymentResource` mode: `origin` (default) records scheme, host and port only, `path` the URL without query string, fragment or user info, `off` nothing; at most 512 characters, and nothing for text whose user info contains `?` or `#`; see [values a remote party chooses](#values-a-remote-party-chooses) | | `mpp.method` | string | payment | on | MPP payment method, e.g. `tempo`; see [values a remote party chooses](#values-a-remote-party-chooses) | | `mpp.intent` | string | payment | on | MPP intent, e.g. `charge`; see [values a remote party chooses](#values-a-remote-party-chooses) | | `mpp.mode` | string | payment | on | how an MPP charge is settled: `push` (the payer broadcasts) \| `pull` (the server broadcasts what the payer signed) | | `mpp.challenge.id` | string | payment | on | id of the MPP challenge paid, chosen by the server: 1 to 128 characters of `A-Z a-z 0-9 - . _ ~`, else not recorded (see also [values a remote party chooses](#values-a-remote-party-chooses)); spans only, never a metric attribute ([ADR 0029](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0029-mpp-payments.md)) | | `mpp.resource` | string | payment | origin | the resource paid for, as `x402.resource` records it | | `error.type` | string | all | on | see *Span status* and [values a remote party chooses](#values-a-remote-party-chooses); reused from OpenTelemetry general conventions | Of this table, JSON-RPC spans carry only `blockchain.chain.id` and `error.type` (see [JSON-RPC spans](#spans)). In `blockchain.tx.*`, `tx` abbreviates transaction, the usual term in this domain; names outside that namespace spell the word out, as `blockchain.call_batch.transaction_hashes` does. The parties of each subject are named after the source the values come from: | Subject | Sender | Recipient | Why | |---|---|---|---| | Transaction | `blockchain.tx.from` | `blockchain.tx.to` | the `from` and `to` fields of a JSON-RPC transaction | | User operation | `blockchain.user_operation.sender` | none | the ERC-4337 `sender`, the smart account; the operation's calls have their own targets | | Call batch | `blockchain.call_batch.sender` | none | the account the calls are sent from, which can differ from the sender of the transaction that carries them, such as a bundler or relayer | | Payment | `blockchain.payment.payer` | `blockchain.payment.recipient` | the parties of the payment; for x402 the settling party, not the payer, sends the transaction; for MPP the payer is the transaction's `from`, and pays its fee unless the server sponsors it | The fee fields follow the receipt of each chain family, as the viem adapter reads it (tested in `packages/viem/test/fee-models.test.ts`): | Family | `blockchain.tx.l1_fee` | `blockchain.tx.fee` | |---|---|---| | Ethereum and other L1s | not recorded | `gas.used × effective_gas_price` | | OP Stack (Base, OP Mainnet, Celo) and Scroll | the receipt's `l1Fee` | plus `l1_fee`; the OP Stack operator fee is recorded apart, as `blockchain.tx.operator_fee` | | Arbitrum | not recorded: `gasUsed` already includes the L1 component (`gasUsedForL1`) | `gas.used × effective_gas_price` | | ZKsync | not recorded | `gas.used × effective_gas_price` | | Celo, a transaction with a `feeCurrency` (type `0x7b`) | the receipt's `l1Fee` | plus `l1_fee`, in the fee currency, which only the transaction names: `blockchain.tx.fee_asset` is the sending call's `feeCurrency` | | Tempo (type `0x76`, and every other type: a receipt that names a `feePayer`) | not recorded | `gas.used × effective_gas_price`, in attodollars (10^-18 USD), charged in the receipt's `feeToken`, recorded as `blockchain.tx.fee_asset`; viem's Tempo chain declares 6 decimals for its currency, so formatting the fee with it is off by 10^12 | | Arc | not recorded | `gas.used × effective_gas_price`, in its native currency, USDC with 18 decimals; no `fee_asset`, since the chain id already says what the unit is | Values are recorded as the receipt gives them, in the chain's fee unit: the unit the chain charges gas in, which is not always wei of its native currency. Nothing is converted, rounded or formatted with token decimals. When the fee was paid in a token, the confirm span names it in `blockchain.tx.fee_asset` and the fee sample carries `blockchain.fee.denomination` `token`, so a dashboard can tell these samples from fees in the native currency ([ADR 0028](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0028-fee-asset.md)). A Celo transaction whose fee currency the adapter does not see, one sent with `sendRawTransaction` or sent elsewhere and confirmed with `watch()`, records no asset and no marker: its sample counts as native. Agent identity is recorded with the GenAI conventions `gen_ai.agent.id` and `gen_ai.agent.name`. A field set in the tracker's static `agent` option always wins; fields it leaves unset are taken from OpenTelemetry Baggage entries with the same keys, unless `agentFromBaggage` is false ([ADR 0011](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0011-agent-identity-precedence.md)). A Baggage value is recorded only if it has at most 128 letters, digits, spaces and `_ . : @ / -` (and, in `hashed` and `off` address mode, see [values a remote party chooses](#values-a-remote-party-chooses)). This lets backends search transactions by agent without joining spans. Baggage is propagated to downstream services; identifiers that must stay internal belong in the static `agent` option, which is never propagated. ## Metrics The tracker records these histograms through the meter provider (the global one unless `meterProvider` is given), so every adapter gets them ([ADR 0020](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0020-metrics.md)). Their attributes are low-cardinality only: `blockchain.system.name`, `blockchain.chain.id`, what the sample is about and its outcome, as the table lists them; never an address, a hash or the agent identity. Samples of user operations ([ADR 0021](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0021-user-operations.md)) and call batches ([ADR 0022](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0022-call-batches.md)) carry `blockchain.operation.subject`; a call batch's raw status code is never recorded on metrics, and batches record no fee. | Metric | Instrument | Unit | Attributes | Recorded when | |---|---|---|---|---| | `blockchain.client.send.duration` | histogram | `s` | chain; `blockchain.operation.subject` for a user operation or call batch; `error.type` if the send failed | a send span ends with a hash or id, or fails: from the start of the sending call until then | | `blockchain.client.confirmation.duration` | histogram | `s` | chain; `blockchain.operation.subject` for a user operation or call batch; the outcome from chain data, `blockchain.tx.status` (for a user operation, `blockchain.user_operation.success`; for a call batch, `blockchain.call_batch.status`), else `error.type` (`timeout`, `not_on_chain`, an adapter's error type, an error class name, or `_OTHER`) | a confirm span ends: from the start of the wait until the receipt or batch status, a replacement, a timeout or a failure; not for a call batch that ended while still pending, nor for a transaction whose receipt was pending | | `blockchain.client.fee` | histogram | `{wei}` | chain; `blockchain.operation.subject` for a user operation; `blockchain.tx.status` (for a user operation, `blockchain.user_operation.success`); `blockchain.fee.payer` when someone other than the sender paid; `blockchain.fee.denomination` when the fee was paid in a token | a receipt with an effective gas price is recorded: `blockchain.tx.fee` as a number, without the OP Stack operator fee, which no metric records; for a user operation, a receipt with its cost: `blockchain.user_operation.gas.cost` | A confirmation sample is a success when its outcome from chain data says so: `blockchain.tx.status` `success`, `blockchain.user_operation.success` `true` or `blockchain.call_batch.status` `success`. Count successes across subjects by matching these values, not by the absence of `error.type`: a sample with an outcome from chain data, such as `reverted` or `replaced`, carries no `error.type`. Bucket boundaries are given as advice: 0.05 s to 300 s for durations, and one bucket per power of ten from 10^8 to 10^18 wei for fees. Fees above 2^53 wei lose precision as numbers; the span attribute keeps the exact value. On metrics, `error.type` is kept only when it is an error class name of letters ending in `Error` (such as `NonceTooLowError`) or a lower-case code of letters and underscores (such as `timeout` or `insufficient_balance`); any other value, which could carry an identifier, an address or a number, is recorded as `_OTHER`. The span keeps its own `error.type`. ## Privacy `blockchain.tx.from`, `blockchain.tx.to`, the `blockchain.payment.*` addresses, the user operation's sender, EntryPoint and paymaster, `blockchain.call_batch.sender` and `blockchain.tx.authorization.addresses` follow the address mode: `raw` (default, the address in lower case), `hashed` (`sha256:` + first 32 hex characters of SHA-256 of the lower-cased address, or a custom function) or `off`. Neither depends on how the source wrote the address, so one address has one value on every span, whether it came checksummed from the call's arguments or lower-cased from a receipt. A redaction hook runs last on every attribute set of the tracker's spans, not on metrics or JSON-RPC spans; if it throws or returns something other than an attributes object, only `blockchain.system.name`, `blockchain.chain.id`, `blockchain.operation.name`, `blockchain.tx.hash`, `blockchain.tx.status`, `blockchain.tx.replacement.hash`, `blockchain.tx.replacement.reason`, `blockchain.payment.protocol`, `blockchain.payment.status`, `blockchain.payment.verified`, `blockchain.user_operation.hash`, `blockchain.user_operation.success`, `blockchain.call_batch.id`, `blockchain.call_batch.status` and `error.type` are recorded. Hashing is pseudonymisation, not anonymisation. See [ADR 0004](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0004-privacy-defaults.md). Neither `hashed` nor `off` hides the parties of a transaction: `blockchain.tx.hash` is always recorded and resolves to them on chain, as `blockchain.user_operation.hash` does for a user operation. The address mode also applies to addresses inside `blockchain.tx.revert.reason`, `blockchain.contract.function.arguments`, `x402.resource`, `mpp.resource`, `error.type` and sanitized error messages (`
` in `off` mode). In `hashed` and `off` mode, hex values longer than an address are recorded as `` in those attributes, because a padded `bytes32` or ABI-encoded `bytes` value can embed an address. What a value longer than its bound becomes is listed under [Bounds](#bounds). The redaction hook also runs on `error.type` and on `exception` event attributes; if it throws, only `exception.type` is kept on the event. Payment values usually come from a remote party (the paid server or the settling party): addresses that are not `0x`-prefixed 20-byte hex, amounts that are not non-negative integers, hashes that are not 32-byte hex and identifiers (protocol, scheme, MPP method and intent, failure reason) that are not short identifiers are not recorded (in `hashed` and `off` address mode, see [values a remote party chooses](#values-a-remote-party-chooses)). Paths of paid APIs often carry user or account identifiers, so `x402.resource` and `mpp.resource` record only the origin by default; with `paymentResource: 'path'` it records the path too, never the query string, fragment or user info, which can carry credentials ([ADR 0004](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0004-privacy-defaults.md)). A user operation's hash and receipt come from the bundler and are checked the same way; its nonce, gas and cost may also be `0x` hex quantities. What a caller or an adapter passes to the tracker is checked as well ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0025-untrusted-input.md)): a call without a positive safe integer chain id, or a confirmation without a 32-byte hex hash, records no span; a send records only well-formed addresses, a value that is a non-negative integer of at most 256 bits, a nonce that is a non-negative safe integer, a Solidity function name and a 4-byte selector. Nothing is recorded from a transaction receipt whose block number or gas used is not a non-negative safe integer; its gas price and L1 fee are recorded only as non-negative integers, and `blockchain.tx.fee` only when every part of it is known. The EIP-7702 attributes describe what the sent transaction asks for, not what took effect: the protocol skips an authorization whose signature, nonce or chain id does not hold, without failing the transaction. The account that signed each authorization (its authority) is not recorded; it is often the sender, but not in a sponsored transaction. The signatures and the authorities' nonces are never recorded. ### Values a remote party chooses In `hashed` and `off` address mode, a value that a remote party chooses and that is not itself an address is not recorded when it holds 40 hex characters in a row, with or without `0x`: a server, a facilitator, a wallet or an upstream caller could put an address there, such as the payer's. This applies to `blockchain.payment.protocol`, `x402.scheme`, `mpp.method`, `mpp.intent`, `mpp.challenge.id`, and `gen_ai.agent.id` and `gen_ai.agent.name` taken from Baggage. In `x402.resource`, `mpp.resource`, `error.type` and `exception.type`, an address with `0x` follows the address mode as above; a resource that still holds 40 hex characters in a row after that, such as a host or path segment without `0x`, is not recorded, and such an error type is recorded as `_OTHER`. `blockchain.call_batch.id`, which the wallet chooses, is treated like an address: in `hashed` mode it records the whole id through the address mode's hash (the same function, so one id has one value on its send and confirm spans), and in `off` mode nothing; the batch is still tracked by its id in every mode. `raw` mode records these values as before, and the static `agent` option, which is the user's own text, is always recorded as given ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0025-untrusted-input.md#amendment-2026-10-18-values-a-remote-party-chooses)). ## Bounds Values come from parties hashspan does not control, and the OpenTelemetry SDK does not bound attribute values by default, so each value read from them has a bound ([ADR 0025](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0025-untrusted-input.md)); the hostile-input tests of each package check the bounds against the values that split at them. | Value | Bound | A longer value becomes | |---|---|---| | `blockchain.tx.revert.reason` | 1024 characters | its first 1024 characters, followed by `...`; a hex value the cut would split is dropped whole. A reason an adapter already cut this way is kept as it is | | `error.type` and `exception.type` from an error's name | 64 characters of `[A-Za-z0-9_.-]` | `_OTHER` | | `rpc.method` and the JSON-RPC span name | a letter, then up to 63 characters of `[A-Za-z0-9_]` | `_OTHER`, and the span is named `jsonrpc`; the value is not recorded, not as `rpc.method_original` either | | `exception.message` in `sanitized` mode | 256 characters of the first line, sanitized from its first 4096 characters | its first 256 characters, followed by `...`; a hex value the cut would split is dropped whole. Text past the 4096th character of the first line is not recorded (a hex value cut there is recorded as if whole, a URL cut there before its path as ``) | | `blockchain.contract.function.arguments` | 4096 characters, nesting depth 32 | the JSON up to the value that crosses 4096 characters, cut there and followed by `...`; a hex value the cut would split is dropped whole; arguments nested deeper are not recorded | | `blockchain.contract.function.selector` of `writeContract` | an ABI of 10 000 items and 100 000 copied values; for an overloaded function, 100 000 copied argument values | no selector; past the ABI bound, no ABI is used for telemetry either, so custom errors in the revert reason show as their selector | | `x402.resource` and `mpp.resource` | 512 characters | its first 512 characters, followed by `...`; a hex value the cut would split is dropped whole | | `mpp.challenge.id` | 128 characters | not recorded | | `blockchain.call_batch.id` | 256 characters | its first 256 characters (in `raw` address mode); an id that is not hex or longer than 8194 characters is not recorded | | `blockchain.tx.authorization.addresses` and `.chain_ids` | 64 entries | the first 64 well-formed entries; `blockchain.tx.authorization.count` keeps the full length, and no further entry is read | | `blockchain.call_batch.transaction_hashes` | 64 hashes | the first 64 distinct well-formed hashes among the first 64 receipts of a status, or of the hashes a send returned; no further entry is read, and `blockchain.block.number` is the highest among those receipts | | Call batch id the viem adapter parses for viem's fallback transactions | 8194 characters | not parsed: no fallback transaction is recorded or confirmed for the batch | | Fallback transactions the viem adapter records and confirms per call batch | 64 | the first 64 non-zero hashes of the id; later calls of the batch get no confirm span | | Receipts the viem adapter reads from a call batch status | 64 | the first 64; no further receipt is read | | x402 payments waiting for their response | 1000 per `withHashspan()` | the oldest payment span ends as `timeout` | | MPP payments waiting for an outcome, counted from their challenge | 1000 per `withHashspan()` | the oldest payment ends as `timeout`; one not started yet records nothing | | MPP challenge ids the adapter correlates events by | 1024 characters | the payment is not traced | | MPP credential and `Payment-Receipt` header the adapter decodes | 65 536 and 4096 characters | the payment is traced without what they carry: no mode or payer's hash, or no settlement (`_OTHER`) | | Background confirmations polling at once | 256 per `withHashspan()` (`maxBackgroundConfirmations`) | the transaction gets no background confirm span, and a `watchUserOperation()` records nothing; a `diag` warning is logged | | Logs of a bundle receipt that `watchUserOperation()` searches for the operation's `UserOperationEvent` | 10 000 | no log is read: nothing is recorded ([ADR 0031](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0031-watch-user-operation.md)) | | Sent transactions, user operations and call batches kept for links | 10 000 each, for 10 minutes (`maxTrackedTransactions`, `linkTtlMs`) | the oldest is forgotten: its confirm span has no link | ## Change policy These conventions stay `development` after 1.0; the public API is what 1.0 freezes ([ADR 0027](https://github.com/selimaytac/hashspan/blob/@hashspan/core@1.3.1/docs/adr/0027-what-1-0-freezes.md)). A change to them follows these rules: - **Additions are minor changes:** a new attribute, metric or enum value. Consumers handle values they do not know, as `_OTHER` already asks. - **Renames and removals** of an attribute, a metric or an enum value record the old and the new name side by side for at least one minor release, are announced in the CHANGELOG, and bump the schema version. A span name, which cannot be recorded twice, changes only in a minor release, announced the same way. - **Meaning is not changed in place:** a new meaning gets a new name, and the old one is deprecated as above. - **Value forms are part of each attribute:** the type and form in the tables above change only through the same steps. - An exported constant for a deprecated name stays exported, marked `@deprecated`, until the next major release. One attribute at a time The same attributes, one heading each: Transaction attributes User operation attributes Call batch attributes Payment attributes Common attributes Metrics Concepts Short explanations of the ideas on this page: Semantic conventions: shared names for spans and attributes Span attributes: putting the tx hash on the span Span status codes: Unset, Error and Ok Span links: connecting spans that aren't parent and child --- # Testnet runs Source: https://hashspan.dev/docs/runs A separate lab repository runs scenarios on Base Sepolia with the published `@hashspan/*` packages. Each run checks, in the same process, the spans hashspan recorded, and writes one line to a run ledger, also when it fails. This page shows that ledger: every run, its outcome and the transactions it sent. ## What each scenario checks - **treasury:** the example agent with a scripted model. It pays 0.00001 ETH to itself and tries a withdrawal that the vault rejects. It passes when each tool span has a `send 84532` and a `confirm 84532` span under it, in one trace, with the receipt data and the decoded revert reason. It checks that these attributes are there and well formed; it does not compare them with the receipts. - **sealed-fees:** sends nothing. It watches up to 50 transactions of other accounts with `watch()` and compares each confirm span with the sealed receipts of its block: status, `blockchain.tx.l1_fee` and `blockchain.tx.fee` must be the sealed receipt's, never a flashblocks preconfirmation's. Spans recorded without fees, the documented fallback, may be up to 20%; fewer than 10 compared transactions is reported as an error (an idle chain or a failing RPC), not as a finding. - **paths:** a send in background confirmation mode without a wait, a transaction confirmed with `watch()`, and three calls that revert with `Error(string)`, `Panic(0x11)` and a custom error. It passes when each gets a confirm span with the receipt data, the three revert reasons are decoded, the watched transaction has no send span, and `traceTransport()` records `eth_sendRawTransaction` under each send span. - **call-batches:** two EIP-5792 batches through viem's fallback to plain transactions. It passes when each batch has a send span with its id and two calls, the first ends `success` and the second `partially_reverted`, and each of the three plain transactions has a confirm span with fees. A run that fails its setup (wrong chain, too little balance) or its span check is listed as failed. The hashspan version on each run is the `@hashspan/viem` version its scenario pinned when it ran. ## Runs - 2026-10-03 15:16 UTC, call-batches, pass, hashspan 0.9.0: no transactions - 2026-10-03 15:15 UTC, paths, pass, hashspan 0.9.0: background success https://sepolia.basescan.org/tx/0x6546f6394fe06f3241936a0c9cdd51215527c67964cc64af8efaa9dfe91d8eb4 fee 131596910870 wei; watch success https://sepolia.basescan.org/tx/0x1e7a175df7441e99eb851fe7e42c5f1f45bd066435ebb92317e931a73c46d11c fee 131596910870 wei; revert-string reverted https://sepolia.basescan.org/tx/0x6c9ad44a1d2e42453b406950dbcbdee252f446ed7fb4dba2b831bc906713dcc9 fee 132556910870 wei; revert-custom reverted https://sepolia.basescan.org/tx/0x9ae6d11d3f03181ea234a0448509b7d4dda8e0759c560ad4f7ab51db7eb8a226 fee 132556910870 wei; revert-panic reverted https://sepolia.basescan.org/tx/0x8116af454ba22b92ce40eb1a362f5a3587e6f1c0097b7750329f831c951d91c6 fee 132556910870 wei - 2026-10-03 15:14 UTC, sealed-fees, pass, hashspan 0.9.0: 50 of 50 watched transactions compared with sealed receipts; 0 without fees - 2026-10-03 15:13 UTC, treasury, pass, hashspan 0.9.0: pay_vendor success https://sepolia.basescan.org/tx/0x5064e56ef6dd00ea2391d39d1f010d9b4fc011dda25ee5e6c1dff6e39963090b fee 132001498455 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0xc4b0b4078bd4130a72de3d87604d4d816348a8a059835505de7ca802f8fd1542 fee 135781498455 wei - 2026-10-03 13:24 UTC, call-batches, pass, hashspan 0.9.0: no transactions - 2026-10-03 13:24 UTC, paths, pass, hashspan 0.9.0: watch success https://sepolia.basescan.org/tx/0x4dbec1bfb935e17b39f0e2a967bfa322c1306fd4351fcb526b6b6fe7c000f7a8 fee 132065777870 wei; revert-string reverted https://sepolia.basescan.org/tx/0x60d4060460094c95349a3a8a81b0d1ee8d827768992b7a3546369cd25bddff40 fee 133025777870 wei; background success https://sepolia.basescan.org/tx/0x53e514ae3f5943c29ef97648475c8dec92d0753f31dfb3e834b5c5d669973d9b fee 132065777870 wei; revert-panic reverted https://sepolia.basescan.org/tx/0xc5a6d086d13ca7c8b9956dae6c51cd9472512687cd3e02c3fe1eab49918e6f99 fee 133121313904 wei; revert-custom reverted https://sepolia.basescan.org/tx/0x9f69393ff2f6807c0f05a7389fdbdb30140e9e7fb5de3cb3cca07b85b392f10a fee 133121313904 wei - 2026-10-03 13:23 UTC, sealed-fees, pass, hashspan 0.9.0: 50 of 50 watched transactions compared with sealed receipts; 0 without fees - 2026-10-03 13:22 UTC, treasury, pass, hashspan 0.9.0: pay_vendor success https://sepolia.basescan.org/tx/0xa123da4e9da4bb90cd9a7bd2f0678c2e1f296ed8d5577d500e57dff175eba42b fee 131912241564 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0xf0c8fe10d1a818073ac59d8988451f764453e9d1b94a8858ea9ff9e7b7ada38f fee 135692241564 wei - 2026-10-03 13:01 UTC, paths, pass, hashspan 0.8.2: background success https://sepolia.basescan.org/tx/0xa01339d3eeddd53cd5a324f17515f79056c0ce9d263516e78fbb3d12e4ff4b9d fee 131938018036 wei; watch success https://sepolia.basescan.org/tx/0xe01d9a863535bc96987cddedb665f50102e52f7eefb9005c5ebbcdcce229f432 fee 131938018036 wei; revert-string reverted https://sepolia.basescan.org/tx/0xb925b98a481eb6d639195d619ed9a736b3466be7ce0ce39845a08057ca4193dc fee 132898018036 wei; revert-custom reverted https://sepolia.basescan.org/tx/0x15a89ebc749ce7f41f69a229c7b6fcf64de003cae93e493bf248ac266a1dc93a fee 132898018036 wei; revert-panic reverted https://sepolia.basescan.org/tx/0xe63ce84a07262505fe7e94ec228919e24726b239cb1b97cd50d2a08fa6343b45 fee 132898018036 wei - 2026-10-03 13:00 UTC, sealed-fees, pass, hashspan 0.8.2: 50 of 50 watched transactions compared with sealed receipts; 0 without fees - 2026-10-03 12:59 UTC, treasury, pass, hashspan 0.8.2: pay_vendor success https://sepolia.basescan.org/tx/0xc8fbb3a6c1db30119f6db3ef1ed1b6cee7a9a7446d6d355dd998e607c531ad31 fee 131887759757 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0x4b50ae2e3f7cb079407900b3659a16608132d59833fcdf9bc3d38c096e2de83a fee 135667759757 wei - 2026-10-03 12:49 UTC, treasury, pass, hashspan 0.8.2: pay_vendor success https://sepolia.basescan.org/tx/0x648170560e9283cd9bbc706242ee60a4849e53550757b21fc64cd5e96f7deaa0 fee 132099490115 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0x9b6131dac0e2b57bcf74fe2384b51f8d2973f7a5688460c48084d5ecdd1cb433 fee 135879490115 wei - 2026-10-03 11:48 UTC, treasury, pass, hashspan 0.8.2: pay_vendor success https://sepolia.basescan.org/tx/0x6c214f7e6ff3aca5fd6c0340a98d0f04aa0e6d4de834af264bf121f3ff38b942 fee 131537549945 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0xfc38774104bd916b3ec7634a4c963563232cbc696802935b2bdc57f767ad46c0 fee 135317549945 wei - 2026-10-02 22:11 UTC, treasury, pass, hashspan 0.8.2: pay_vendor success https://sepolia.basescan.org/tx/0x137924ff1707c603ffb6d6585fc868247c023c16c3102b3d3acfbcb6da181972 fee 133148995271 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0x9af975a8f7454b5c8a50eafb812006d99737c514493111a48c7dad0cc9648892 fee 136928995271 wei - 2026-10-02 21:44 UTC, treasury, fail, hashspan 0.8.0: pay_vendor success https://sepolia.basescan.org/tx/0x5b016ec3f00f2eb32e0dd7bfe911a47d5fac430c5163e959bdb4f4891e0fd5a5 fee 132481290848 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0xf27dada0c2e1787325dbd80bf973e955d7d5ccd200d057b3fb7785bbb5f574e2 fee 136261290848 wei - 2026-10-02 13:05 UTC, treasury, pass, hashspan 0.7.0: pay_vendor success https://sepolia.basescan.org/tx/0x4e17cfffb725ff7f9232cf5ab906f314f9d55e77f2903be3e52d85c96d42b54e fee 131831174020 wei; withdraw_from_vault reverted https://sepolia.basescan.org/tx/0xd9c9f77f8a48c13b75014da0d56f8c47b29d46675de7ec82f0962f45d226898c fee 135611174020 wei More setups, run once and checked by hand, are on [Tested setups](https://hashspan.dev/docs/tested). --- # Security, privacy and stability Source: https://hashspan.dev/docs/security Evaluating hashspan for a team? This page collects the answers in one place, with a link to the source of each. ## Can it touch my keys or move funds? No. hashspan never signs or broadcasts a transaction, and it needs no private key. Its own RPC calls are read-only: it reads receipts, and replays a reverted transaction as a call to decode the reason ([requests it adds](https://hashspan.dev/docs/architecture)). It runs inside your app, and its instrumentation never throws into or changes the result of your call. ## What does it record by default? | Data | Default | How to change it | |---|---|---| | Addresses (sender, recipient, payer, paymaster…) | Recorded, in lower case | `address: 'hashed'` or `'off'`, or your own `hash` function | | Transaction hash | Recorded | Keep it: it is how spans are matched to the chain | | Amounts (value, payment amount, fees) | Recorded | Remove from spans with a `redact` hook; the fee histogram keeps fees, so drop it with a metrics View | | Agent identity (`gen_ai.agent.*`) | Recorded when set, also from incoming Baggage | `agentFromBaggage: false`; don't put personal names in `agent` | | Calldata arguments | Not recorded | Opt in with `recordFunctionArguments` | | Error messages | Not recorded, only the error type | `errorMessages: 'sanitized'` or `'raw'` to opt in | | Paid URLs (x402, MPP) | Only the origin | `paymentResource: 'path'` or `'off'` | | JSON-RPC spans (RPC host and method) | Not recorded | Opt in with `traceTransport()` (viem) | | Metrics | Durations and fees, by chain, operation and outcome; never an address, hash or agent identity | Drop a histogram with a metrics View | The full list is in [what is recorded](https://hashspan.dev/docs/adapters/core#what-is-recorded). The `redact` hook runs on every span attribute set, but not on metrics or JSON-RPC spans. ### For strict privacy reviews - **Hashed mode is not secret.** It is an unkeyed SHA-256, and addresses are public, so a hash can be matched by hashing candidates. Use `'off'`, or a keyed hash such as an HMAC with a secret you keep. - **Retention and location are your backend's.** hashspan keeps nothing; spans live where your OpenTelemetry backend stores them, for as long as it keeps them. A strict setup: ```ts import { createHmac } from 'node:crypto'; import { withHashspan } from '@hashspan/viem'; declare const key: string; // a secret from your secret store, not from code // Strict: no plain addresses, no error text, no identity from incoming Baggage, no paid URLs. export const strict = withHashspan({ address: { mode: 'hashed', hash: (address) => `hmac:${createHmac('sha256', key).update(address).digest('hex').slice(0, 32)}` }, errorMessages: 'off', agentFromBaggage: false, paymentResource: 'off', }); ``` The library sends no telemetry to the project. Spans go only to the OpenTelemetry backend you set up. Hashing or dropping addresses limits what your backend stores, but it is not anonymity: a transaction hash leads to its parties on chain. The [privacy section](https://hashspan.dev/docs/reference/semantic-conventions#privacy) lists every attribute each mode covers. To drop them in a Collector instead, see the [Collector recipe](https://hashspan.dev/docs/recipes/collector). ## How do I report a vulnerability? Privately, through GitHub's [private vulnerability reporting](https://github.com/selimaytac/hashspan/security/advisories/new), not in a public issue. The [security policy](https://github.com/selimaytac/hashspan/blob/main/SECURITY.md) says to expect an acknowledgement within 5 business days. A confirmed report then gets a fix, a disclosure date agreed with you, and a published advisory. Security fixes go to the latest minor of each package: 1.x for `core`, `viem`, `cdp` and `x402`, and the latest 0.x for `mpp`. The policy also lists what is in and out of scope, such as an address recorded in a mode that should keep it out. ## What does 1.0 keep stable? - **The public API.** Since 1.0, a breaking change to the TypeScript API needs a major release. Each package's API report lists what is public, and CI fails when it is stale ([ADR 0027](https://github.com/selimaytac/hashspan/blob/main/docs/adr/0027-what-1-0-freezes.md)). - **Not yet the span and attribute names.** They stay development. A renamed attribute or metric is recorded under both names for at least one minor release, announced in the changelog, with a schema version bump. A span name changes only in a minor release, announced the same way ([change policy](https://hashspan.dev/docs/reference/semantic-conventions)). So your code keeps compiling across 1.x. Check the [changelog](https://hashspan.dev/changelog) before a minor upgrade if dashboards depend on attribute names. ## How is it released? From GitHub Actions, with npm provenance: npm shows a provenance badge for each version, linking it to its build. Each GitHub release carries a CycloneDX SBOM of the lockfile. ## Who maintains it? hashspan is open source under the Apache-2.0 license, maintained in the open on [GitHub](https://github.com/selimaytac/hashspan). Significant design decisions are recorded as [ADRs](https://github.com/selimaytac/hashspan/tree/main/docs/adr). The license lets anyone use, change and fork it. --- # Tested setups Source: https://hashspan.dev/docs/tested hashspan emits standard OpenTelemetry spans, so it works with anything that speaks OTLP. This page shows what it is checked against: first what CI runs again on every change or every week, then the networks, backends and setups it was run with, each with its evidence. Integration setups are on [Integrations](https://hashspan.dev/docs/integrations), the scenarios on Base Sepolia on [Testnet runs](https://hashspan.dev/docs/runs). ## Checked in CI Run again on every change, or every week against the newest releases, so each release keeps them. | Setup | Run | What was checked | Evidence | |---|---|---|---| | Anvil, local chain (31337) (Anvil 1.8.3) | every CI run | Send and confirm spans, decoded revert reasons and replaced transactions, in the integration tests. | [CI workflow](https://github.com/selimaytac/hashspan/actions/workflows/ci.yml) | | ERC-4337 bundler on Anvil (Alto 0.0.21, EntryPoint 0.7) | every CI run | User operations sent through a real bundler: send and confirm spans with the operation hash, its success and the bundle transaction. | [test](https://github.com/selimaytac/hashspan/blob/main/packages/viem/test/real-bundler.int.test.ts) | | Node.js (22.3, the latest 22.x and 24.x) | every CI run | The packed packages are installed into an empty project and loaded with require() and import on each version. | [CI workflow](https://github.com/selimaytac/hashspan/actions/workflows/ci.yml), [smoke test](https://github.com/selimaytac/hashspan/blob/main/scripts/smoke-packages.mjs) | | viem (2.21.0 and the newest 2.x) | weekly in CI | Both ends of the supported range: the newest release with typecheck and every viem test, the oldest with the unit and integration tests. | [workflow](https://github.com/selimaytac/hashspan/actions/workflows/viem-range.yml) | | x402 SDK (@x402/core 2.13.0 and the newest 2.x) | weekly in CI | Both ends of the supported range: the newest release with every x402 test, the oldest with the adapter’s own tests. | [workflow](https://github.com/selimaytac/hashspan/actions/workflows/x402-sdk.yml) | | mppx (MPP SDK) (mppx 0.13.5 and the newest 0.13.x) | weekly in CI | Both ends of the supported range, on Anvil in Tempo mode: the newest release with typecheck and every MPP test, the oldest with the adapter’s own tests. | [workflow](https://github.com/selimaytac/hashspan/actions/workflows/mppx-sdk.yml) | | Coinbase CDP SDK (1.57 and the newest 1.x) | every CI run, weekly | Both ends of the supported range every week, each with typecheck and the CDP unit and integration tests; the locked release on every change. | [workflow](https://github.com/selimaytac/hashspan/actions/workflows/cdp-sdk-range.yml) | | Vercel AI SDK (ai 7, @ai-sdk/otel 1) | every CI run | The example agent: hashspan spans nest under the SDK’s execute_tool spans. | [recipe](https://hashspan.dev/docs/recipes/ai-sdk), [example agent](https://github.com/selimaytac/hashspan/tree/main/examples/ai-sdk-agent) | | Coinbase AgentKit (0.10.4) | weekly in CI | ViemWalletProvider, CdpEvmWalletProvider and CdpSmartWalletProvider: send and confirm spans for sendTransaction, nativeTransfer and the ERC-20 approve action. | [setup](https://hashspan.dev/docs/integrations#coinbase-agentkit), [viem test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/agentkit-viem.int.test.ts), [CDP test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/agentkit-cdp.int.test.ts) | | GOAT SDK (@goat-sdk/wallet-viem 0.3.0) | weekly in CI | The send_token and approve_token_evm tools record send and confirm spans. | [setup](https://hashspan.dev/docs/integrations#goat-sdk), [test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/goat-viem.int.test.ts) | | Mastra (@mastra/core 1.74, @mastra/otel-bridge 1.5) | weekly in CI | With the OpenTelemetry bridge, send and confirm spans nest under Mastra’s execute_tool span. | [setup](https://hashspan.dev/docs/integrations#agent-frameworks), [test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/mastra.int.test.ts) | | LangChain and OpenAI Agents (OpenInference) (LangChain 1.5, OpenAI Agents SDK 0.18) | weekly in CI | Both instrumentations and hashspan in one trace, with the agent run inside an active span. | [setup](https://hashspan.dev/docs/integrations#agent-frameworks), [LangChain test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/openinference-langchain.int.test.ts), [OpenAI Agents test](https://github.com/selimaytac/hashspan/blob/main/integrations/test/openinference-openai-agents.int.test.ts) | ## Run on a testnet Real transactions on Base Sepolia, with the spans checked against the receipts. The lab’s scenario runs are on Testnet runs. | Setup | Run | What was checked | Evidence | |---|---|---|---| | Base Sepolia (84532) (public RPC) | 2026-10-19, hashspan 1.3.0 | A payment and a reverted withdrawal. Status, block, gas, the L1 data fee (92 wei on each) and the total fee on the confirm spans equal the sealed receipts; the revert reason was decoded. @hashspan/viem 1.3.0 with @hashspan/core 1.2.1, from main (1a0d07a). This run is the trace on the home page. First run on 2026-10-02 with 0.6.0 and 0.8.0 (other transactions, the same checks); the 0.8.0 run is the real run in the local lab. | [payment](https://sepolia.basescan.org/tx/0x2e6839c430819054eeafe6a747c4b9446e9d819af4dc650a376555bb7d8d3fa8), [reverted withdrawal](https://sepolia.basescan.org/tx/0x66b9776e481c880d196a5685a9ed0dd7ceb2380f8326efcba886617c890d1fb0), [testnet runs](https://hashspan.dev/docs/runs) | ## Tracing backends Where the agent trace was sent over OTLP and opened in the UI. Setup for each is on the backends page. | Setup | Run | What was checked | Evidence | |---|---|---|---| | Jaeger (2.21.0) | 2026-10-03, hashspan 0.3.0 to 0.9.0 | The local lab backend: the full trace, with span links between send and confirm. | [local lab](https://hashspan.dev/docs/local-lab) | | Prometheus and Grafana (Prometheus 3.15.0, Grafana 13.2.3) | 2026-10-03, hashspan 0.9.0 | The send, confirmation and fee histograms over OTLP into Prometheus, on the hashspan Grafana dashboard. | [recipe](https://hashspan.dev/docs/recipes/fees-in-grafana), [dashboard](https://hashspan.dev/docs/backends#grafana-dashboard-for-the-metrics) | | Grafana Tempo (Tempo 3.0.0, Grafana 13.2.3) | 2026-10-02, hashspan 0.5.0 | All 13 spans received; TraceQL finds the failed confirm under its tool call. | [hashspan#135](https://github.com/selimaytac/hashspan/pull/135), [setup](https://hashspan.dev/docs/backends) | | Langfuse, self-hosted (4.49.0) | 2026-10-02, hashspan 0.5.0 | All 13 observations; GenAI spans become agent, tool and generation observations, the reverted confirm has level ERROR. | [hashspan#135](https://github.com/selimaytac/hashspan/pull/135), [setup](https://hashspan.dev/docs/backends) | | Honeycomb (US region) | 2026-10-02, hashspan 0.5.0 | All 13 spans; attribute types kept; the GenAI view shows the agent span with its token usage. | [hashspan#135](https://github.com/selimaytac/hashspan/pull/135), [setup](https://hashspan.dev/docs/backends) | ## Other setups Run once, end to end, and checked by hand. | Setup | Run | What was checked | Evidence | |---|---|---|---| | OpenTelemetry JS SDK, from an empty project (sdk-trace-node 2.11.0, exporter-trace-otlp-proto 0.222.0, viem 2.57.2) | 2026-10-03, hashspan 0.9.0 | The quickstart’s complete example, from npm init to an OTLP export with the tool, send and confirm spans. The home page, quickstart and recipe code samples are also typechecked daily against the latest packages on npm. | [quickstart](https://hashspan.dev/docs/quickstart#a-complete-example-on-a-local-chain) | | A service that is not an agent (viem 2.57.2, sdk-trace-node 2.11.0, local Anvil (chain 31337)) | 2026-10-05, hashspan 0.12.0 | A request handler span with a send and a confirm span under it, and no agent option: the confirm span has status success, block, gas used and fee, and no span has a gen_ai attribute. The account is an address only (Anvil signs); the service.name of the OpenTelemetry resource tells services apart. | [home page](https://hashspan.dev/#services), [quickstart](https://hashspan.dev/docs/quickstart) | | An EIP-1193 provider through viem custom() (viem 2.57.2, a provider that forwards to Anvil (chain 31337)) | 2026-10-05, hashspan 0.12.0 | A job span with a send and a confirm span under it. The provider received eth_chainId and eth_sendTransaction; the confirm span has status success and the fee. This is the path Circle and Fireblocks providers use. Neither service was run, only the path. | [wallet services](https://hashspan.dev/docs/integrations#wallet-services) | | Pimlico OpenTelemetry instrumentation for viem (0.0.5-alpha.1, viem 2.57.2) | 2026-10-02, hashspan 0.8.2 | Both on the same clients: its JSON-RPC spans for signing and sending nest under the send span, in one trace. Use it or traceTransport(), not both: with both, every request is recorded twice. | [compare page](https://hashspan.dev/compare#pimlico) | ## How releases are checked All of it is public in the [repository](https://github.com/selimaytac/hashspan): - **More than 2,000 tests:** unit tests and integration tests against Anvil, with coverage thresholds of 98% of lines and 92% of branches. - **Mutation testing** of the code that computes fees and matches receipts, so a wrong fee or a receipt given to the wrong transaction fails a test ([#207](https://github.com/selimaytac/hashspan/issues/207)). - **A real ERC-4337 bundler** for user operations, and every documented integration run against Anvil. - **Releases with npm provenance** and a CycloneDX SBOM attached to each GitHub release. - **OpenSSF Scorecard** on every push to main, with the workflows audited by zizmor. - **Bugs found upstream** while building hashspan are reported to their projects and listed, with the fix and the release that carries it, in [upstream findings](https://github.com/selimaytac/hashspan/blob/main/docs/upstream-findings.md). --- # Troubleshooting Source: https://hashspan.dev/docs/troubleshooting Synced from GitHub This page mirrors docs/troubleshooting.md at main. Organised by what you see in your backend. Each entry says why it happens and how to fix it, and links to the place where the behaviour is defined. For the setups that were checked, see [integrations](https://hashspan.dev/docs/integrations). ## The process exits with ECONNREFUSED on port 4318 Nothing listens on the OTLP endpoint, `http://localhost:4318` unless `OTEL_EXPORTER_OTLP_ENDPOINT` says otherwise. The spans are recorded and `hashspan.flush()` still resolves, but exporting them fails, and `sdk.shutdown()` rejects with the connection error once the exporter gives up: a script that awaits it at top level, as the quick start does, exits with code 1. Start a [backend](https://hashspan.dev/docs/backends), or print the spans in the terminal ([console](https://hashspan.dev/docs/backends#console-without-a-backend)). ## No hashspan spans at all - **No tracer provider is registered.** hashspan records through `@opentelemetry/api`, which drops spans until an OpenTelemetry SDK registers a provider. Start the SDK before your agent sends anything, or pass `tracerProvider` ([core options](https://hashspan.dev/docs/adapters/core#options)). - **The client that sends was not extended.** Only clients extended with `withHashspan()` are traced, and a library that builds its own client or calls viem's actions as functions (`sendTransaction(client, ...)`) bypasses the extension ([libraries that take a viem client](https://hashspan.dev/docs/integrations#libraries-that-take-a-viem-client)). For a transaction you only learn the hash of, use [`watch()`](https://hashspan.dev/docs/adapters/viem#transactions-sent-elsewhere). - **Another extension was applied after `withHashspan()`** and replaced the traced actions ([apply it last](https://hashspan.dev/docs/adapters/viem#apply-it-last)). - **User operations or call batches are missing with a tracker you passed yourself.** A tracker from an older `@hashspan/core` records only what that core supports; the calls are passed on untraced ([viem options](https://hashspan.dev/docs/adapters/viem#usage)). - **x402 payments are missing.** x402 v1 payments and networks other than `eip155` are not traced, and a hook of your own registered before hashspan's can keep it from seeing the outcome ([x402 usage](https://hashspan.dev/docs/adapters/x402#usage), [known limits](https://hashspan.dev/docs/adapters/x402#known-limits)). - **MPP payments are missing.** Only Tempo `charge` payments made through the client `Mppx.create()` returns (its `fetch` or the global `fetch` it polyfills) are traced: sessions, other methods and paid MCP tools are not, and a tracker from a core before 1.2 traces nothing, with a `diag` warning ([MPP usage](https://hashspan.dev/docs/adapters/mpp#usage), [known limits](https://hashspan.dev/docs/adapters/mpp#known-limits)). - **A span processor filters them out.** Langfuse's `LangfuseSpanProcessor` exports by default only GenAI and known LLM spans, and drops hashspan's spans without an agent identity and its JSON-RPC spans, logging that at debug level only ([keep them](https://hashspan.dev/docs/backends#with-langfusespanprocessor)). ## Spans in a separate trace from the agent's hashspan's spans are children of the span that is active when the transaction is sent. - **No span is active at the call.** Run the call inside your framework's tool span, or inside an active span of your own ([agent frameworks](https://hashspan.dev/docs/integrations#agent-frameworks)). - **The framework records tool spans without making them active**, as OpenInference's LangChain JS and OpenAI Agents instrumentations do: run the tool's function in an active span of your own ([agent frameworks](https://hashspan.dev/docs/integrations#agent-frameworks)). - **Mastra without observability:** the send and confirm spans then start traces of their own; configure its OpenTelemetry bridge ([agent frameworks](https://hashspan.dev/docs/integrations#agent-frameworks)). - **A confirm span is not linked to its send span.** Clients extended with different `withHashspan()` results have different trackers: reuse one result for every client of an agent ([viem usage](https://hashspan.dev/docs/adapters/viem#usage)), and give `@hashspan/cdp` and `@hashspan/x402` the same `tracker`. Without the link, the fee sample of an x402 settlement is also recorded as paid by its sender, not with `blockchain.fee.payer` `facilitator`. Links are kept for the tracker's `linkTtlMs` (10 minutes by default; [core options](https://hashspan.dev/docs/adapters/core#options)). ## A confirm span with no send span next to it A confirm span usually sits next to its send span, under the same parent: the span active at the wait, or the send span's parent for a confirmation in the background or through `watch()` ([confirm span parent](https://hashspan.dev/docs/reference/semantic-conventions#spans)). When no span was active at the send, the send span has no parent, and the confirm span then starts a trace of its own: only its span link relates it to the send. - **The backend keeps no span links.** Langfuse does not store them, so a confirm span in a trace of its own shows no relation to its send there ([which backends keep links](https://hashspan.dev/docs/backends#span-links)). Run the send inside an active span, so that both spans share a parent and a trace, or relate them by the key both carry: | Span | Key on both spans | |---|---| | transaction (`send`, `confirm`; a `payment` span carries its settlement's) | `blockchain.tx.hash` | | user operation | `blockchain.user_operation.hash`, with `blockchain.chain.id` | | call batch | `blockchain.call_batch.id` | In Grafana Tempo, for example, `{span.blockchain.tx.hash = "0x…"}` finds the trace of the send and the trace of the confirm span. - **The confirm span has no link either.** See [a confirm span is not linked to its send span](#spans-in-a-separate-trace-from-the-agents). ## A send span but no confirm span - **The process exited before the confirmation ended.** Background confirmations, revert reasons and sealed receipts are recorded after your call returned: call `flush()` before shutting the SDK down ([shutting down](https://hashspan.dev/docs/adapters/viem#shutting-down)). - **Nothing waited on a traced client.** A library that waits for receipts on a client of its own records no confirm span; use [background confirmation](https://hashspan.dev/docs/adapters/viem#background-confirmation). With `@hashspan/cdp` and `@hashspan/x402`, confirmations need a `reader` ([cdp usage](https://hashspan.dev/docs/adapters/cdp#usage), [x402 usage](https://hashspan.dev/docs/adapters/x402#usage)). - **The background confirmation limit was reached.** A transaction sent while `maxBackgroundConfirmations` (256 by default) confirmations are polling gets no background confirm span, and a `diag` warning is logged ([background confirmation](https://hashspan.dev/docs/adapters/viem#background-confirmation)). - **A sampler kept the send and dropped the confirm span, or the other way round.** With a parent-based sampler, a span follows its parent's decision. A confirm span's parent is the span active when it starts, else the parent of its send (the tool span), so a send and its confirm span are sampled together, also when the confirmation ends after the tool span (background confirmation, `watch()`, an x402 settlement). They are sampled apart when the confirm span gets another parent: `watch()` run inside another trace (the confirm span joins that trace), a hash the tracker did not send, or a send it no longer keeps (`maxTrackedTransactions`, `linkTtlMs`), whose confirm span starts a trace of its own ([core options](https://hashspan.dev/docs/adapters/core#options)). - **User operations and call batches** get a confirm span only from a wait you make (`waitForUserOperationReceipt`, CDP's `waitForUserOperation`, `waitForCallsStatus`); background confirmation and `watch()` cover transactions only ([smart accounts](https://hashspan.dev/docs/adapters/viem#smart-accounts-erc-4337), [call batches](https://hashspan.dev/docs/adapters/viem#call-batches-eip-5792)). A confirm span that gave up waiting is not missing: it ends with error status and `error.type` `timeout` ([semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions#span-status)). ## A confirm span disagrees with the chain after a reorganisation A confirm span keeps the receipt its wait ended with; what a reorganisation changes afterwards, or during a wait for several confirmations, is described under [chain reorganisations](https://hashspan.dev/docs/reference/semantic-conventions#spans). ## Fees missing from a confirm span - **The receipt was a flashblocks preconfirmation** (Base) and no sealed receipt came in time, or `flush()` could not wait for it: the span has no `effective_gas_price`, `l1_fee` or `fee` ([preconfirmed receipts](https://hashspan.dev/docs/adapters/viem#preconfirmed-receipts-flashblocks)). With `@hashspan/cdp` and no reader, a preconfirmed receipt from the SDK's wait is recorded the same way ([traced calls](https://hashspan.dev/docs/adapters/cdp#traced)). - **The receipt has no gas price.** `blockchain.tx.fee` needs `effective_gas_price` ([semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions#attributes)). - **User operations and call batches** record no transaction fee by design: a user operation records its own cost, and a call batch none ([semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions#spans)). ## Revert reason missing from a reverted transaction - **Decoding is off.** `@hashspan/viem` and `@hashspan/cdp` decode by default; `@hashspan/x402` only with `decodeRevertReason: true` ([revert reasons](https://hashspan.dev/docs/adapters/viem#revert-reasons), [x402 usage](https://hashspan.dev/docs/adapters/x402#usage)). - **The replay could not reproduce it.** The reason comes from replaying the transaction: a provider without historical state cannot replay it, earlier transactions in the same block can change the result, and the replay has a timeout (10 s by default). It is best effort ([revert reasons](https://hashspan.dev/docs/adapters/viem#revert-reasons)). - **A custom error shows as a selector.** Pass the contract's ABI (`writeContract` does, `watch()` takes `abi`) to decode it by name ([revert reasons](https://hashspan.dev/docs/adapters/viem#revert-reasons)). ## Addresses, arguments or error messages missing These are privacy defaults, not failures: addresses follow the `address` mode, call arguments need `recordFunctionArguments`, and error messages need `errorMessages`; a `redact` hook can remove more ([core options](https://hashspan.dev/docs/adapters/core#options), [privacy notes](https://hashspan.dev/docs/adapters/core#privacy-notes)). Concepts Short explanations of the ideas on this page: Context propagation in async Node.js Span links: connecting spans that aren't parent and child Exporters and OTLP: getting spans to Jaeger or Langfuse --- # viem OpenTelemetry instrumentation Source: https://hashspan.dev/docs/viem-opentelemetry Yes, viem has OpenTelemetry instrumentation: [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem), an open-source (Apache-2.0) client extension. Add `.extend(withHashspan())` to your viem clients, and each `sendTransaction` or `writeContract` becomes a `send {chainId}` span. Each `waitForTransactionReceipt` becomes a linked `confirm {chainId}` span with the status, gas, fee and revert reason. - **Spans:** `send` when the hash comes back, `confirm` when the receipt does, and one span per JSON-RPC request with `traceTransport()`. - **Setup:** one package, any OpenTelemetry SDK, about 10 lines. No change to how you call viem. - **Covers:** `sendTransaction`, `writeContract`, `sendRawTransaction`, their sync forms, ERC-4337 user operations and EIP-5792 call batches. ## How do I add OpenTelemetry to viem? Install the adapter next to viem and the OpenTelemetry API: ```sh npm install @hashspan/viem @opentelemetry/api viem ``` Any OpenTelemetry SDK setup works. To see spans in the terminal first, this one prints them: ```sh npm install @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base ``` ```ts title="telemetry.ts" import { ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'; // Prints every span to the terminal. Swap in an OTLP exporter for Jaeger, Tempo or Honeycomb. export const provider = new NodeTracerProvider({ spanProcessors: [new SimpleSpanProcessor(new ConsoleSpanExporter())], }); provider.register(); ``` Then extend your clients. This sends on a local [Anvil](https://www.getfoundry.sh/anvil) chain: ```ts title="send.ts" import { provider } from './telemetry.js'; import { trace } from '@opentelemetry/api'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { privateKeyToAccount } from 'viem/accounts'; import { foundry } from 'viem/chains'; import { traceTransport, withHashspan } from '@hashspan/viem'; // Anvil's first test account. Its key is public: never use it on a real chain. const account = privateKeyToAccount( '0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80', ); const to = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; const hashspan = withHashspan(); const wallet = createWalletClient({ account, chain: foundry, transport: traceTransport(http()), // one span per JSON-RPC request }).extend(hashspan); const reader = createPublicClient({ chain: foundry, transport: http() }).extend(hashspan); await trace.getTracer('my-app').startActiveSpan('pay invoice', async (span) => { const hash = await wallet.sendTransaction({ to, value: parseEther('0.01') }); // send span await reader.waitForTransactionReceipt({ hash }); // confirm span span.end(); }); await hashspan.flush(); await provider.shutdown(); ``` Run `anvil` in one terminal and `npx tsx send.ts` in another. Reuse one `withHashspan()` result for every client, so each confirm span links to its send span. `flush()` waits for spans that end after your code moved on; long-running services don't need it. ## What does a viem transaction look like as a trace? From that run (`@hashspan/viem` 1.1.1, viem 2.57.7, Anvil 1.8.3): ```text pay invoice ├─ send 31337 (76 ms) │ ├─ eth_fillTransaction (61 ms) │ └─ eth_sendRawTransaction (7 ms) ├─ eth_getTransactionByHash (3 ms) └─ confirm 31337 (9 ms), links to send blockchain.tx.status = success blockchain.block.number = 1 blockchain.tx.gas.used = 21000 blockchain.tx.fee = 42000000000000 ``` The spans nest under whatever span is active when you send: here `pay invoice`, in an agent the framework's [tool call span](https://hashspan.dev/learn/otel-traces-and-spans). The confirm span is a sibling of the send span and points back to it with a [span link](https://hashspan.dev/learn/otel-span-links). | Span | Main attributes | |---|---| | `send 31337` | `blockchain.tx.hash`, `.from`, `.to`, `.value` | | `confirm 31337` | `blockchain.tx.status`, `blockchain.block.number`, `blockchain.tx.gas.used`, `.effective_gas_price`, `.fee` | | `eth_*` | `rpc.method`, `rpc.system.name` `jsonrpc`, `server.address`, `blockchain.chain.id` | JSON-RPC spans record no parameters or results, and of the URL only host and port. The `eth_getTransactionByHash` comes from hashspan: it reads the nonce of a send that passed none, off your call's path, so a replacement can be found later. Every name is listed in the [semantic conventions](https://hashspan.dev/docs/reference/semantic-conventions). ## What happens when a writeContract reverts? A reverted receipt is a value, not an exception, so your code and its parent span carry on as if nothing failed. The confirm span doesn't. For a reverted transaction, hashspan replays the call to read the reason: ```ts const hash = await wallet.writeContract({ address: vault, abi: vaultAbi, functionName: 'withdraw', args: [250n], gas: 100_000n, // skip the estimate, so the revert happens on chain }); await reader.waitForTransactionReceipt({ hash }); ``` In the revert run, the send span recorded `blockchain.contract.function.name` `withdraw` and its selector `0x2e1a7d4d`. The confirm span ended with error status: ```text wrap confirm 31337 ✕ error blockchain.tx.status = reverted blockchain.tx.revert.reason = WithdrawalLimitExceeded(100, 250) error.type = reverted ``` The `withdraw` span that sent it had no error. Custom errors decode when the ABI is known, as with `writeContract`. Turn the replay off with `withHashspan({ decodeRevertReason: false })`. The [transaction lifecycle](https://hashspan.dev/learn/tx-lifecycle) explains why a mined transaction can still fail. ## What if my code never calls waitForTransactionReceipt? Many agent frameworks send and return the hash. Background confirmation gives every sent transaction a confirm span anyway: ```ts const hashspan = withHashspan({ confirm: { mode: 'background', timeoutMs: 60_000 } }); const wallet = createWalletClient({ account, chain, transport: http() }).extend(hashspan); await wallet.sendTransaction({ to, value }); // no wait in your code ``` The adapter polls for the receipt through the sending client and doesn't delay the send. In the run, the confirm span (`blockchain.tx.status` `success`, block 5) landed under the same parent as the send span. Polling adds one receipt request per polling interval, and at most 256 background confirmations poll at once by default (`maxBackgroundConfirmations`). ## Which viem actions are traced? | Action | Spans | |---|---| | `sendTransaction`, `writeContract` | `send` | | `sendRawTransaction` | `send`, without the sender, which only the signature gives | | `sendTransactionSync`, `writeContractSync`, `sendRawTransactionSync` | `send` and `confirm` | | `waitForTransactionReceipt` | `confirm` | | `sendUserOperation`, `waitForUserOperationReceipt` (bundler client) | `send` and `confirm` of an ERC-4337 user operation | | `sendCalls`, `waitForCallsStatus`, `sendCallsSync` | `send` and `confirm` of an EIP-5792 call batch | `deployContract` isn't traced yet ([#36](https://github.com/selimaytac/hashspan/issues/36)). Apply `withHashspan()` after extensions such as `publicActions`, which define `waitForTransactionReceipt` again. Smart account SDKs and the full option list are in the [viem adapter reference](https://hashspan.dev/docs/adapters/viem#smart-accounts-erc-4337). ## Can I write viem tracing by hand? You can. The usual pattern wraps the transport in a span per request, or overrides `sendTransaction` with `client.extend()`: ```ts .extend((client) => ({ sendTransaction: (args) => tracer.startActiveSpan('sendTransaction', async (span) => { try { return await client.sendTransaction(args); } finally { span.end(); } }), })) ``` That gives you the duration of the call. The span ends when the hash comes back, before anyone reads the receipt. In the run, the receipt said `success` and no span recorded it. What a send span alone can't tell you: - **The outcome.** Mined, reverted or never mined is only in the receipt, which arrives later and often in another function. - **Replacement.** When a sped-up or cancelled transaction replaces yours, viem resolves the wait with another hash's receipt. hashspan ends the awaited hash as `replaced` and records the receipt on the hash that was mined. - **Fees.** Gas used, effective gas price and, on OP Stack chains, the L1 data fee, taken from the sealed receipt. - **The revert reason.** A receipt only says `reverted`. The reason needs a replay at the previous block's state. Write it yourself if a span around the call is all you need. If you want the outcome in the trace, the adapter already handles those cases. ## FAQ ### Does viem have built-in OpenTelemetry support? viem doesn't ship OpenTelemetry instrumentation itself. @hashspan/viem adds it as a client extension, so your calls and their results stay unchanged. ### Does it work with OTLP, Jaeger or Grafana Tempo? Yes. hashspan only uses the OpenTelemetry API (@opentelemetry/api), so spans go wherever your SDK exports them: an OTLP endpoint, Jaeger, Grafana Tempo, Honeycomb or Langfuse. ### Does it record private keys or call arguments? No keys or signatures. Function arguments are off by default and need recordFunctionArguments: true. JSON-RPC spans keep no parameters, results or URL path. ### Does it work with ethers or web3.js? No. hashspan has adapters for viem, the Coinbase CDP SDK, x402 and MPP (mppx). Another send path can use @hashspan/core, the tracker behind every adapter, directly. ### Does tracing slow down viem transactions? Slightly. On a local chain, @hashspan/viem 1.3.1 added about 0.25 ms at p50 to a send plus receipt wait, about 0.5 ms when viem polls for the receipt, and one eth_getTransactionByHash per send, none when you pass nonce. The tracing overhead benchmark has the method, the data and the caveats. ### Which Node.js and viem versions does it need? Node.js 22.3 or later and viem 2.21.0 or later (the package's peer range). The run on this page used viem 2.57.7. How much tracing adds per transaction, with the method and data: [tracing overhead benchmark](https://hashspan.dev/docs/benchmarks/tracing-overhead). Never sent a transaction before? The [x402 tutorial](https://hashspan.dev/docs/x402-tutorial) walks through one from an empty wallet. The [Quickstart](https://hashspan.dev/docs/quickstart) sets up OTLP and Jaeger, and [tracing backends](https://hashspan.dev/docs/backends) covers Tempo, Honeycomb and Langfuse. For a send path without an adapter, see [`@hashspan/core`](https://hashspan.dev/docs/adapters/core). The adapter is open source: [selimaytac/hashspan](https://github.com/selimaytac/hashspan). Questions and bug reports go to its issues. --- # x402 tutorial: your first AI agent that pays (and the trace that shows it) Source: https://hashspan.dev/docs/x402-tutorial This x402 tutorial takes you from no wallet to an AI agent that pays for an API call, and shows every payment as a trace. Everything runs on a local chain on your machine: no sign-up, no faucet, no real money. - **Part 1:** give an [AI SDK](https://ai-sdk.dev) agent a wallet, watch it fail with no funds, fund it, and pay. - **Part 2:** the agent pays 0.001 of a test token per request with x402. It only signs; someone else sends the transaction. - **Each step** ends with a checkpoint and the trace that shows what happened. The complete, CI-tested version of this code is [examples/x402-agent](https://github.com/selimaytac/hashspan/tree/main/examples/x402-agent) in the hashspan repository. [Run it instead](#want-the-complete-ci-tested-version) if you'd rather clone than type. Know wallets and gas already? [Jump to Part 2](#part-2-how-does-an-agent-pay-for-an-api-call-with-x402). It builds on Part 1's setup: Foundry with Anvil running, Jaeger, and the project from steps 1 to 4 (`npm init`, `telemetry.ts`, `wallet.ts`, `model.ts`). **Before you start:** - **You need:** Node.js 22.3 or later, Docker, a terminal and about 30 minutes. On Windows, run everything in WSL: Foundry's installer needs WSL or Git Bash, and doesn't support PowerShell or Command Prompt. - **No real money.** Both parts use Anvil, a private chain on your machine, and test tokens. No credit card, no API key. - **Tested** on 2026-10-19 with Node.js 22.14, Foundry 1.8.5, solc 0.8.28, `ai` 7.0.143, viem 2.57.7, `@x402/*` 2.28.0, `@hashspan/viem` 1.3.0, `@hashspan/x402` 1.2.1 and Jaeger 2.21.0. Figure: What you build, all on a local chain. Part 1: the agent's pay_vendor tool signs and sends 0.005 ETH to a vendor, and your wallet pays the gas; Jaeger shows execute_tool pay_vendor with a send 31337 and a confirm 31337 span with status success. Part 2, x402: the get_weather tool only signs, a facilitator sends the 0.001 TUSD payment to the paid API and pays the gas; Jaeger shows execute_tool get_weather with a payment 31337 span with status settled and a confirm 31337 span, and no send span. By the end you can: 1. Create a wallet and keep its key out of your repository and your prompts. 2. Fund it, and tell ETH, which pays for gas, from a token, which pays for things. 3. Give an agent a tool that pays, with a cap on what it can spend. 4. Read a payment in a trace, and spot three failures in it. ## Part 1: how do you give an AI agent a wallet? A wallet is a private key. The key is a secret number; the address is derived from it and is safe to share. Whoever holds the key can spend what the address holds. So the key lives in a file only you can read, and never in your code, your repository or a prompt. Part 1 runs on Anvil, a private blockchain on your machine. It starts empty, mines instantly and forgets everything when you stop it. Its chain id is 31337. 1. ### Start a local chain and Jaeger Install Foundry, which ships Anvil and the `cast` command line tool. The installer puts it in `~/.foundry/bin`. Add that folder to your `PATH` in your shell profile (`~/.zshrc` here, `~/.bashrc` on bash), so every terminal finds it: ```sh curl -L https://foundry.paradigm.xyz | bash echo 'export PATH="$PATH:$HOME/.foundry/bin"' >> ~/.zshrc export PATH="$PATH:$HOME/.foundry/bin" foundryup ``` Then start the chain in one terminal. It listens on localhost:8545: ```sh anvil ``` In a second terminal, start Jaeger, a trace viewer: ```sh wrap docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:2.21.0 ``` **Checkpoint:** in a third terminal, `cast block-number` prints `0`, and [localhost:16686](http://localhost:16686) opens Jaeger. 2. ### Create the project ```sh wrap mkdir first-paying-agent && cd first-paying-agent npm init -y && npm pkg set type=module npm install ai @ai-sdk/otel viem zod @hashspan/viem @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources @opentelemetry/semantic-conventions tsx printf 'node_modules\n*.hex\n' > .gitignore ``` The last line keeps key files (`*.hex`) out of git. Next, tracing. A **span** is one timed piece of work with a name and some attributes, such as "the tool call" or "sending the transaction". A **trace** is the tree of spans for one run of the agent. [Traces and spans](https://hashspan.dev/learn/otel-traces-and-spans) shows one in three minutes. Add `telemetry.ts`. It sends the agent's spans and hashspan's spans to Jaeger, through one OpenTelemetry provider: ```ts title="telemetry.ts" import { OpenTelemetry } from '@ai-sdk/otel'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto'; import { resourceFromAttributes } from '@opentelemetry/resources'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'; import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'; import { registerTelemetry } from 'ai'; export const provider = new NodeTracerProvider({ resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'first-paying-agent' }), // Sends spans to Jaeger on localhost:4318 spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())], }); provider.register(); // The AI SDK's own spans (invoke_agent, execute_tool) use the same provider. registerTelemetry(new OpenTelemetry()); ``` 3. ### Give the agent a wallet `wallet.ts` loads a key from a file, or makes a new one there the first time: ```ts title="wallet.ts" import { existsSync, readFileSync, writeFileSync } from 'node:fs'; import { type Hex } from 'viem'; import { generatePrivateKey, privateKeyToAccount } from 'viem/accounts'; // Reads the private key from `file`, or creates a new one there. // Only you can read the file (mode 600). Never commit it, never paste it in a prompt. export function loadAccount(file: string) { if (!existsSync(file)) writeFileSync(file, generatePrivateKey(), { mode: 0o600 }); return privateKeyToAccount(readFileSync(file, 'utf8').trim() as Hex); } ``` `balance.ts` prints the address and what it holds: ```ts title="balance.ts" import { createPublicClient, formatEther, http } from 'viem'; import { foundry } from 'viem/chains'; import { loadAccount } from './wallet.js'; const account = loadAccount('agent-wallet.hex'); const reader = createPublicClient({ chain: foundry, transport: http() }); const balance = await reader.getBalance({ address: account.address }); console.log(`address: ${account.address}`); console.log(`balance: ${formatEther(balance)} ETH`); ``` **Checkpoint:** `npx tsx balance.ts` prints a new address and `balance: 0 ETH`. Yours differs from mine: ```text address: 0xfA818720ae6029D0F76C84A377cA6cD610d40dDb balance: 0 ETH ``` **What just happened:** viem made a random private key and wrote it to `agent-wallet.hex`. The address comes from the key, so the same file always gives the same address. Nothing was sent to the chain: a new address exists before anyone pays it. 4. ### Let the agent pay with an empty wallet The agent gets one tool, `pay_vendor`. The tool refuses anything over 0.01 ETH before it signs: a cap in your own code is the first safety net, whatever the model asks for. ```ts title="agent.ts" import { provider } from './telemetry.js'; import { withHashspan } from '@hashspan/viem'; import { stepCountIs, tool, ToolLoopAgent } from 'ai'; import { createPublicClient, createWalletClient, http, parseEther } from 'viem'; import { foundry } from 'viem/chains'; import { z } from 'zod'; import { scriptedModel } from './model.js'; import { loadAccount } from './wallet.js'; const hashspan = withHashspan({ agent: { name: 'first-paying-agent' } }); const account = loadAccount('agent-wallet.hex'); // Anvil mines at once, so poll for the receipt every 250 ms instead of viem's default 4 s. const options = { chain: foundry, transport: http(), pollingInterval: 250 }; const wallet = createWalletClient({ account, ...options }).extend(hashspan); const reader = createPublicClient(options).extend(hashspan); const VENDOR = '0x70997970C51812dc3A010C7d01b50e0d17dc79C8'; // Anvil's second account const MAX_ETH = '0.01'; const payVendor = tool({ description: `Pay the vendor in ETH. At most ${MAX_ETH} ETH per call.`, inputSchema: z.object({ amountEth: z.string() }), execute: async ({ amountEth }) => { const value = parseEther(amountEth); if (value > parseEther(MAX_ETH)) throw new Error(`Refused: over ${MAX_ETH} ETH`); const hash = await wallet.sendTransaction({ to: VENDOR, value }); const receipt = await reader.waitForTransactionReceipt({ hash }); return { hash, status: receipt.status }; }, }); const agent = new ToolLoopAgent({ model: scriptedModel('pay_vendor', { amountEth: '0.005' }), telemetry: { functionId: 'first-paying-agent' }, instructions: 'You pay vendors with the pay_vendor tool.', tools: { pay_vendor: payVendor }, stopWhen: stepCountIs(3), }); const result = await agent.generate({ prompt: 'Pay the vendor 0.005 ETH.' }); console.log(result.text); await hashspan.flush(); await provider.shutdown(); ``` `.extend(hashspan)` is the only tracing code. The AI SDK runs each tool inside an `execute_tool` span, and hashspan puts its spans under whatever span is active. The model is scripted, so you need no API key. It calls the tool once, then repeats what the tool returned. `model.ts` below holds it. To use a real LLM instead, swap the `model` line for an [AI SDK provider](https://ai-sdk.dev/docs/foundations/providers-and-models). With [OpenAI](https://ai-sdk.dev/providers/ai-sdk-providers/openai), for example, `npm install @ai-sdk/openai`, set `OPENAI_API_KEY`, and: ```ts import { openai } from '@ai-sdk/openai'; // in the ToolLoopAgent options, instead of scriptedModel(...): model: openai('model-id'), // a model id from the provider page ``` hashspan's spans come from the payment calls inside the tool, not from the model. So the trace has the same shape whichever model calls the tool. If a real model takes another path, such as asking a question instead of paying, the trace shows that too. ```ts title="model.ts" import { MockLanguageModelV4 } from 'ai/test'; const usage = { inputTokens: { total: 0, noCache: 0, cacheRead: 0, cacheWrite: 0 }, outputTokens: { total: 0, text: 0, reasoning: 0 }, }; // Stands in for an LLM, so the tutorial runs without an API key. // Turn 1: call `tool` with `input`. Turn 2: report what the tool returned. export function scriptedModel(tool: string, input: object) { return new MockLanguageModelV4({ provider: 'scripted', modelId: 'scripted-model', doGenerate: async ({ prompt }) => { const last = prompt.at(-1); if (last?.role !== 'tool') { return { content: [{ type: 'tool-call', toolCallId: 'call_1', toolName: tool, input: JSON.stringify(input) }], finishReason: { unified: 'tool-calls', raw: 'tool_calls' }, usage, warnings: [], }; } const result = last.content.find((part) => part.type === 'tool-result'); return { content: [{ type: 'text', text: `Tool result: ${JSON.stringify(result?.output)}` }], finishReason: { unified: 'stop', raw: 'stop' }, usage, warnings: [], }; }, }); } ``` Run it with the empty wallet: `npx tsx agent.ts`. **Checkpoint:** the tool fails. The output starts with: ```text wrap Tool result: {"type":"error-text","value":"TransactionExecutionError: The total cost (gas * gas fee + value) of executing this transaction exceeds the balance of the account. ``` In Jaeger, pick the service `first-paying-agent` and open the trace. The drawings on this page are simplified: Jaeger also shows a `step 1` span with a `chat scripted-model` span in it, and `execute_tool` sits under `step 1`, one level deeper. Trace (agent.ts with 0 ETH, simplified · durations from the run): - `invoke_agent scripted-model`: 53 ms - `execute_tool pay_vendor`: 47 ms · error - `send 31337`: 34 ms · InsufficientFundsError The send span carries error.type = InsufficientFundsError, and the tool span is marked as an error. **What just happened:** Every transaction costs gas, paid in ETH by the sender, on top of the value sent. The wallet held nothing, so viem refused to send. Nothing reached the chain, and there is no `confirm` span. [Insufficient funds](https://hashspan.dev/errors/insufficient-funds) explains this error, and [the error decoder](https://hashspan.dev/errors/decode) reads any message like it. 5. ### Fund the wallet On a testnet, funds come from a faucet. On Anvil you set the balance yourself. Use your address from step 3: ```sh cast rpc anvil_setBalance 0xYourAddress 1000000000000000000 ``` It prints `null`: the call returns nothing. The amount is in wei: 1018 wei is 1 ETH. **Checkpoint:** `npx tsx balance.ts` prints `balance: 1 ETH`. 6. ### Pay again and read the trace Run `npx tsx agent.ts` again. **Checkpoint:** the tool returns a hash and `"status":"success"`, and `npx tsx balance.ts` prints `balance: 0.994958 ETH`. Trace (agent.ts with 1 ETH, simplified · durations from the run): - `invoke_agent scripted-model`: 311 ms - `execute_tool pay_vendor`: 301 ms - `send 31337`: 30 ms - `confirm 31337`: 269 ms · success send ends when the hash comes back. confirm ends when the receipt does, and links back to its send span. The `confirm 31337` span holds the receipt: | Attribute | Value | |---|---| | `blockchain.tx.status` | `success` | | `blockchain.tx.gas.used` | `21000` | | `blockchain.tx.effective_gas_price` | `2000000000` (2 gwei) | | `blockchain.tx.fee` | `42000000000000` (0.000042 ETH) | **What just happened:** The wallet signed a transaction and sent it. Anvil put it in block 1 and wrote a receipt. The fee is gas used times gas price: 21,000 × 2 gwei. That is why 1 ETH became 0.994958, not 0.995. [Transaction lifecycle](https://hashspan.dev/learn/tx-lifecycle) covers the steps, and [span links](https://hashspan.dev/learn/otel-span-links) explains the link from confirm to send. A receipt can also say `reverted`: the transaction was mined, but the contract refused it. That is a value, not an exception, so your tool carries on unless it checks `status`. The [viem OpenTelemetry page](https://hashspan.dev/docs/viem-opentelemetry#what-happens-when-a-writecontract-reverts) shows how that looks in a trace. ## Part 2: how does an agent pay for an API call with x402? x402 puts a price on an HTTP request. The server answers `402 Payment Required` with a price. The client signs a payment and sends the request again. A third party, the facilitator, checks the signature and moves the money on chain. Figure: One x402 payment in four steps between the agent, the paid API and the facilitator: 1, the agent sends GET /weather and gets 402 with a PAYMENT-REQUIRED header naming the price, the asset and payTo; 2, the agent signs a token authorization, which is no transaction and costs no gas; 3, it sends the GET again with a PAYMENT-SIGNATURE header and the API asks the facilitator to verify and settle; 4, the facilitator sends the transfer and pays the gas, and the API answers 200 with a PAYMENT-RESPONSE header that carries the transaction hash. The big difference from Part 1 is step 2. The agent signs, but sends nothing. So it needs the token it pays with, and no ETH at all. Part 2 builds on Part 1. You need Anvil and Jaeger running, and the project from Part 1 steps 1 to 4: `npm init`, `telemetry.ts`, `wallet.ts` and `model.ts`. On a public network, the token would be USDC and the facilitator a hosted service. Here you deploy a test token and run the facilitator yourself, so nothing leaves your machine. Three words you meet around x402, in one line each: - **Testnet:** a public blockchain for testing, whose coins have no value. - **Faucet:** a website that gives out testnet coins for free. - **Stablecoin:** a token that tracks a currency, such as USDC, where 1 USDC tracks 1 US dollar. **Local test keys only:** The keys in this tutorial are throwaway keys for a local chain. Never put a key that holds real money into this code, and never point it at a mainnet. 1. ### Deploy a test token A token is a contract that keeps a balance per address. The x402 payment here uses [EIP-3009](https://eips.ethereum.org/EIPS/eip-3009): the holder signs a transfer, and anyone can submit it. USDC supports it. Test USD is a small one from the hashspan repository, made for tests: anyone can mint it. ```sh wrap curl -sLO "https://raw.githubusercontent.com/selimaytac/hashspan/%40hashspan/x402%401.2.1/packages/x402/test/token/TestUsd.sol" forge create TestUsd.sol:TestUsd --broadcast --rpc-url http://localhost:8545 --unlocked --from 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 ``` `forge` compiles the contract and deploys it from Anvil's first account, which Anvil unlocks for you. The output includes the address: ```text Deployed to: 0x5FbDB2315678afecb367f032d93F642f64180aa3 ``` `forge` also leaves `cache/` and `out/` folders. Keep them out of git: `printf 'cache\nout\n' >> .gitignore`. Keep it in a variable, in every terminal you use for Part 2: ```sh export TOKEN=0x5FbDB2315678afecb367f032d93F642f64180aa3 ``` 2. ### Give the buyer a wallet Add the x402 packages to the same project: ```sh wrap npm install @x402/core @x402/evm @x402/fetch @x402/hono hono @hono/node-server @hashspan/x402 ``` The paying agent gets its own key, in `buyer-wallet.hex`. `buyer-balance.ts` creates it and prints both balances: ```ts title="buyer-balance.ts" import { type Address, createPublicClient, erc20Abi, formatEther, formatUnits, http } from 'viem'; import { foundry } from 'viem/chains'; import { loadAccount } from './wallet.js'; const TOKEN = process.env.TOKEN as Address; const { address } = loadAccount('buyer-wallet.hex'); const reader = createPublicClient({ chain: foundry, transport: http() }); const [tusd, eth] = await Promise.all([ reader.readContract({ address: TOKEN, abi: erc20Abi, functionName: 'balanceOf', args: [address] }), reader.getBalance({ address }), ]); console.log(`address: ${address}`); console.log(`TUSD: ${formatUnits(tusd, 6)}`); console.log(`ETH: ${formatEther(eth)}`); ``` **Checkpoint:** `npx tsx buyer-balance.ts` prints a new address with `TUSD: 0` and `ETH: 0`. 3. ### Run a paid API on your machine `seller.ts` is a small API with one paid route, priced at 1000 units of Test USD, which is 0.001 TUSD (6 decimals, like USDC). The facilitator runs inside it, and settles from Anvil's first account, which pays the gas: ```ts title="seller.ts" import { serve } from '@hono/node-server'; import { x402Facilitator } from '@x402/core/facilitator'; import type { FacilitatorClient } from '@x402/core/server'; import type { SupportedResponse } from '@x402/core/types'; import { toFacilitatorEvmSigner } from '@x402/evm'; import { registerExactEvmScheme } from '@x402/evm/exact/facilitator'; import { ExactEvmScheme } from '@x402/evm/exact/server'; import { paymentMiddleware, x402ResourceServer } from '@x402/hono'; import { Hono } from 'hono'; import { type Address, createWalletClient, http, publicActions } from 'viem'; import { foundry } from 'viem/chains'; const TOKEN = process.env.TOKEN as Address; // the Test USD contract you deployed const PAY_TO = '0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC'; // Anvil's third account // The facilitator settles payments from Anvil's first account, which pays the gas. const settler = createWalletClient({ account: '0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266', chain: foundry, transport: http(), pollingInterval: 250, }).extend(publicActions); // viem's client types are wider than the ones x402 expects; the cast bridges them. type SignerInput = Parameters[0]; const signer = toFacilitatorEvmSigner({ ...settler, address: settler.account.address } as unknown as SignerInput); const facilitator = new x402Facilitator(); registerExactEvmScheme(facilitator, { signer, networks: 'eip155:31337' }); const facilitatorClient: FacilitatorClient = { verify: (payload, requirements) => facilitator.verify(payload, requirements), settle: (payload, requirements) => facilitator.settle(payload, requirements), getSupported: async () => facilitator.getSupported() as SupportedResponse, }; const server = new x402ResourceServer(facilitatorClient).register('eip155:31337', new ExactEvmScheme()); const price = { asset: TOKEN, amount: '1000', extra: { name: 'Test USD', version: '1' } }; // 0.001 TUSD const app = new Hono(); app.use( paymentMiddleware( { 'GET /weather': { accepts: [{ scheme: 'exact', price, network: 'eip155:31337', payTo: PAY_TO }] } }, server, ), ); app.get('/weather', (c) => c.json({ city: 'Lisbon', forecast: 'sunny, 24 °C' })); serve({ fetch: app.fetch, port: 4021 }); console.log('Paid API on http://localhost:4021/weather'); ``` Start it in a new terminal: ```sh cd first-paying-agent export TOKEN=0x5FbDB2315678afecb367f032d93F642f64180aa3 npx tsx seller.ts ``` `eip155:31337` is how x402 v2 names a chain: the chain id with a prefix. **Checkpoint:** `curl -i localhost:4021/weather` answers `HTTP/1.1 402 Payment Required`, with a `payment-required` header. That header is base64 JSON. To decode it: ```sh wrap curl -si localhost:4021/weather | grep -i '^payment-required' | cut -d' ' -f2 | tr -d '\r' | base64 -d ``` Its offer reads: ```json { "scheme": "exact", "network": "eip155:31337", "amount": "1000", "asset": "0x5FbDB2315678afecb367f032d93F642f64180aa3", "payTo": "0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC", "maxTimeoutSeconds": 300, "extra": { "name": "Test USD", "version": "1" } } ``` 4. ### Let the agent pay before it has tokens The buyer is a second agent with one tool, `get_weather`. The tool pays through `fetchWithPayment`, which handles the 402, the signature and the retry: ```ts title="buyer.ts" import { provider } from './telemetry.js'; import { withHashspan } from '@hashspan/x402'; import { x402Client } from '@x402/core/client'; import { registerExactEvmScheme } from '@x402/evm/exact/client'; import { wrapFetchWithPayment } from '@x402/fetch'; import { stepCountIs, tool, ToolLoopAgent } from 'ai'; import { type Address, createPublicClient, http } from 'viem'; import { foundry } from 'viem/chains'; import { z } from 'zod'; import { scriptedModel } from './model.js'; import { loadAccount } from './wallet.js'; const TOKEN = process.env.TOKEN as Address; const client = new x402Client(); // Before any hooks of your own, so none of them can hide a payment's outcome from hashspan. const hashspan = withHashspan(client, { agent: { name: 'first-paying-agent' }, reader: createPublicClient({ chain: foundry, transport: http(), pollingInterval: 250 }), }); registerExactEvmScheme(client, { signer: loadAccount('buyer-wallet.hex') }); // Allow Test USD, at most 0.01 (10000 units) per payment. client.setSpendControls({ allowedAssets: [{ network: 'eip155:31337', asset: TOKEN, maxAmountPerPayment: '10000' }], }); // Pay in Test USD only: drop every other asset an API offers. client.registerPolicy((_version, offers) => offers.filter((offer) => offer.asset.toLowerCase() === TOKEN.toLowerCase()), ); const fetchWithPayment = wrapFetchWithPayment(fetch, client); const getWeather = tool({ description: 'Get the weather forecast from a paid API (costs 0.001 TUSD).', inputSchema: z.object({}), execute: async () => { const response = await fetchWithPayment('http://localhost:4021/weather'); if (!response.ok) { // A refused payment comes back as a second 402; its header says why. const required = response.headers.get('payment-required'); const reason = required ? JSON.parse(atob(required)).error : ''; throw new Error(`Paid API answered ${response.status} ${reason}`); } return response.json(); }, }); const agent = new ToolLoopAgent({ model: scriptedModel('get_weather', {}), telemetry: { functionId: 'first-paying-agent' }, instructions: 'You answer weather questions with the get_weather tool.', tools: { get_weather: getWeather }, stopWhen: stepCountIs(3), }); const result = await agent.generate({ prompt: "What's the weather in Lisbon?" }); console.log(result.text); await hashspan.flush(); await provider.shutdown(); ``` The x402 client has two safety nets here. `setSpendControls` adds Test USD to the assets the client accepts, capped at 10000 units per payment. A list adds to the defaults, it doesn't replace them: USDC and the other default assets stay allowed, capped at $1 per payment. So `registerPolicy` drops every offer that isn't Test USD. Both refuse before anything is signed. `withHashspan` comes before any hooks you add yourself, as the [x402 adapter](https://hashspan.dev/docs/adapters/x402#usage) asks: a hook registered earlier could keep it from seeing a payment's outcome. Its `reader` lets it check each settlement on chain. Run `npx tsx buyer.ts` while the buyer still holds no tokens. **Checkpoint:** ```text wrap Tool result: {"type":"error-text","value":"Error: Paid API answered 402 invalid_exact_evm_transaction_simulation_failed"} ``` Trace (buyer.ts with 0 TUSD, simplified · durations from the run): - `invoke_agent scripted-model`: 110 ms - `execute_tool get_weather`: 105 ms · error - `payment 31337`: 65 ms · no_settlement The payment span records the offer it signed for, and error.type = no_settlement. **What just happened:** The agent signed for 0.001 TUSD, and the facilitator refused to settle it. The `payment 31337` span shows who paid whom, what and how much (`blockchain.payment.amount` `1000`), and that nothing settled. The payment span doesn't carry the reason. The second 402 does, in the `error` field of its `payment-required` header, and the tool puts it in its error. So the reason shows on the `execute_tool get_weather` span: ```text invalid_exact_evm_transaction_simulation_failed ``` The facilitator tried the transfer first, and it failed for lack of funds. 5. ### Mint test tokens to the buyer On a public testnet this is a faucet. Here, Test USD lets anyone mint. Use the buyer's address from step 2; 20000000 units are 20 TUSD: ```sh wrap cast send $TOKEN "mint(address,uint256)" 0xYourBuyerAddress 20000000 --unlocked --from 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 ``` **Checkpoint:** `npx tsx buyer-balance.ts` shows `TUSD: 20` and still `ETH: 0`. 6. ### Pay with tokens and read the trace Run `npx tsx buyer.ts` again. **Checkpoint:** the tool returns the forecast, `{"city":"Lisbon","forecast":"sunny, 24 °C"}`, and `buyer-balance.ts` shows `TUSD: 19.999` and still `ETH: 0`. Trace (buyer.ts with 20 TUSD, simplified · durations from the run): - `invoke_agent scripted-model`: 312 ms - `execute_tool get_weather`: 306 ms - `payment 31337`: 286 ms · settled - `confirm 31337`: 11 ms No send span: the agent sent no transaction. The confirm span links back to the payment span. | Span | Attribute | Value | |---|---|---| | `payment 31337` | `blockchain.payment.status` | `settled` | | | `blockchain.payment.verified` | `true` | | | `blockchain.payment.amount` | `1000` | | `confirm 31337` | `blockchain.tx.status` | `success` | | | `blockchain.tx.gas.used` | about 86,500 (`86486` in this run) | **What just happened:** The agent signed an authorization for exactly 1000 units to `payTo`. The facilitator sent it to the token contract, which moved the funds, and paid the gas: the buyer still holds 0 ETH. The payment span carries the settlement's transaction hash. With a `reader`, hashspan fetched that receipt and checked it carries this payment: that is `verified = true`. The confirm span holds the receipt, and its [span link](https://hashspan.dev/learn/otel-span-links) points back to the payment. ### What if the price is over your cap, or the asset is wrong? Set the cap below the price, `maxAmountPerPayment: '500'`, and run again. The x402 client refuses before it signs. Nothing is paid and no payment span is recorded. Your tool throws, so `execute_tool get_weather` carries the error: ```text wrap Failed to create payment payload: All payment requirements were rejected by spendControls.allowedAssets maxAmountPerPayment. ``` The policy works the same way. Make it keep another asset, such as Base Sepolia's USDC address, and the Test USD offer is dropped: ```text wrap Failed to create payment payload: All payment requirements were filtered out by policies for x402 version: 2 ``` Both are by design: a refused payment never left your process. The [x402 adapter](https://hashspan.dev/docs/adapters/x402#known-limits) lists what is and isn't traced. ## What do the failures look like in the trace? | Symptom | What the trace shows | Fix | |---|---|---| | Tool fails with "exceeds the balance" | `send` with `error.type` `InsufficientFundsError`, no `confirm` | fund the address; see [insufficient funds](https://hashspan.dev/errors/insufficient-funds) | | x402 request ends in a second 402 | `payment` with `error.type` `no_settlement`; the reason on the tool span | the `error` from the `payment-required` header; often a token balance below the price | | Tool fails with a `spendControls` or policy message | no payment span; the tool span has the error | raise `maxAmountPerPayment`, or check the asset your policy keeps | | Two parallel payments, one fails with "nonce too low" | a `send` with `error.type` `NonceTooLowError` | see [nonce too low](https://hashspan.dev/errors/nonce-too-low) and the [AI SDK recipe](https://hashspan.dev/docs/recipes/ai-sdk) | | A retry paid twice | two `settled` payment spans under one tool span | see [payment, request and task outcomes](https://hashspan.dev/docs/adapters/x402#payment-request-and-task-outcomes) | | No spans in Jaeger | nothing for `first-paying-agent` | import `telemetry.ts` first, and keep the `flush()` and `shutdown()` lines | | `anvil` fails to start | port 8545 already in use | stop the other chain, or run `anvil --port 8546` and set the RPC URL in `http()` | Paste any error message into [the error decoder](https://hashspan.dev/errors/decode) to see what it means and how hashspan records it. To look up an attribute name, use the [OpenTelemetry attribute lookup](https://hashspan.dev/learn/otel-attributes). ## What did you build? - An agent with a wallet whose key never enters your code, your repository or a prompt. - Two ways to pay: sending a transaction yourself (Part 1) and signing an x402 payment that someone else settles (Part 2). - Two spending caps: one in your tool, one in the x402 client. - A trace for every payment, failed or not, under the tool call that made it. On a public network, the paid API you call runs its own facilitator or uses a hosted one, and the token is a real one such as USDC. The [example's README](https://github.com/selimaytac/hashspan/tree/main/examples/x402-agent#base-sepolia-optional-not-run-in-ci) lists what changes for the Base Sepolia testnet. The setup on your side is two packages: [`@hashspan/viem`](https://hashspan.dev/docs/adapters/viem) and [`@hashspan/x402`](https://hashspan.dev/docs/adapters/x402). hashspan is open source on [GitHub](https://github.com/selimaytac/hashspan). ## Want the complete, CI-tested version? [examples/x402-agent](https://github.com/selimaytac/hashspan/tree/main/examples/x402-agent) is this tutorial as one agent. Its `research-agent` tips an author in ETH (`tip_author`, Part 1's send and confirm) and buys a report over x402 (`buy_report`, Part 2's payment and confirm), with the same in-process facilitator and Test USD. Its test checks that trace in CI: a `send 31337` and a linked `confirm 31337` under one tool, and a settled, verified `payment 31337` with a `confirm 31337` linked to it under the other. The example needs Node.js 24 (`nvm use` reads it from `.nvmrc`; install [nvm](https://github.com/nvm-sh/nvm) first) and Docker for `make lab-up`. Clone [selimaytac/hashspan](https://github.com/selimaytac/hashspan) and, from the repository root, after `nvm use && corepack enable pnpm && pnpm install`: ```sh make lab-up make anvil pnpm build pnpm --filter @hashspan/example-x402-agent start ``` `make lab-up` starts Jaeger on [localhost:16686](http://localhost:16686), and `make anvil` runs the chain, so give it its own terminal. Then open the service `research-agent` in Jaeger. ## FAQ ### Do I need ETH to pay with x402? Not on the paying side. With the exact scheme and an EIP-3009 token such as USDC, the agent only signs an authorization. The facilitator sends the transaction and pays its gas. In this tutorial the paying wallet holds 0 ETH from start to finish. ### Can I try x402 without a testnet or a faucet? Yes. This tutorial runs the whole flow on Anvil, a local chain: a test token, a paid API, a facilitator in the API's process and the paying agent. Nothing leaves your machine and no account is needed. ### How do I stop an agent from overspending? Cap it in two places. In your tool, check the amount before you pay. In the x402 client, setSpendControls refuses a pricier payment before anything is signed: a USD cap for default assets such as USDC ($1 by default in @x402/core 2.28), or a per-asset cap in units for a token you add. Listing a token adds it to the defaults; to pay in one asset only, add a policy with registerPolicy that drops the other offers. ### Where is the USDC contract on Base Sepolia? 0x036CbD53842c5426634e7929541eC2318f3dCF7e. It has 6 decimals, so 1000 units is $0.001, and its EIP-712 name and version are USDC and 2. ### Can I use LangChain or the OpenAI Agents SDK instead of the AI SDK? Yes. The payment spans nest under whatever span is active when the tool pays. The AI SDK opens an execute_tool span for you; with other frameworks, see integrations for the span to open yourself. ## Next - Add hashspan to an agent you already have: [Quickstart](https://hashspan.dev/docs/quickstart). - Nonce handling, background confirmation and naming the agent: [AI SDK recipe](https://hashspan.dev/docs/recipes/ai-sdk). - Let a coding agent do the setup: [Build with AI](https://hashspan.dev/docs/build-with-ai). - What each viem action records: [viem OpenTelemetry instrumentation](https://hashspan.dev/docs/viem-opentelemetry).