# Agent fees and confirmation times in Grafana

> Chart what your AI agent's transactions cost and how long they take to confirm: hashspan's histograms over OTLP to Prometheus, with a ready Grafana dashboard.

Source: https://hashspan.dev/docs/recipes/fees-in-grafana

How much did the agent spend on gas today, and how long did its transactions take to land? hashspan answers both with three histograms, recorded next to its spans: send duration, confirmation duration and fee. This page puts them in Grafana, first in the local lab, then from your own agent.

## What gets recorded

- **Send duration:** `blockchain.client.send.duration`, in Prometheus `blockchain_client_send_duration_seconds`. Labels: the chain, and `error.type` when the send failed.
- **Confirmation duration:** `blockchain.client.confirmation.duration`, in Prometheus `blockchain_client_confirmation_duration_seconds`. Labels: the chain and the outcome.
- **Fee:** `blockchain.client.fee`, in Prometheus `blockchain_client_fee`. Labels: the chain, `blockchain.tx.status` and, since 0.11, `blockchain.fee.payer` on samples whose fee the sender did not pay (`facilitator` for a payment's settlement, `paymaster` for a user operation).

The labels are low-cardinality on purpose: no address, hash or agent identity. Prometheus turns `blockchain.chain.id` into `blockchain_chain_id` and `service.name` into the `job` label. Every field is described on the [metrics reference](https://hashspan.dev/docs/reference/attributes/metrics).

## Try it in the local lab

1. Clone the repository and install it, as in the [local lab](https://hashspan.dev/docs/local-lab).

2. Start Jaeger, Prometheus and Grafana with the dashboard:

   ```sh
   make lab-metrics
   ```

3. Run the example agent a few times. When Prometheus is up, `make demo` sends the metrics to it too.

   ```sh
   make demo
   ```

4. Open http://localhost:3000. The hashspan dashboard is the home page: sends, confirmation p95, median fee, latency and fee per chain, and confirmation outcomes.

## From your own agent

hashspan records the histograms through the global meter provider, so the only setup is an OpenTelemetry metrics SDK with an OTLP exporter:

```sh
npm install @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-proto @opentelemetry/resources
```

Import this file first in your agent's entry point, before anything sends:

```ts title="metrics.ts"

// The exporter reads OTEL_EXPORTER_OTLP_METRICS_ENDPOINT.
export const meterProvider = new MeterProvider({
  resource: resourceFromAttributes({ 'service.name': 'my-agent' }),
  readers: [new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() })],
});
metrics.setGlobalMeterProvider(meterProvider);
```

Point it at Prometheus's OTLP receiver, which needs `--web.enable-otlp-receiver`:

```sh wrap
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:9090/api/v1/otlp/v1/metrics node agent.js
```

A short-lived script must call `await meterProvider.shutdown()` before it exits, or its last samples are lost. If it exports only once, Prometheus also needs `--enable-feature=created-timestamp-zero-ingestion` for `increase()` and `rate()` to see it, as the lab sets. Then import the [dashboard file](https://github.com/selimaytac/hashspan/blob/main/docker/grafana/dashboards/hashspan.json) into Grafana, as the [backends page](https://hashspan.dev/docs/backends#grafana-dashboard-for-the-metrics) explains.

## Queries to start from

What the agent spent per chain over the last day, in wei. Samples with a `blockchain_fee_payer` label are fees someone else paid, so the query leaves them out:

```text wrap
sum by (blockchain_chain_id) (increase(blockchain_client_fee_sum{blockchain_fee_payer=""}[1d]))
```

How many transactions reverted:

```text wrap
sum by (blockchain_chain_id) (increase(blockchain_client_fee_count{blockchain_tx_status="reverted"}[1d]))
```

Confirmation time at the 95th percentile:

```text wrap
histogram_quantile(0.95, sum by (le, blockchain_chain_id) (rate(blockchain_client_confirmation_duration_seconds_bucket[1h])))
```

These are estimates, not counts. `increase()` extrapolates to the edges of the range, and a short-lived process starts its counters again on every run: in a local test, three reverted transactions from three runs of the demo showed as 3.3 over 15 minutes and 7.6 over a day. For an exact count, count the confirm spans in your trace backend.

## What it doesn't do

The fee histogram has no agent label, so per-agent spend comes from the spans: each confirm span carries `blockchain.tx.fee` and `gen_ai.agent.name`. Alerting is your backend's job; hashspan only records.