Skip to content
Star5

Agent fees and confirmation times in Grafana

How much did the agent spend on gas today, and how long did its transactions take to land? hashspan answers both with three histograms, recorded next to its spans: send duration, confirmation duration and fee. This page puts them in Grafana, first in the local lab, then from your own agent.

What gets recorded

Section titled “What gets recorded”
  • Send duration: blockchain.client.send.duration, in Prometheus blockchain_client_send_duration_seconds. Labels: the chain, and error.type when the send failed.
  • Confirmation duration: blockchain.client.confirmation.duration, in Prometheus blockchain_client_confirmation_duration_seconds. Labels: the chain and the outcome.
  • Fee: blockchain.client.fee, in Prometheus blockchain_client_fee. Labels: the chain, blockchain.tx.status and, since 0.11, blockchain.fee.payer on samples whose fee the sender did not pay (facilitator for a payment’s settlement, paymaster for a user operation).

The labels are low-cardinality on purpose: no address, hash or agent identity. Prometheus turns blockchain.chain.id into blockchain_chain_id and service.name into the job label. Every field is described on the metrics reference.

Try it in the local lab

Section titled “Try it in the local lab”

  1. Clone the repository and install it, as in the local lab.

  2. Start Jaeger, Prometheus and Grafana with the dashboard:

    Terminal window
    make lab-metrics

  3. Run the example agent a few times. When Prometheus is up, make demo sends the metrics to it too.

    Terminal window
    make demo

  4. Open http://localhost:3000. The hashspan dashboard is the home page: sends, confirmation p95, median fee, latency and fee per chain, and confirmation outcomes.

From your own agent

Section titled “From your own agent”

hashspan records the histograms through the global meter provider, so the only setup is an OpenTelemetry metrics SDK with an OTLP exporter:

Terminal window
npm install @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-proto @opentelemetry/resources

Import this file first in your agent’s entry point, before anything sends:

metrics.ts
import { metrics } from '@opentelemetry/api';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';


// The exporter reads OTEL_EXPORTER_OTLP_METRICS_ENDPOINT.
export const meterProvider = new MeterProvider({
  resource: resourceFromAttributes({ 'service.name': 'my-agent' }),
  readers: [new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() })],
});
metrics.setGlobalMeterProvider(meterProvider);

Point it at Prometheus’s OTLP receiver, which needs --web.enable-otlp-receiver:

Terminal window
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:9090/api/v1/otlp/v1/metrics node agent.js

A short-lived script must call await meterProvider.shutdown() before it exits, or its last samples are lost. If it exports only once, Prometheus also needs --enable-feature=created-timestamp-zero-ingestion for increase() and rate() to see it, as the lab sets. Then import the dashboard file into Grafana, as the backends page explains.

Queries to start from

Section titled “Queries to start from”

What the agent spent per chain over the last day, in wei. Samples with a blockchain_fee_payer label are fees someone else paid, so the query leaves them out:

sum by (blockchain_chain_id) (increase(blockchain_client_fee_sum{blockchain_fee_payer=""}[1d]))

How many transactions reverted:

sum by (blockchain_chain_id) (increase(blockchain_client_fee_count{blockchain_tx_status="reverted"}[1d]))

Confirmation time at the 95th percentile:

histogram_quantile(0.95, sum by (le, blockchain_chain_id) (rate(blockchain_client_confirmation_duration_seconds_bucket[1h])))

These are estimates, not counts. increase() extrapolates to the edges of the range, and a short-lived process starts its counters again on every run: in a local test, three reverted transactions from three runs of the demo showed as 3.3 over 15 minutes and 7.6 over a day. For an exact count, count the confirm spans in your trace backend.

What it doesn’t do

Section titled “What it doesn’t do”

The fee histogram has no agent label, so per-agent spend comes from the spans: each confirm span carries blockchain.tx.fee and gen_ai.agent.name. Alerting is your backend’s job; hashspan only records.