Run hashspan behind an OpenTelemetry Collector

Send everything to a Collector, then decide there what to keep. This page has a config we ran against real hashspan traces, and what it kept. The run wrote to a file exporter; the page shows otlphttp, so swap in your own backend.

The config

Tested with otel/opentelemetry-collector-contrib 0.163.1. Both processors used here come with the contrib build, not the core one.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Addresses are public and few: an unkeyed hash of one can be reversed by hashing candidates. Drop them here, or
  # use hashspan's own address modes in the SDK.
  transform/drop-addresses:
    error_mode: ignore
    trace_statements:
      - delete_matching_keys(span.attributes, "^blockchain\\.(tx\\.(from|to|fee_asset|authorization\\.addresses)|payment\\.(payer|recipient|asset)|call_batch\\.sender|user_operation\\.(sender|entry_point|paymaster))$")
  tail_sampling:
    decision_wait: 5s
    decision_cache:
      sampled_cache_size: 200000
    policies:
      - name: failed
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: moved-money
        type: ottl_condition
        ottl_condition:
          error_mode: ignore
          span:
            - span.attributes["blockchain.operation.name"] != nil
      - name: everything-else
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlphttp:
    endpoint: https://your-backend.example:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, transform/drop-addresses]
      exporters: [otlphttp]

In your app, keep head sampling at 100% and export everything to the Collector. Tail sampling needs every span of a trace to decide. Metrics are recorded for every transaction, whatever a sampler decides.

What did it keep?

We sent 200 agent runs through it: 20 that sent a transaction, 10 that failed without one, and 170 plain ones.

Runs	Sent	Kept
With a transaction	20	20, each with its send and confirm span
Failed, no transaction	10	10
Plain	170	13, close to the 10% sample

No address attribute was left in the output.

Why a policy on blockchain.*, not only on errors?

Because a failed transaction isn’t an error yet when the receipt takes longer than decision_wait.

hashspan’s confirm span ends when the receipt arrives, often after the rest of the trace. A transaction that reverts on chain only shows as an error on that span. The tool span that sent it ended fine.

We tested this with 12-second blocks and a 5-second decision_wait. Each run sent one transaction that reverted. The confirm span ended about 12 seconds after the send, 7 seconds after the decision:

Policies	Reverted payments kept, of 20
failed + 10% of the rest	3
failed + moved-money + 10% of the rest	20, whole, with the revert reason

The send span carries blockchain.operation.name from its start, so the moved-money policy keeps the trace before the confirm arrives. In a run of 10 successful transactions, all 10 confirm spans arrived after the decision and were kept with their traces.

Notes

• decision_wait is how long the Collector waits after a trace’s first span. Longer waits hold more traces in memory. With the moved-money policy, a short wait still keeps transaction traces.
• decision_cache keeps the decision for spans that arrive after the trace left the processor’s buffer (num_traces, 50,000 by default). The processor’s README asks for a cache much larger than num_traces.
• Addresses: the regex drops the address attributes. Addresses can also sit inside other values, such as a revert reason or error.type; only hashspan’s address mode off covers those (address modes). An unkeyed hash of a public address can be reversed by hashing candidates.
• Background confirmations need the app to run until the receipt arrives. A short-lived process should call flush() first, or its open confirm spans end as timeout.

Sampling explains head and tail sampling with a simulation.