Querying spans


Uptrace provides a powerful querying language that supports filters (where span.status.code = "error), grouping (group by span.group_id), and aggregates (p50(span.duration)).


To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributes and events. For logs, you can use structured logging.


Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where span.name contains 'hello' or span.count > 100. Uptrace automatically translates filters into SQL WHERE or HAVING, so you don't have to worry about that.

Uptrace supports the following span attribute types:

Attribute typeSupported comparison operators
string=, like, contains, exists
int64 and float64=, <, <=, >, >=, exists
array of the types abovecontains, exists
Uptrace filterDescription
span.status_code = "error"Filter spans with error status code. Case-sensitive.
span.name like "hello%"Filter span names that start with "hello". Case-insensitive.
span.name like "%hello"Filter span names that end with "hello". Case-insensitive.
span.name contains "hello"Filter span names that contain "hello". Case-insensitive.
span.name contains "foo|bar"Same as span.name contains "foo" OR span.name contains "bar".
span.duration > 1msSame as span.duration > 1000. Uptrace supports μs, ms, and s units.
http.request_content_length > 1kbSame as http.request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units.
span.event_count > 0Filter spans with events.
span.event_error_count > 0Filter spans with error events.
span.event_log_count > 0Filter spans with log events.
where span.is_eventFilter event spans, for example, exceptions or logs.
where foo existsFilter spans that have attribute foo.


Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host.name groups spans by the attribute host.name and at the same time selects the host.name.

Uptrace groupingNote
group by span.group_idGroup similar spans together.
group by span.start_of_minuteGroup spans by the minute they were created. Uptrace also supports grouping by hour, day, and week.
group by host.nameGroup spans by the host.name attribute.
group by service.name, service.versionGroup spans by the combination of service.name and service.version attributes.


Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.

Aggregate functionExampleNote
anyany(span.name)Any (random) span name.
avgavg(span.duration)Average span duration.
min, maxmax(span.duration)Maximum span duration.
p50, p75, p90, p99p50(span.duration)Span duration percentile.
sumsum(http.request_content_length)Total number of processed bytes.
top3, top10top3(code.function)Top 3 most popular function names.
uniquniq(http.client_ip)Number of unique IP addresses.

There is also a number of common pre-aggregated columns:

Virtual columnNote
span.countThe equivalent of SQL count(*) that takes in account adjusted countsopen in new window.
span.error_countThe number of spans with span.status.code = 'error'.
span.error_pctThe result of span.error_count / span.count.
span.count_per_minThe average number of spans per minute.

Combining all together

You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:

where http.user_agent.bot not exists | uniq(http.client_ip) | group by span.start_of_day


Nested queries

Uptrace also supports nested queries, for example, the following 2 queries:

# group spans by the group id while counting number of spans
group by span.group_id | span.count as num_span

# calculate average/min/max number of spans in the groups from the previous query
count() | avg(num_span) | min(num_span) | max(num_span)

are compiled to the following pseudo-SQL:

SELECT count(), avg(num_span), min(num_span), max(num_span)
    count() AS num_span
  GROUP BY span.group_id
Last Updated: