I have Grafana with Loki and Tempo setup and I am trying to add correlation that will show traces related to given log line (log containing trace ID). Important note: trace ID is sent to loki as separate field (structured metadata), not inside log mesage, hence I use correlations (I couldn't find way to do it with derived fields cause of this).
I was able to create desired correlation in the Grafana UI, and that worked as expected. But when replicating it with config it seems as if it didn't search after clicking correlation link (although correct trace id is sent to tempo window).
This is how my loki datasource correlation looks like:
correlations:
- targetUID: tempo
label: "Trace"
description: "Show traces related to this log in Tempo"
config:
type: "query"
target:
queryType: "traceId"
query: $${trace_id}
field: Line
transformations:
- type: regex
field: labels
expression: '"trace_id":"(\w+)"'
mapValue: trace_id
It "almost" works - links show up when viewing logs, I click on one, Tempo window opens, but I don't see any traces. Only after I click one of "Search" or "TraceQL" buttons, then traces show up (trace ID is already in TraceQL window without me putting it there, so seems as if issue was only search not starting, or not selecting some search option).
Tempo window after clicking correlation link in log
... and after clicking "TraceQL" button
I got so far using docs, but I have no idea now what is wrong or what I am missing there. Thanks in advance.