31 questions
1
vote
0
answers
83
views
Istio tracing issue
Does Istio’s VirtualService routing and header manipulation apply to tracing traffic generated by istio-proxy itself?
Here’s the situation: I have Istio configured to send spans to an external tracing ...
0
votes
0
answers
45
views
Is it possible to use the trace status as a field in visualisations based on a Grafana Tempo data source?
I have traces going into Grafana Tempo. The traces are not collected by one of the standard collectors; I'm using the OpenTelemetry SDK to generate custom spans, but I don't think that matters.
I'm ...
1
vote
0
answers
260
views
Grafana Tempo not receiving OpenTelemetry traces from .NET services - only /health and /metrics traces visible
I have a multi-service .NET application running in Docker Compose with Grafana Tempo for distributed tracing. While Tempo is running and accessible, I'm only seeing traces for /health and /metrics ...
0
votes
0
answers
48
views
How to Accurately Calculate Per-Service Trace Durations and P95 Using PromQL or TraceQL
I'm using Tempo's metrics generator to extract spanmetrics and calculate the duration of each trace.
However, when I use the following PromQL expression, the results differ significantly from the ...
0
votes
0
answers
199
views
Unable to connect to Tempo search: "Please ensure that Tempo is configured with search enabled"
I have installed Tempo on Linux and it is running using docker. The version of Tempo is 2.6.1.
My config is:
---
server:
http_listen_port: 3200
distributor:
receivers:
jaeger:
protocols:
...
0
votes
0
answers
131
views
Grafana tempo / faro - joining traces
I'm relatively new to tempo but I have setup a faro -> alloy -> tempo flow which joins traces from my otel traces from my backend process, this works well, but what I would like to see a trace ...
0
votes
0
answers
112
views
Ingesting manual spans to Grafana Tempo via OpenTelemetry in Python
I am running Grafana Tempo with the example docker-compose local configuration from https://github.com/grafana/tempo/tree/main/example/docker-compose/local. To keep the traces clear, I disabled the ...
0
votes
1
answer
1k
views
Cannot see any traces from Alloy in Grafana [closed]
I am trying to use Grafana Alloy with Grafana Beyla enabled and hope it can send some traces to Grafana Tempo.
With this setup, Alloy succeed sending logs to Loki. However, for traces, I cannot see ...
1
vote
0
answers
288
views
k6 failed to upload metrics to otel and mimir remote write
I am unable to make k6 use OpenTelemetry Collector, and save metrics to Mimir and traces to Tempo, do you know how?
Following is what I have tried, but I keep getting error:
INFO[0030] 2025/01/15 22:...
0
votes
0
answers
99
views
Grafana add Tempo trace Id link inside chart tooltip
I successfully configured the Link to trace id from logs:
In my dashboard:
In the first panel we see an expanded log entry with the link to Tempo trace.
The second panel is a Bar chart that show ...
0
votes
0
answers
160
views
Merge two traces in grafana tempo
I’m currently working on a project using OpenTelemetry to instrument FastAPI services, and we’re using Grafana Tempo as the tracing backend. I have a scenario where I need to combine two separate ...
2
votes
0
answers
2k
views
Grafana Loki to Tempo correlation with traceId
I have Grafana with Loki and Tempo setup and I am trying to add correlation that will show traces related to given log line (log containing trace ID).
Important note: trace ID is sent to loki as ...
0
votes
1
answer
319
views
memcached error "Maximum connections must be greater than 0" in tempo-distributed
I am using Tempo Distributed Helm Chart to version : 1.9.0
And enabled memcached, but I am getting below error in memcached logs and memcache pods are not getting up.
Maximum connections must be ...
2
votes
0
answers
605
views
Grafana Tempo search not always show result
I’m having a problem using tempo and grafana. Looks like the search sometimes works and sometimes it doesn’t. For example, I’m trying to search for logs for a specific time - from 13:00 to 13:05 - and ...
-1
votes
1
answer
807
views
Running OTEL agent as windows service is not working
Want to run Otel collecor agent as Windows services. So downloaded latest windows release package and run the following command to create service and start the service.
> sc.exe create otelcol ...
0
votes
1
answer
899
views
Not able to get traces in tempo from Grafana loki and OTEL collector
Its been more than 2 weeks since i am stuck in this issue. I have Fluent bit to send the logs like this ( Fluent Bit -> Loki and Otel Collector -> Tempo). I have done all the changes but not ...
0
votes
1
answer
105
views
Tempo Search Autocomplete Nonsense Values
I am succesfully searching for traces on Tempo using Grafana. All my trace resources have at least these three labels service, environment, instance. In the dropdowns for these labels all the correct ...
0
votes
1
answer
225
views
Do we need to stitch spans before pushing to clickhouse database for grafana tempo UI?
I have a opentelmetry collector and full setup till the Clickhouse database. As of now I am putting spans to the clickhouse database following this blog [https://clickhouse.com/blog/storing-traces-and-...
0
votes
1
answer
841
views
Can i change distributor's base url in Grafana Tempo with Spring Boot?
I have an application that consists of several microservices that I run in Docker. I'm currently learning observation and have already managed to connect logs with Loki, metrics with Prometheus and ...
1
vote
0
answers
339
views
OTLP Collector to Graphana Cloud Tempo - 401
I have a OTLP instance running that collects traces successfully over grcp. But when I try to uplad the traces to my grafana cloud tempo instance I get this error:
grafana-collector-1 | 2023-12-...
0
votes
1
answer
2k
views
Grafana Tempo doesn't show traces after sometime
hi i have setup grafana tempo with opentelemetry in my EKS cluster. at initial time tempo show traces but after 1 hr none trace show on grafana it show me 404 Not Found
here is my tempo configuration
...
0
votes
1
answer
847
views
Grafana tempo: What is multitenancy?
I am going through grafana tempo documentation, and I came across multi-tenancy.
Also, there is a github example for it.
But I dont really understand it as a whole. What are we trying to achieve here? ...
2
votes
1
answer
2k
views
Send traces directly from python application to Grafana Cloud's tempo instance without Grafana agent
Here is python script that sends test trace directly to tempo instance :
import base64
from typing import Tuple
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http....
0
votes
1
answer
1k
views
Why do my grafana tempo ingester pods go into Backoff restarting state after max_block_duration?
I am using grafana-tempo distributed helm chart. It is successfully deployed and its backend is configured on Azure Storage (blob containers) and working fine.
I have a demo application which is ...
0
votes
1
answer
584
views
How to check health/query a Tempo datasource using the Grafana API - v9.1.6
I want to test the connectivity between Grafana(v9.1.6) and a Tempo datasource using the API. In the documentation, it appears that I can use the GET path /api/datasources/:datasourceId/health
(https:/...
0
votes
1
answer
820
views
Unable to retrieve my OpenTelemetry logs in Grafana Tempo
By following the .NET Getting started of OpenTelemetry documention I'm able to catch every traces in my Grafana Tempo docker. I can see my HTTP calls, my crashes and my database queries in my Grafana ...
0
votes
1
answer
422
views
grafana tempo sigv4 support for prometheus backend?
Can grafana tempo backend sign (sigv4) it's request that it sends to aws prometheus (AMP)?
metrics_generator:
registry:
external_labels:
source: tempo
cluster: example
storage:
...
1
vote
1
answer
1k
views
How to send authenticated traces to tempo cloud using OLTP exporter python?
Im trying to use the OTLPSpanExporter (either the http or grpc) version to send traces to tempo cloud but i wasn't able to find any documentation on how to add the authentication.
I tried something ...
0
votes
1
answer
2k
views
Sending Traces to Grafana Tempo
I've been trying to get into this for about 2 days. I have a .Net6.0 web application that needs to send tracing to a Grafana Tempo instance. Already configured is Jaeger for tracing, which is to be ...
-1
votes
1
answer
1k
views
grafana/loki-stack how can i add grafana tempo
I'm trying to customize the trace display using Grafana Tempo. I use helm grafana/loki-stack https://grafana.com/docs/loki/latest/installation/helm/. Installing command helm upgrade --install loki ...
5
votes
1
answer
8k
views
unable to hook up opentelemetry collector with grafana tempo
I am unable to get otel collector to work, I'm new to opentelemetry, so it feels I'm making a stupid error in congiration somewhere
this is my python sample script that is supposed to create a sample ...