Add Jaeger proposal #42
Conversation
|
Thanks @yurishkuro, requesting that the @cncf/toc and wider CNCF community look at the proposal before we formally call for a vote at the end of this month |
|
@yurishkuro et. al, nice to see this work being proposed to the CNCF. I have a few question and comments:
|
|
great questions, @bassam
Backend storage is the primary scalability limit. Of course if you replace it with /dev/null, there will be some overhead of collectors themselves processing the messages, but collectors are stateless and horizontally scalable.
In the current version all indices are written at the same time as the main span record.
Collectors are stateless. In order to do aggregations there needs to be a stateful processing layer, which in our case is Kafka + Flink that sit after the collectors.
For Cassandra we simply rely on the TTL it supports natively. For Elasticsearch we have a script that purges old indices and needs to run on a schedule (e.g. once a day).
Yes, that's precisely what it is designed for and how we run things internally. Under Kubernetes agents run as a daemon set.
They are not. Our decision was that there are many of other ways to secure access (e.g. this blog post), it did not need to be built into the product.
It has been suggested before, but it was not a priority for us to replace Cassandra (we've been using it for 2yrs now). It would be interesting if Prometheus could support the indexing/search use case for traces.
It's on the roadmap for agent->collector. The client libs in the app are primarily built to use UDP to avoid extra dependencies (even though - thrift, I know).
It's actually brand new, added as a migration path. To clarify, support is only to accept spans in Thrift or JSON Zipkin format, then they are immediately converted to Jaeger format that is modelled after OpenTracing spec. |
I'm trying to understand which tables/indices are required for collection, vs. the ones needed to support the UI/cli (and can lag behind, or built as needed). Can collection just write to the main span record?
sorry I'm not sure I understood. consider the following case, a given trace (id7) has 3 spans (a,b,c) being sent concurrently by one agent to one or more collectors. Independent collectors might receive a,b,c out of order and might write to the backend store at different times. The schema seems to have some implicit ordering (for example, I assume the parent must be written before the child for parent_id to be correct). Who ensures this ordering when the traces and spans are being spread to stateless collectors? |
Spans can be written to storage in any order. The full trace is assembled at query time by retrieving all spans with a given trace ID and arranging them in-memory according to parent-child references. |
|
@yurishkuro thanks! |
|
We are using Jaeger at stagemonitor and we support Jaeger becoming a CNCF member project |
|
We use Jaeger at Northwestern Mutual along with Opentracing to trace 100s or services across 1000s of developers. Jaeger has been an essential tool for us and I fully support the project joining the CNF :) |
|
We are using Jeager at microcloud and support the project joining CNF. |
|
request from @monadic, can you update the project proposal with a new section that discusses how Jaeger fits in with Zipkin, thanks! |
|
👍 from GrafanaLabs. happily using jaeger in production since a few weeks ago. seems like the best open source distributed tracing system out there. |
|
At Base CRM we're also using Jaeger and would be quite happy if it joined CNCF. |
|
At FarmersEdge we are just rolling out Jaeger alongside the other CNCF projects we use |
|
Currently, we are testing and using Jaeger with OpenShift, our Kubernetes-based Container Platform. We have it on our road map to use it as part of the monitoring tooling to be provided with OpenShift. We chose it because it fully supports OpenTracing and has been used in production at high scale/performance within Uber for two years. We look forward to the continued evolution of distributed tracing and will continue to support and collaborate with the CNCF community on Jaeger. |
|
We are slowly adopting Jaeger at Symantec (Elastica) for couple of services to start and it will be great if Jaeger becomes CNCF member project. |
| CNCF efforts like the OpenTracing specification came to existence to help unify existing tracing implementations | ||
| out there. | ||
|
|
||
| Jaeger is a battle tested distributing system that takes advantage of OpenTracing and advances the state |
There was a problem hiding this comment.
Can you provide more detail on what you mean by battle tested? At what level of scale has this tracing been tested and what are the performance implications? In other open source cloud infrastructure communities a monitoring project was accepted for inclusion before it could prove it could scale and before it could prove that it could work for both private and public clouds. This was a huge mistake as that monitoring project's architecture prohibited it from working at very large scale and the project was accepted accepted yet never able to meet the requirements of the overall community. For a project like this I would really recommend that you keep the bar very high and make sure the community needs are well satisfied before a project like this goes beyond incubator status.
There was a problem hiding this comment.
Regarding "battle tested", Jaeger has been running in production at Uber for two years, as of today tracing more than 1200 microservices. We're using fairly aggressive sampling rates (typically between 1/1000 and 1/10000 for top tier services), which make performance overhead negligible. Based on anecdotal evidence from other commercial vendors, tracing 100% of traffic may incur up to 5% of performance overhead, which can be tuned by less extensive instrumentation and further perf optimization in the tracing libraries.
Regarding the scalability of Jaeger architecture, all layers in Jaeger backend are horizontally scalable. Typically the bottleneck is in the storage backends (Cassandra, ES), which are also horizontally scalable. There are may avenues for improving the storage backend throughput and indexing capabilities, as well as reducing the amount of data needed to be stored in the first place, by investing in techniques likes on-demand sampling, post-trace sampling, trace discovery through aggregates rather than heavy indexing.
There was a problem hiding this comment.
1200 services does that mean 1200 different software projects, or 1200 running daemons (possibly multiple copies of the same code). though does it matter? can you shed some light on the volume of spans and traces being generated, or the volume of log lines, tags, etc, whatever it is that affects workload.
anecdotal evidence from other commercial vendors, tracing 100% of traffic may incur up to 5% of performance overhead
can you clarify whether this means commercial vendors selling jaeger-based solutions, or alternatives to jaeger? if so, do we have stats on the overhead of jaeger?
There was a problem hiding this comment.
1200 different services / applications / products, not just different instances - those are probably in the 500,000 ballpark. Our tracing backend receives ~50K spans per second, median span size is ~500b, max ~60Kb.
can you clarify whether this means commercial vendors selling jaeger-based solutions, or alternatives to jaeger? if so, do we have stats on the overhead of jaeger?
I meant commercial alternatives to Jaeger, like Lightstep, but my point was that the actual instrumentation libraries are very similar in functionality / architecture, so I am fairly comfortable extrapolating their perf overhead numbers. We don't have stats on perf overhead of Jaeger libs at 100% sampling - it's actually a non-linear problem with many contributing factors, so raw numbers like 5% are not very meaningful without describing the exact methodology and conditions of measurements. I guess what I'm getting at is that it's always possible to limit perf overhead by applying sampling, and even with low sampling rates like ours we get extremely rich data about application behavior that the overhead is totally worth it.
|
The project is now out for official @cncf/toc vote: https://lists.cncf.io/pipermail/cncf-toc/2017-September/001149.html Thanks everyone for participating in the community vetting and due diligence process. |
|
Good news, the @cncf/toc has accepted Jaeger as our 12th CNCF project! :) https://lists.cncf.io/pipermail/cncf-toc/2017-September/001204.html Welcome! |
No description provided.