From ec67f7e46a71128d3edd96044a861de0369d7e50 Mon Sep 17 00:00:00 2001 From: Marc Lopez Rubio Date: Wed, 2 Jul 2025 23:51:48 +0200 Subject: [PATCH] Add APM Server known issue for TBS (#4862) * Add APM Server known issue for TBS Signed-off-by: Marc Lopez Rubio * Apply suggestions from code review Co-authored-by: Colleen McGinnis --------- Signed-off-by: Marc Lopez Rubio Co-authored-by: Brandon Morelli Co-authored-by: Colleen McGinnis (cherry picked from commit ed75d36fa5094b21c291664110521aa46f8eb64c) # Conflicts: # docs/en/observability/apm/known-issues.asciidoc --- .../observability/apm/known-issues.asciidoc | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 8dd9345514..477873a3eb 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -22,6 +22,58 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ //// [discrete] +<<<<<<< HEAD +======= +== Tail Sampling may not compact / expired TTLs as quickly as desired, causing increased storage usage. + +_Elastic Stack versions: All 8.x versions_ + +There are some issues with the tail sampling implementation in all 8.x versions that may prevent buffered traces from being compacted or expired as quickly as expected. This can lead to increased storage usage for longer than the default 30m TTL. + +This may manifest in two ways, increased value log (vlog) file size and increased SST (LSM) file size. LSM growth and late compaction is particularly troublesome given how the underlying K/V database performs compactions on its layers. There is noticeable LSM growth for use-cases where traces are under 1KB in size, since they are written to the LSM layer directly. + +This issue is fixed in 9.0.0, due to a re-implementation of how the underlying tail sampling databases are used. The new implementation uses a more efficient partitioning scheme, allowing more efficient expiration of traces. + +[discrete] +== APM Server v8.6.x and prior with Elasticsearch v8.15.x and later has broken APM UI + +_Elastic Stack versions: 8.15.0+_ + +// The conditions in which this issue occurs +The issue occurs when using APM Server versions <= 8.6.x with {stack} versions 8.15.x and later. + +// Describe why it happens +In APM Server versions prior to 8.7.0, the aggregated metrics were indexed in APM's internal data stream. The aggregated metrics require some additional mappings to properly index the data and allow the APM UI to be able to use the data to populate service inventory page properly. From APM Server version 8.7.0, aggregated metrics were moved to their dedicated datastreams (https://github.com/elastic/apm-server/issues/9703[ref]), however, the internal mappings were kept to support backward compatibility. In APM Server version 8.15.0 we migrated to the https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/apm-data[apm-data plugin] and the internal mappings required to make aggregated metrics work were dropped. + +// How to fix it +If the deployment is running 8.15.0 and APM Server versions are still at 8.6.x or older then the recommendation would be to upgrade the APM Server to the same version as the stack version. If upgrade is not possible then the custom component template should be updated to include custom mappings for making internal datastreams support aggregated metrics: + +[source,txt] +---- +PUT _component_template/metrics-apm.internal@custom +{ + "template": { + "mappings": { + "dynamic_templates": [], + "properties": { + "transaction.duration.histogram": { + "type": "histogram" + } + } + } + }, + "_meta": { + "package": { + "name": "apm" + }, + "managed_by": "fleet", + "managed": true + } +} +---- + +[discrete] +>>>>>>> ed75d36f (Add APM Server known issue for TBS (#4862)) == `prefer_ilm` required in component templates to create custom lifecycle policies _Elastic Stack versions: 8.15.1+_