[processor/metricstarttime, receiver/prometheus] stabilize start time and reset adjustment

[eriksywu]

Component(s)

processor/metricstarttime

Describe the issue you're reporting

Recent movement on the promtsdb and promql side of things (arguably orthogonal to this issue) prompted me to think about what changes are worth bringing into processor/metristarttime before we promote it to Beta (as discussed here)

Problem Statement:
There's a lack of adoption of CT/ST instrumentation in the prom->otel chain - expected as the CT feature is experimental. Most scraped prometheus targets:

1. do not support an exposition format (prom-proto or om-text or om2.0 in the future) that exposes a CT field and/or
2. or lack of proper instrumentation to set CT

My understanding is the prom-receiver’s built-in adjustment functionality is an attempt to provide a first-order approximation of CTs. I’ve raised an issue to disable the receiver’s default adjustment.

The receiver’s adjustment functionality has been made and expanded into its own processor which exposes a lot more configuration ergonomics to the user. The ideal scenario (pending prom ct adoption) for prom-scraped metrics is for the operator to explicitly configure their collectors on how to adjust STs for metrics. I.e some metrics known to be initialized at process start and some may be lazily initialized. The start_time_metric strategy would be a good fit for the former. The subtract_initial_point strategy would work for the latter with the caveat that it is stateful to the collector which incurs a similar issue as this.

What do we think about adding some additional knobs to the processor?

• a new start_time_attribute strategy that works in a similar fashion to start_time_metric but will use the resource’s pod start time or last transition time to the ContainersReady condition
• a skip_if_ct_exists config oaramter: if true, the processor shall not do adjustments if the metric already has ST set - allowing fallback strategies to be chained
• a include_metrics and exclude_metrics config parameter: filters on the metric name to apply the processor on

All proposed changes are captured in this PR: #43694

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• processor/metricstarttime: @dashpole @ridwanmsharif

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

skip_if_ct_exists shouldn't be needed. We only adjust points if the CT is not already set. If we don't, that is a bug.

I like include/exclude metrics. We should look at other processors to see what they do. E.g.

opentelemetry-collector-contrib/internal/filter/filterconfig/config.go

Line 22 in 00b2b8f

type MatchConfig struct {

looks like a good starting point.

For start_time_attribute, would this still be stateful? If it isn't, we could just document how use the transform processor do set the start time based on a resource or metric attribute in the README.

Are pod conditions actually added as resource attributes? I assume we don't want this processor watching pods.

[dashpole]

I see... Looks like we don't put the pod start time in resource attributes (which makes sense, since it can change). It does seem like a nice feature to have, though, so lets work through what it needs to be in config.

I think we should make sure that we properly set start times only if they are not already set prior to going beta.

In terms of adding include/exclude config, and the new adjuster option, I think those can be added in a backwards-compatible manner, and shouldn't block beta. We had previously said that we wanted this component to be beta prior to removing the adjustment in the prometheus receiver, so I would like to not block unless we have a good reason.

[eriksywu]

For start_time_attribute, would this still be stateful? If it isn't, we could just document how use the transform processor do set the start time based on a resource or metric attribute in the README.

Not stateful to the collector - similar to the start_time_metric strategy. Yea, currently I'm using the k8s_attr_processor to set STs. My poc implementation of the attributes strategy will create a single k8s pod informer as you mentioned, since pod ST isn't a resource attribute.

My main reasoning for having a start_time_attribute in the metric ST processor rather than using the k8s_attrs processor:

• more light weight and will have share config ergonomics (i.e a potential include/exclude filter) as the other strategies
• adds an additional behaviour to use the last transition time to the containers ready condition as the ST rather than pod starttime, which isn't available for mapping in the k8s_attrs processor. This isn't a perfect solution but it would be a better approximation of STs for workloads that are expected to experience pod churns (i.e metric producing container is expected to restart multiple times within the pod life cycle). The operator will have to know about which metrics would fall in these use cases - hence the include/exclude filter.

We had previously said that we wanted this component to be beta prior to removing the adjustment in the prometheus receiver, so I would like to not block unless we have a good reason.

Totally. I think we should proceed with moving both the ST processor and adjustment removal in prom receiver to beta, unencumbered by this issue.

As an aside, stepping back even further: the impetus of the start_time_attribute strategy is to add another surface area for ST approximations for prom-scraped metrics because of lack of stability/support of exposition formats that natively expose CT. Prom-proto has very limited language support and openmetrics-text has an inefficient parsing mechanism for CTs that is disabled by default. I was digging around the latter issue and came across some findings.

I can get out a PR for an include/exclude filter (already implemented in poc).

[dashpole]

The containers ready condition is for all containers, right? We ideally want to choose the earliest possible time a metric could have been incremented to ensure we don't get rates that are too high. Schedule time might be better...