What is the difference between the buffer_ and component_ metrics/telemetry?

[alex-rowe]

We are looking to monitor our sinks for any potential errors but I can't work out the exact difference between the buffer_ and component_ telemetry metrics.

Do the buffer metrics roll up into the component metrics?

For example, if we have a buffer configured and set to drop_newest, what metric does that show under? does it show under buffer_discarded_events_total or component_discarded_events_total or both? For this example, we'd want to alert on any events being dropped by the sink.

Similarly, if we wanted to monitor a kind of throughput, would we use buffer_sent_events_total or component_sent_events_total? For this example, we'd want to alert on something like "is sent volume over X less than the average volume"

[jszwedko]

Hi @alex-rowe !

The buffer_* metrics only apply to the buffers and are not rolled up into the component_* metrics. In your case, if the buffer is the one dropping events, it will appear in buffer_discarded_events_total.

[alex-rowe]

The component we were looking at was aws_kinesis_streams. I was probably looking in the wrong place, but I couldn't see where the intentional metric label was configured for this one.

[jszwedko]

It should have an intentional label on component_discarded_events_total.

[alex-rowe]

Is this where they're defined src/internal_events/aws_kinesis.rs#L32?

I couldn't see anything on the docs for the sink aws_kinesis_streams/#component_discarded_events_total

[jszwedko]

Is this where they're defined src/internal_events/aws_kinesis.rs#L32?

That's one of them anyway. There are other shared ones as well in src/internal_events.

I couldn't see anything on the docs for the sink aws_kinesis_streams/#component_discarded_events_total

Doh, yes, it looks like that is missing from the docs. I'll follow up on that.

[jszwedko]

PR to update the docs: #18622