Support `max_bytes` for memory buffers

[jszwedko]

Requested by user in discord.

They are trying to limit Vector memory usage by tweaking the in-memory buffer sizes, but since we only support max_events there it is difficult to estimate unless you know the average event sizes before-hand.

Given we recently added a method for approximating memory use of an event (#8396), I think we could leverage this to support max_bytes on in-memory buffers as well.

[jerome-kleinen-kbc-be]

In general I am trying to grasp how to keep the memory usage of the whole vector instance under control. The problem tends to arise if there is much data in the source, and the speed of the sink is next to unlimited. What I feel that happens is that the buffer and several concurrent batches are being kept in memory, leading to my vector instance hitting the predefined resource limits of my container. I have the feeling that vector is not aware of these limits and will just allocate as much as it can.

I am also wondering if there is some "formula" that can be used to estimate the memory usage of vector, for example request.concurrent * batch_size in bytes + buffer max_events * event size?

[binarylogic]

Isn't #12574 also related here? I think we'll want to do that before implementing max_bytes.

[jszwedko]

@binarylogic

Isn't #12574 also related here? I think we'll want to do that before implementing max_bytes.

Hmm, I'm not seeing how that is related. That issue is about fixing the metric calculation for the vector source's BytesReceived.

This issue is about capping memory use of memory buffers by size rather than number of events. Here we do want to use the in-memory size of the events as the intent of restricting the size here is to restrict the amount of memory Vector is consuming.