[Feature request] Setting Mem_Buf_limit of td-agent-bit

[tsubasaxZZZ]

In an AKS environment, logs are lost in an environment where a large amount of logs are output from containers.
I forced myself to change Mem_Buf_limit from its default value in the container of omsagent's DaemonSet, and this improved the situation.
It would be very helpful if the Mem_Buf_limit setting of td-agent-bit could be changed using ConfigMap.

# cat /etc/opt/microsoft/docker-cimprov/td-agent-bit.conf
[INPUT]
    Name tail
    Tag oms.container.log.la.*
    Path ${AZMON_LOG_TAIL_PATH}
    DB /var/log/omsagent-fblogs.db
    DB.Sync Off
    Parser cri
    Mem_Buf_Limit 10m <------------------------------- I would like to change this parameter
    Rotate_Wait 20
    Refresh_Interval 30
    Path_Key filepath
    Skip_Long_Lines On
    Ignore_Older 5m
    Exclude_Path ${AZMON_CLUSTER_LOG_TAIL_EXCLUDE_PATH}

[ykwyuta]

The default value of Mem_Buf_Limit is so small that logs can be lost in the event of a temporary increase in traffic.
In fact, in our project, even though the node's CPU, memory, network, and other resources have sufficient reserves, there was an event in which logs could not be transferred to Log Analytics and logs were missing.
Since Mem_Buf_Limit has become a fixed value, currently, the only thing that can be done to prevent the log from being lost is to increase the number of nodes.

[daweim0]

@tsubasaxZZZ Thank you very much for submitting a patch to add this feature, we'll review it in the next few days. Just to set expectations properly, it takes about a month for updates to become available in all regions (so don't hold your breath waiting for this feature).

[phonojp]

Logs are necessary for business and system operations, so we want them to be written to LogAnalytics without loss.
If the configuration values can be changed, I want to manage them in ConfigMap like other settings in k8s.
I'm looking forward to it.

[daweim0]

PR is merged, thank you again for the fix. I think it will go out in the next few weeks. @tsubasaxZZZ can I close this issue?

Just keep in mind that increasing the buffer size here will not increase the container memory limit. Also increasing the container memory limit and request might be a good idea, although this is only possible when onboarding though the helm.

[tsubasaxZZZ]

@daweim0 Thank you for your quick review and merge! Great!
For my issue management, could you open it until the rollout is completed and I can check it on AKS?

[tsubasaxZZZ]

I confirmed roll-out with creating new cluster. So I will close this issue. Thank you for your cooperation.
And I create new PR to sample ConfigMap. Please review and merge.
#606