tail plugin with db-file: start reading from tail on the first execution

[Marki4711]

Is your feature request related to a problem? Please describe.
We like to fetch access-logs of java application servers (websphere and tomcat). This file is not "too" large and therefore not rotated. I use fluent-bit and elastic search as backend. each day gets an individual index. (default-behaviour for sending in logstash-format)
When I start fluent-bit the first time, hundrets of indices get created and my elastic search instances crashes.

Describe the solution you'd like
I ask to have an option that configures the tail-plugin, that it does not read the whole file when fluent-bit was first started but instead start at the end of the files.

Describe alternatives you've considered
I think about creating an configuration that send all the records to null output and wait, until the db-files are created and reached the end of all files. Then change the configuration to habe the correct output and get all new record to the elastic-search cluster. This is an very ugly workaround but I don't have a better idea at the moment.

Additional context
I use the db-parameter and it works fine, when the first transfer completed. I am currently in preprod-zones and the "small" files there already result in large problems. To use the solution in production, I need to find a solution for this issue.

[nigels-com]

It doesn't look like a straight-forward code change to me, due to log rotation.
I do agree with the use-case of only wanting new logs to flow, does seem useful.
Sorry I can't be more helpful.

[Marki4711]

Hello nigels,
thanks for your feedback.

Would it be thinkable to implement the approach I do as workaround as a standard functionality in the plugin?

When it starts, the current end of the files are remembered. Then the reading of lines is done as usual but not data is sent as result of the reading but all db-changes are made. When the remembered position on the files is reached, the behaviour changes back to normal. Lines are produced to the output and db-changes are made.

If it is not possible to remeber a specific position where the files and, it would be also sufficient to specify a timeframe for this null forwarding mode.
For example: If a db-file is specified and it does not exist on the disc and the null-timeframe option is specified, the plugin will work for the specified time in null-forwarding mode. After that, it forward the events as usual.

BR,
Matthias

[nigels-com]

One thought is to have time-based filter that drops all records for a configurable amount of time. It would have the advantage of not further complicating the tail plugin in particular.

[Marki4711]

I also thought about the filter approach but I can't figure out how thils filter can know if the tail-plugin was executed the first time and therefore needs to initialize the database. I don't want to loose logs on every restart of fluent-bit.
The filter is only allowed to do this, when the db-file is initialized for the first time.
Do you have an idea on this problem?

[edsiper]

fixed in 70e33fa

This will be part of release v1.6 next week.