Skip to content

Fix Kafka consumer not being closed on error in ConsumeFromTopicOperator#69641

Open
TimurRakhmatullin86 wants to merge 1 commit into
apache:mainfrom
TimurRakhmatullin86:fix/kafka-consume-operator-consumer-leak
Open

Fix Kafka consumer not being closed on error in ConsumeFromTopicOperator#69641
TimurRakhmatullin86 wants to merge 1 commit into
apache:mainfrom
TimurRakhmatullin86:fix/kafka-consume-operator-consumer-leak

Conversation

@TimurRakhmatullin86

Copy link
Copy Markdown

ConsumeFromTopicOperator.execute only called consumer.close() on the happy path. If apply_function / apply_function_batch raised, or consumer.consume() / consumer.commit() failed, the operator leaked the consumer: the group member stays registered on the broker until session.timeout.ms expires (delaying partition rebalance for the rest of the consumer group), and the underlying librdkafka handle keeps its sockets and background threads alive in the worker until GC.

This wraps the processing loop in try/finally so the consumer is always closed:

  • the loop body is unchanged (re-indent only);
  • offset commits stay inside try, so nothing is committed for a failed batch;
  • close() in finally is guarded with a log.warning, mirroring the AwaitMessageTrigger consumer cleanup from Add cleanup to Kafka AwaitMessageTrigger for consumer management #64612close() performs a final synchronous offset commit when enable.auto.commit is on (the librdkafka default), so it can itself raise and would otherwise replace the original processing error as the task failure reason.

Tests: test_execute_closes_consumer_when_apply_function_raises (fails without the fix) and test_execute_does_not_mask_error_when_close_raises.

Signed-off-by: Timur Rakhmatullin <174210871+TimurRakhmatullin86@users.noreply.github.com>
@eladkal
eladkal requested a review from jason810496 July 9, 2026 15:31
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants