Fix Cluster Activity historical metrics for ongoing dagrun if end_date is provided#33488
Merged
pierrejeambrun merged 1 commit intoAug 18, 2023
Conversation
pierrejeambrun
requested review from
ashb,
bbovenzi and
ryanahamilton
as code owners
August 18, 2023 00:09
bbovenzi
approved these changes
Aug 18, 2023
pierrejeambrun
deleted the
fix-cluster-activity-historical-metric-end-date
branch
August 18, 2023 17:09
ephraimbuddy
pushed a commit
that referenced
this pull request
Aug 28, 2023
(cherry picked from commit 7c51c87)
97 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We were selecting DagRuns/Tasks that were either:
Issue, if the end_date is chosen in the past, runs without end_date (ongoing run) will be counted in the metrics, which is wrong.
This handle such cases i.e when
dagrun.end_date is None:before
An old range such as this one would be updated with data regarding ongoing runs (i.e end_date is None)

After
This will not be the case anymore, data will remain stable on old ranges, because ongoing run will be taken out. In this example there is no records on this period.

To still be able to get data from ongoing run in the query we need to provide an
end_datethat is greater than 'now' by default we look ahead 1 hour.