Skip to content

Remove gauge scheduler.tasks.running #30374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 7 commits into from
Apr 13, 2023
Merged

Remove gauge scheduler.tasks.running #30374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c124505
Remove gauge scheduler.tasks.running
vincbeck Mar 30, 2023
8e279e6
Merge branch 'main' into vincbeck/scheduler_tasks_running
vincbeck Apr 3, 2023
71367a2
Merge branch 'main' into vincbeck/scheduler_tasks_running
vincbeck Apr 4, 2023
1b34061
Add significant.rst file
vincbeck Apr 4, 2023
09bb792
Merge branch 'main' into vincbeck/scheduler_tasks_running
vincbeck Apr 11, 2023
8615409
Update newsfragments/30374.significant.rst
vincbeck Apr 12, 2023
00167cb
Merge branch 'main' into vincbeck/scheduler_tasks_running
vincbeck Apr 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions airflow/jobs/scheduler_job_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,6 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session) -
states=list(EXECUTION_STATES), session=session
)

num_tasks_in_executor = 0
# Number of tasks that cannot be scheduled because of no open slot in pool
num_starving_tasks_total = 0

Expand Down Expand Up @@ -523,7 +522,6 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session) -
Stats.gauge(f"pool.starving_tasks.{pool_name}", num_starving_tasks)

Stats.gauge("scheduler.tasks.starving", num_starving_tasks_total)
Stats.gauge("scheduler.tasks.running", num_tasks_in_executor)
Stats.gauge("scheduler.tasks.executable", len(executable_tis))

if len(executable_tis) > 0:
Expand Down
1 change: 0 additions & 1 deletion docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,6 @@ Name Description
``dag_processing.file_path_queue_size`` Number of DAG files to be considered for the next scan
``dag_processing.last_run.seconds_ago.<dag_file>`` Seconds since ``<dag_file>`` was last processed
``dag_processing.file_path_queue_size`` Size of the dag file queue.
``scheduler.tasks.running`` Number of tasks running in executor
``scheduler.tasks.starving`` Number of tasks that cannot be scheduled because of no open slot in pool
``scheduler.tasks.executable`` Number of tasks that are ready for execution (set to queued)
with respect to pool limits, DAG concurrency, executor state,
Expand Down
5 changes: 5 additions & 0 deletions newsfragments/30374.significant.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
The gauge ``scheduler.tasks.running`` no longer exist

The gauge has never been working and its value has always been 0. Having an accurate
value for this metric is complex so it has been decided that removing this gauge makes
more sense than fixing it with no certainty of the correctness of its value.