Emit metrics for compaction priority queues

[keith-turner]

In #3460 a new bounded priority queue was introduced for compaction jobs. Metrics need to be emitted for these priority queues so that the current amount queued can be known. Knowing this can useful for users tuning settings and for K8s autoscaling of compactors.

[ddanielr]

I spent some time looking over #3640 for these changes and started working on the logic to add the following metrics.

1. Queue Size
2. Jobs Queued
3. Job Time Spent in Queue.

Are there any other metrics that should be added?

[keith-turner]

Are there any other metrics that should be added?

It may be useful to have a sense of the number of jobs being rejected or ejected because the queue is at max size. But I am not sure how to capture this as a metric.

[ddanielr]

Added the following metrics:

1. Number of Queues
2. Size of Queue
3. Number of Jobs in Queue
4. Number of Jobs Rejected
5. Number of Jobs Dequeued.

I'm not sure how to best track job queue time as that starts making a LOT of time series metrics that are short lived.
We could work on adding telemetry exposing the job queue time there, but that's outside of metrics.

We could also expose a MaxWait metric per queue, but I'd love to actively track the wait time vs calculate it based on when the job leaves the queue.

[dlmarion]

@ddanielr - Did #3551 resolve this, or is there more to do here?

[ddanielr]

@ddanielr - Did #3551 resolve this, or is there more to do here?

@dlmarion Yes I believe this ticket is completed.
I added a separate follow-on issue for improving a test condition. See #3993