Feature request: allow Prometheus to scrape log-cache directly

[richardTowers]

Hello there,

I’m an operator of a multi-tenant OSS CF for the UK Government.

We provide a mechanism for our tenants to extract their app metrics and store them in Prometheus. To do this, we've had to build a few moving parts, with one significant piece being a /metrics endpoint which sits in front of log-cache and provides a view of the user visible metrics (based on the Authorization header) in a format which standard Prometheus scraping can ingest.

Suggested feature

We understand from this project's README and some #logcache Slack chat that a future aim is to provide endpoints which would enable existing PromQL API clients to talk to log-cache natively. We wonder if there would be any interest in also providing a /metrics endpoint such that Prometheus itself could scrape log-cache directly - effectively providing a prometheus exporter for log-cache?

This would bring several advantages for consumers, allowing:

• teams to use Prometheus' AlertManager and other non-PromQL ecosystem components
• application teams to decouple their stats from the platform
• operators to use existing Promethesis, which can be persistent and durable in a manner which we don't believe log-cache is currently designed (or aiming) to achieve.

We feel the /metrics contract is a good one for log-cache to expose to the Prometheus universe, as log-cache already uses the concept of "you'll see all the stats for which your API token gives you visibility" via the Authorization header.

Prometheus also contains the concept of a /federate API, which is similar to /metrics. /federate requires (or, at least, strongly suggests) that a consumer feeds the endpoint with a filter for stats it would like to see.

We think that /federate is probably not a great fit for log-cache's use case, where the set of metrics that a single OAuth token can access is implicit within the system. Providing a secondary restriction over the top of that set seems to run counter to log-cache's existing approach: just exposing all the stats which are available to the requestor.

Potential difficulties

Prometheus expects some metrics to reset (or be removed) when their last value becomes stale. If this isn't done properly we have observed issues like:

• metrics aren't removed when apps are deleted
• metrics for a given cell aren't removed when the app migrates to another cell
• metrics for an instance aren't removed when the app scales down

This may be solvable only by reading logs from Doppler, or may require log-cache to reach out to other parts of the system (which may be undesirable).

Next steps

If adding a /metrics endpoint aligns with your plans for log-cache we (GOV.UK PaaS) would be happy to contribute design and code as required. In the short term it's likely we'll implement something similar in spirit ourselves, as we already have live tenants using Prometheus via the projects mentioned below.

If this is not something that is likely to be added to log-cache then we may alter the design for our own metrics solutions to make them a more long-term part of our platform.

References

• https://github.com/alphagov/paas-metric-exporter - an internal project (useful for context only) which exports metrics to statsd and prometheus from Doppler.
• https://github.com/alphagov/paas-log-cache-adapter - an internal project (useful for context only) which exports metrics to prometheus from log-cache.
• https://cloudfoundry.slack.com/archives/CBFB7NP9B/p1540392230000100 - #logcache Slack chat
• https://prometheus.io/docs/instrumenting/writing_exporters/ - writing prometheus exporters
• Explicitly expire prometheus metrics when apps are deleted govuk-paas/paas-metric-exporter#33: an example of non-obvious difficulties we've found while working in this problem space

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/161482412

The labels on this github issue will be updated when the story is started.

[jtuchscherer]

@richardTowers just out of curiousity, have you look at this project: https://github.com/promregator/promregator
and at this:
https://github.com/bosh-prometheus/firehose_exporter

[richardTowers]

Hi @jtuchscherer - I hadn't seen the promregator project, but we had looked into the firehose_exporter.

This issue provides a slightly different feature in the following ways:

1. Unlike in promregator, apps don't need to have /metrics endpoints on them, since things like HttpStartStop metrics and container CPU usage are already available in log-cache.
2. Unlike in the firehose_exporter, this would solve the Authz issue of "should this tenant be able to view this metric" (since that's a feature that's already built in to log-cache)

(edit: s/PR/issue/)

[jtuchscherer]

@richardTowers Sorry, which PR are you referring to in your previous comment?

[richardTowers]

@jtuchscherer - sorry, I meant issue, not PR. I've edited my comment accordingly.

[jtuchscherer]

@richardTowers We would happily support you if you were adding the /metrics endpoint as a PR. But we think that this is actually a pretty large amount of work to get it right. Before you start working on this as a PR, it might be good if we all could jump on a call to coordinate this effort.

[hev]

This is a really cool idea and there might be a way to do this without modifying the bosh release. Similar to the space drain concept an app that lived in your space, and provided a routable prometheus endpoint would really be a powerful piece of glue for bridging Log Cache -> Prometheus

[richardTowers]

A call would be great - let me chat to my team and set something up.

We're currently using basically the space drain concept @ahevenor mentions, only it's scraping from the firehose rather than log-cache. We're keen to make sure that whatever solutions we build are in line with the direction you're taking though.

[richardTowers]

An update on our thoughts on this:

GOV.UK PaaS are using an app the provides a routable prometheus endpoint that scrapes log-cache (and the doppler firehose at the moment) - https://github.com/alphagov/paas-prometheus-exporter (pretty much as @ahevenor suggests above).

There's also been some discussion around different approaches, for example deploying prometheus itself into the platform (using prometheus-boshrelease) and then fronting that with some authentication to expose metrics to tenants (which is what Stratos are doing here - https://github.com/cloudfoundry-incubator/stratos/blob/856e13f14640dc96da994d483cadc80190244482/docs/bosh-metrics.md).

Thinking some more about this suggestion, I'm not sure it would work quite as neatly as I first thought. Although prometheus has support for basic auth on exporters, it certainly doesn't know how to authenticate to UAA and set an Authorization header. That means even if log-cache knew how to talk prometheus it's likely that tenants would still need to put something in front of it to handle the authentication, which negates some of the convenience.

I think there might still a case to be made that putting a log-cache-prometheus-exporter inside log-cache itself would make things easier for tenants, but I think the complexity we'd need to add to log-cache might outweigh the benefits.

I'll leave this open for now in case anyone else has any thoughts, but I'd be happy for the issue to be closed if people don't think there's a compelling case.