Support health status interface

[eyalkraft]

Motivation

Once libbeat is migrated to v2 cloudbeat should support the new health status interface to report detailed health information to the user. Including requiring the user to upgrade.
This information will surfaced in the UI. which is being tracked in .

Definition of done

• If there's an error, cloudbeat should report it an a way viewable from Kibana
• If there's a mismatch between the integration and the cloudbeat version (which could mean missing rules/fetchers on cloudbeats side) cloudbeat should indicate the user he needs to upgrade his agent.
• make sure we understand what is the behavior once the status is not healthy - does the agent restarts cloudbeat?

Out of scope

What is not included in this task

Related tasks/epics

• https://github.com/elastic/security-team/issues/3494
• Ensure integration compatibility when multiple agent versions are enrolled in a given agent policy package-spec#165 (comment)
• Design doc
• blocked on Display which integration is causing the agent to become unhealthy elastic-agent#100

[yashtewari]

BeatV2Manager with the new status/health update interfaces is not yet as mature as we'd expected. It's not being used by the "core" beats yet, as described by this issue

• https://github.com/elastic/ingest-dev/issues/1367

So it would be risky to touch it for the v8.6.0 release. I'll add more details about this in another comment.

For now, I've setup the required information/logic for version compatibility, and also started using V1 of Manager.UpdateStatus so that it may be easier to update to V2 going forward.

• Add version compatibility information to HBS files. integrations#4695
• Validate version compatibility in cloudbeat. #538

The current behavior is the same as any other error in cloudbeat: the error about incompatible versions is logged and cloudbeat is restarted by the Agent. But this is only tentative and we should make a decision on it.

[yashtewari]

I was able to send a Degraded signal using the V1 interface and make the Agent status on Kibana change to "Unhealthy".

We have two options now:

1. Fail fast: cloudbeat aborts when it is running an incompatible version

Pro: It makes doing upgrades to compatible versions "compulsory" in a way, so the user will immediately know that this action needs to be taken.
Con: If we change direction in the future on how we want to deal with version compatibility, we will have to still keep accounting for the current versions that abort on having a incompatible version.

2. Mark cloudbeat as "degraded" when running incompatible versions but keep going

Pro: Doesn't disrupt flow and is more "forward compatible".
Con: Currently the status change to "degraded" doesn't reveal too much information on Kibana to the user. It's not clear from the UI which integration specifically makes the Agent "unhealthy", and so it's not easy to look at the right logs, find the problem, and rectify it.

[eyalkraft]

2 seems closer to what we ant to end up with when the v2 health status reports work as expected

[oren-zohar]

@eyalkraft @yashtewari agreed, let's go with the second option

[yashtewari]

Sounds good, PRs have been updated.