update our agent spec to fit the new integration stream format

[oren-zohar]

Background
As @DaveSys911 pointed out, currently, our beat is parsing the stream as is, instead of defining the expected params in the cloudbeat agent spec

In Depth
if we'll look at the new permission parameters we added to the integration, for example, session_token, our beat expects the following format to come from the integration:

stream:
   session_token: <some_session_token>

instead of just:

session_token: <some_session_token>

This has the following disadvantages:

1. it will force our cloudbeat reference to be nested under the stream key, for any param we're expecting to receive from the integration.
2. Setting default values in the configuration yaml can be more complex, as it's not flat
3. this is not following the convention that is seen in other beats
4. this will force the standalone configuration to add streams to the config, which is confusing since the standalone should not handle outside streams at all.

Definition of done

• update cloudbeat spec to support the new params
• remove the streams from our deployment ymls
• update our stream parsing (should not expect the stream key)

[orouz]

this would also be useful for us in kibana. currently to update the vars we access the first stream element, which is kinda arbitrary.

[eyalkraft]

related:

• https://github.com/elastic/security-team/issues/4499

[eyalkraft]

After briefly going through the rest of the specs, I'm pretty sure session_token doesn't belong in cloudbeat's spec.
For example, there is a session_token input var in the aws integration but it doesn't show on any of the specs.

Additionally, the whole point of the integrations is to provide us the flexibility of introducing additional configurations and changes to our integration without requiring changes in the elastic agent or Kibana, and thus without requiring a stack release.

That being said, I also don't fully understand the spec files and it could definitely be that we're not properly utilizing it.
Although I'm not sure it's the right call, I think we will be able to move the session token from the stream specific configuration to the general cloudbeat configuration changing the spec file without explicitly listing session_token there (something like this).

Maybe @cmacknz or @ph could help us understand the full capabilities of the agent input spec files.
It would be really helpful for us to understand:

1. What should logically belong in the spec file vs what should belong in the integration .hbs file
2. What's possible doing in the spec file, when does it take effect in the process of installing and running an integration?
3. What configurations are supposed to reside scoped in the streams config and which belong to the whole input, Given that we have a single stream in our integration?
4. What are possible motivations for splitting our single stream to multiple streams?

Very much appreciated 😃

[cmacknz]

This shouldn't live in the spec file, and in the V2 agent architecture putting this in the spec file will be impossible as all configuration related content is being removed.

The spec files are meant to tell the agent what inputs map to a known binary/executable along with what architectures and outputs the binary supports. You can see an example of the V2 spec file for cloudbeat in the agent V2 feature branch here which contains only this information.

The history for why the V1 spec files contained configuration and configuration transformations was to implement the agent as a supervisor for existing beats without having to modify the beats themselves. This pushed a lot of complexity into the agent as it needed to know how to transform an agent policy into a beat configuration file. Now that we have time to redesign the agent and rework the beats themselves this functionality is being removed.

To answer your specific questions:

1. What should logically belong in the spec file vs what should belong in the integration .hbs file.

The spec files will be restricted to containing only the information needed by the agent to determine how and when the agent should start a binary. All configuration should be controlled through integrations. Inputs/binaries should parse the relevant integration configuration themselves.

2. What's possible doing in the spec file, when does it take effect in the process of installing and running an integration

The spec files defines a mapping between an input as defined by an agent policy and the binary that implements it. This will be the only interaction between integrations configurations and spec files in the V2 architecture - controlling which input to start.

3. What configurations are supposed to reside scoped in the streams config and which belong to the whole input, Given that we have a single stream in our integration?

Each stream defines an instance of an input with a different configuration. For a Beat like Filebeat, there could be a single logfile input defined in the agent policy multiple streams configured. That results in one instance of the Filebeat process with multiple instances of the log input configured to run.

With that model anything that can vary per instance of an input would go into the stream section, and only configuration that is meant to apply to all instances of an input would be defined at the input level.

4. What are possible motivations for splitting our single stream to multiple streams?

At least for the existing Beat inputs, the biggest reason would be that you can only define the data_stream parameter once per stream so if you want the same input type to write to multiple data streams you would have multiple streams in the agent policy.

This is not entirely consistent everywhere, for example the endpoint security integration uses an almost totally different set of conventions and does not list the data streams it uses in an agent policy. How and when to do this is an implementation detail of a particular input from the agents perspective, the agent ideally does not care about how any particular input defines its configuration.

[DaveSys911]

@oren-zohar @eyalkraft the v2 agent seems to handle this area quite differently. Depending on the delivery date of the v2 arch - it might be a good idea to postpone this or make an interim solution, with v2 in mind.

[oren-zohar]

@cmacknz, thank you for your response. Let's close this case for now and revisit it when we begin integrating with v2. If you think we should implement something like this in the meanwhile, or if you have any other suggestions, please let me know @eyalkraft @DaveSys911.

[eyalkraft]

Thank you for the detailed response @cmacknz!