Weekly environment - Vanilla POC by ofiriro3 · Pull Request #746 · elastic/cloudbeat

ofiriro3 · 2023-02-14T13:43:59Z

Summary of your changes

This is the first item in the development of the Daily-environment epic.

In this task, we created a job that deploys a vanilla KSPM integration on an Elastic cloud environment.
The job currently uses a pre-exited EC2 machine and it deploys an agent there (no new EC2 machines will be created).

The expected behavior is:

A new ELK environment is created with each execution.
A new Vanilla integration is installed on this environment.
The same EC2 machine whose credentials exist in the GitHub secret will deploy the agent.
A slack message will be sent to ofir-personal-channel with the relevant credentials for this environment.

How to check the daily job
Currently, the flow is deployed using a dispatch workflow, meaning - you will need to apply it using GitHub actions.
You can apply the flow from my fork, using actions.
Please set the branch to vanilla_poc before running the dispatch workflow.
It should look like this:

The flow will deploy the integration to https://cloud.elastic.co/home, in the next PRs we will change it to a staging environment and replace the latest image with the SNAPSHOT version.

Related Issues

Related https://github.com/elastic/security-team/issues/5903

Checklist

I have added tests that prove my fix is effective or that my feature works
I have added the necessary README/documentation (if appropriate)

… script

oren-zohar

Only went over the workflow

oren-zohar · 2023-02-15T10:00:05Z

+  TF_VAR_ec_api_key: ${{ secrets.WEEKLY_ENVIRONMENT_EC_API_KEY }}
+  TF_VAR_environment: ${{ github.event.inputs.logLevel }}
+  TF_LOG: ${{ github.event.inputs.logLevel }}
+  TF_VAR: 8.6.1


is it hard coded by design? can we use the branch version instead?

Yes, we would like the weekly environment to load a specific version every time it runs.

We would like it to run the latest SNAPSHOT version available. But we would like to change it manually every time since we would like to update the integration and the relevant rules.

I'm not sure I understand the logic behind it. Why not use the branch version as a default value?

So just to verify that I understood you, you meant for example if the Cloudbeat version is 8.8.0, so deploy 8.8.0-SNAPSHOT?

So that is a great idea, but for the matter of this PR, we wanted to start with a POC that will use a hard-coded version.
In the future we would like even to support multiple versions:

https://github.com/elastic/security-team/issues/6078

But for this POC, we assumed that it uses a hard-coded version 8.6.1 as the current Terraform default value suggests.

oren-zohar · 2023-02-15T10:02:54Z

+      - name: Setup Terraform
+        uses: hashicorp/setup-terraform@v2
+        with:
+          terraform_version: 1.3.5


don't we have the terraform version as part of hermit? I think we need to work with it in the CI, as it will make sure it will work properly locally for everyone in the future. What do you think?

cloudbeat/.github/workflows/cloudbeat-ci.yml

Line 27 in 41cd884

- name: Init Hermit

Tested it, good advice! I removed the Terraform setup step and added a Init Hermit step

oren-zohar · 2023-02-15T10:06:56Z

+      - name: Terraform fmt
+        id: fmt
+        run: terraform fmt -check
+        continue-on-error: true
+
+      - name: Terraform Init
+        id: init
+        run: terraform init -no-color
+
+      - name: Terraform Validate
+        id: validate
+        run: terraform validate -no-color


Nice! but I think the linting should be in a separate step, as it is not part of the deployment logic, can run in parallel, and on failure, easier to understand what part failed and when.

I removed it 👍

I still think it's valuable to have it - just in a separate step, maybe in one of the other workflows where we run linters?

olegsu · 2023-02-16T07:38:22Z

Do we plan to consolidate it to the other terraform module at some point? what are the requirementes?
If not, why?

ofiriro3 · 2023-02-20T09:29:19Z

Do we plan to consolidate it to the other terraform module at some point? what are the requirements?

Currently, the weekly environment job uses terraform to create a new Elastic cloud environment and does not require any other terraform modules from our solutions. Other actions are done by ssh to the relevant EC2 machine and applying the specific operations there.

For the first deliverable solution, we decided that the job won't create a new EC2 machine and its infrastructure, and instead will use an existing machine. This assumption was made to enable us to move fast with a reliable solution and not be dependent on the rest of the terraform modules. So in this case, I thought that doing the ssh will be done much easier in a bash script.

As I see it, in the future we will might decide to consolidate it with other terraform modules.
For example, we will might decide to create an EC2 machine if such does not exist as this ticket suggests - https://github.com/elastic/security-team/issues/5788, but before doing so, we will have to reconsider some of the dependencies we currently have:

EC2 creation + agent deployment - we would like to deploy an agent on every job execution, but not to deploy a new EC2 every time.
EKS creation + agent deployment - same as 1.
API module - I guess that this entire module can be refactored.

olegsu · 2023-02-21T06:19:28Z

Thank you for a detailed explanation, @ofiriro3.

Currently, the weekly environment job uses terraform to create a new Elastic cloud environment and does not require any other terraform modules from our solutions. Other actions are done by ssh to the relevant EC2 machine and applying the specific operations there.

Imo, this is not ideal to have multiple scripts doing practically the same thing. I think a consolidation should be added to #622, not to DOD but to something that we want to track and achieve when possible. Otherwise, it would be hard to maintain two scripts.

ofiriro3 · 2023-02-21T08:31:56Z

#622, not to DOD but to something that we want to track and achieve when possible. Otherwise, it would be hard to maintain two scripts.

Done. I commented on the ticket

olegsu

Looks good!
What happens if the machine does not exist?
Does the slack message still goes to ofir-personal-channel ?

olegsu · 2023-02-22T12:05:21Z

+      - name: Init Hermit
+        run: ./bin/hermit env -r >> $GITHUB_ENV
+
+      - name: Terraform fmt


Lets have it atm only on the weekly env dir

I don't mind changing it, but why we wouldn't want it to run on the entire solution?

The CI passes, and it will keep all the terraform in format.

olegsu · 2023-02-22T12:05:52Z

+env:
+  WORKING_DIR: deploy/weekly-environment
+  SCRIPTS_DIR: deploy/weekly-environment/scripts/benchmarks/kspm_vanilla
+  TF_VAR_ec_api_key: ${{ secrets.WEEKLY_ENVIRONMENT_EC_API_KEY }}


Who is the user behind this key?

Currently its my user, we have two tickets related:

Short term solution - https://github.com/elastic/security-team/issues/6017, add the entire team to have permission on my account.

Long term solution - Create a team account - https://github.com/elastic/security-team/issues/5921

ofiriro3 · 2023-02-22T12:40:01Z

Looks good!
What happens if the machine does not exist?
Does the slack message still goes to ofir-personal-channel ?

If the machine does not exist, no message is sent to the slack channel, but the job fails.

ofiriro3 added 30 commits February 7, 2023 14:56

update integration to 1.1.1

aa43057

Adding only elastic stack

8163f42

update working directory

212965d

update working directory

061b309

adding the ec provider

4bb7688

Fixing gitignore files + adding github actions that uses the new bash…

050aeb7

… script

Moving scripts to the weekly directory

d6b9978

Updating executing permissions

b06001e

Update local variable to lower_case

e615ee6

updating path

cf80da8

Updating path

bfc4c0a

Fixing path

4c18e6d

Updating to full path

5a78456

Updating to full path

e3ad992

Updating CI to full path

fc81e64

using variable

18fd270

using variable

7c324e6

flow without agent deployment

bfe4eb3

Fixing integration

f781ffb

Update CI to full E2E

2b6346a

Fixing show manifest file step

6c18d50

update ci

48050c6

Deploy yaml file

2e63a33

Another try

ae06992

updating deployment file

52b55ac

Remove image replacement

9adc67c

Remove secret echo to file

a14ad1d

Trying to solve ssh issue

1006b22

Trying to solve ssh issue

d0b9bc7

Adding the entire flow

d71688f

ofiriro3 marked this pull request as ready for review February 15, 2023 09:31

ofiriro3 added 3 commits February 15, 2023 11:49

Merge branch 'main' into vanilla_poc

d02a1ba

Update weekly-enviroment.yml EC2 Private key parameter name

65a8a33

Merge remote-tracking branch 'origin/vanilla_poc' into vanilla_poc

0426cd7

oren-zohar requested changes Feb 15, 2023

View reviewed changes

apply review comments

2c64595

ofiriro3 requested a review from oren-zohar February 20, 2023 10:29

ofiriro3 mentioned this pull request Feb 20, 2023

Update daily environment to use 8.7 SNAPSHOT #758

Merged

4 tasks

ofiriro3 added 3 commits February 21, 2023 09:29

Trying to use Hermit without Terraform setup step

e726da8

Updating hermit path

7694167

using Hermit

2f337e7

ofiriro3 added 2 commits February 21, 2023 13:52

Run terraform format

b46b90e

Adding lint for terraform

eab79a4

olegsu reviewed Feb 22, 2023

View reviewed changes

ofiriro3 requested a review from olegsu February 22, 2023 12:45

olegsu approved these changes Feb 23, 2023

View reviewed changes

Merge branch 'main' into vanilla_poc

8a18f5c

oren-zohar approved these changes Feb 23, 2023

View reviewed changes

ofiriro3 enabled auto-merge (squash) February 23, 2023 09:24

ofiriro3 removed the request for review from kfirpeled February 23, 2023 09:24

Merge branch 'main' into vanilla_poc

fac6d42

ofiriro3 disabled auto-merge February 23, 2023 09:24

ofiriro3 enabled auto-merge (squash) February 23, 2023 09:25

ofiriro3 disabled auto-merge February 23, 2023 10:25

ofiriro3 merged commit f217bc8 into elastic:main Feb 23, 2023

Uh oh!

Conversation

ofiriro3 commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of your changes

Related Issues

Checklist

Uh oh!

oren-zohar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

olegsu commented Feb 16, 2023

Uh oh!

ofiriro3 commented Feb 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

olegsu commented Feb 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ofiriro3 commented Feb 21, 2023

Uh oh!

olegsu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ofiriro3 commented Feb 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ofiriro3 commented Feb 14, 2023 •

edited

Loading

ofiriro3 commented Feb 20, 2023 •

edited

Loading

olegsu commented Feb 21, 2023 •

edited

Loading