upload nightly tps metrics to s3

[THardy98]

What was changed

Upload nightly prometheus worker metrics to S3 for querying in Omni

Why?

observability

[cretz]

Are there no secrets required here? So anyone can upload to this bucket?

[THardy98]

We assume an ARN that gives us perms to write to this specific bucket, from specific github repos (the sdk repos)

[cretz]

But doesn't that let anyone do the same since the ARN is not a secret and is visible by everyone? Or is there some way that the AWS endpoint knows it's from this GH workflow and not from someone else's S3 client? Otherwise, it there some secret/auth that we can set to prove it came from us in these GH workflows?

[THardy98]

From what I was told, the ARN only permits uploads from our SDK repos. (based on its trust policy), but I can move it behind a secret regardless

[cretz]

I'd be interested to confirm how that works. So the ARN restricts to uploading from GH worker IPs only? Yes, would definitely prefer using secrets when uploading.

[THardy98]

moved to a secret

I believe it uses GH OIDC in-tandem with a trust policy to scope to specific repos with specific permissible actions (basically saying "only repo X have permission to do action Y"). Given that this workflow is only trigger-able via cron and manual dispatch (only available to maintainers), it can't be abused.

This blog post covers the setup
https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/

[cretz]

Ah, it does an OIDC handshake and AWS validates the JWT against GH actions signed token. If setup on the AWS side to validate the GH key, then yes, no concerns about a user being able to use this. Approving (though up to you whether to move back our from secrets now that I understand what aws-actions/configure-aws-credentials is doing).

[cretz]

This logic here is getting more and more involved. I think it'd be wise to consider either a Go project to encapsulate this logic or moving this workflow file to a centralized place (e.g. omes itself?) that is only invoked from here, similar to what we do with features. Arguably you don't even need to invoke from the SDK repos because there's not really any value in doing it in the SDK repos.

[THardy98]

Yeah I agree.
I have a draft PR in omes that covers the non-lang specific part of the workflow (prior to the S3 upload), which basically was the omes run itself, but I liked being able to enable flags independently in each SDK. Arguably you could allow these flags to be inputs though.

I'll move this to a shared workflow in the future.