AWSCloudWatch and awsemf exporters do not support role_arn AWS STS auth

[pavolloffay]

Component(s)

exporter/awscloudwatchlogs

What happened?

Description

The role_arn support was removed from cloudwach exporter in
99934f4#diff-f44af95874288433fb5e94e9bbb285bcced86e0a67770e04c30d3eb219401194

Steps to Reproduce

Create a collector with awscloudwatch exporter and role_arn.

Expected Result

Actual Result

Collector version

0.132.0 or main/HEAD

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-otel-collector
spec:
  # ... other OpenTelemetry Collector configuration ...

  # This is where you add environment variables
  env:
    - name: AWS_ENDPOINT_URL_CLOUDWATCHLOGS
      value: "https://add-cloudwatch-endpoint-here"
  
  # Ensure you remove the 'endpoint' from the exporter config if you're using the env var
  config: |
    exporters:
      awscloudwatchlogs:
        role_arn: arn:aws:iam::<masked account number>:role/CloudWatchAgentServer-CrossAccountSharingRole
        log_group_name: "/aws/rosa/logs/sb-replica-rosa-01"
        log_stream_name: ${NODE_NAME}
        log_retention: 3
        region: me-central-1
        # IMPORTANT: Remove the 'endpoint' field here
        # endpoint: https://.  <-- REMOVE THIS LINE
    # ... rest of your collector config ..

Log output

2025-07-22T12:02:16.466Z        info    internal/retry_sender.go:133    Exporting failed. Will retry the request after interval.        {"resource": {}, "otelcol.component.id": "awscloudwatchlogs", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "error flushing logs: operation error CloudWatch Logs: PutLogEvents, https response error StatusCode: 400, RequestID: 388627df-8a4b-4c17-a0b1-7901b57a0261, api error AccessDeniedException: User: arn:aws:sts::<account>:assumed-role/sb-rep-rosa-01-account-Worker-Role/<masked> is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:me-central-1:<account>:log-group:/aws/rosa/logs/sb-replica-rosa-01:log-stream:<masked>.me-central-1.compute.internal because no identity-based policy allows the logs:PutLogEvents action", "interval": "34.613421457s"}

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[pavolloffay]

The workaround is to define the role ARN in an env variable

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol-cloudwatch
spec:
  #image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.132.4
  serviceAccount: otelcol-cloudwatch
  env:
    - name: AWS_REGION
      valueFrom:
        secretKeyRef:
          name: aws-sts-cloudwatch
          key: region
    - name: AWS_ROLE_ARN
      valueFrom:
        secretKeyRef:
          name: aws-sts-cloudwatch
          key: role_arn
    - name: LOG_GROUP_NAME
      valueFrom:
        secretKeyRef:
          name: aws-sts-cloudwatch
          key: log_group_name
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}

    exporters:
      debug:
        verbosity: detailed
      # AWS CloudWatch Logs exporter with STS configuration
      awscloudwatchlogs:
        log_group_name: "${env:LOG_GROUP_NAME}"
        log_stream_name: "tracing-otelcol-stream"
        raw_log: true
        region: "${env:AWS_REGION}"
        endpoint: "https://logs.us-east-2.amazonaws.com"
        log_retention: 1
        tags: { 'tracing-otel': 'true', 'test-type': 'aws-sts' }

    processors:
      batch: {}
      # Add resource processor to include metadata
      resource:
        attributes:
        - key: test.scenario
          value: "role-arn-authentication"
          action: insert
        - key: deployment.environment
          value: "test"
          action: insert

    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [resource, batch]
          exporters: [awscloudwatchlogs, debug]

[mayank-shah05-personal]

@pavolloffay We have a scenario where we have to transfer logs/metrics cross account from host account to Destination account.

We specify our host arn in the service account, like shown below:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-liberty-mayank
  namespace: otel-test
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::HOSTACCOUNTID:role/HOST_ROLE_NAME

and earlier till otel version 0.120.0 we used role_arn within the exporter where we specify the Destination account, like shown below and things use to work just fine.

exporters:
      debug:
        verbosity: detailed
      awscloudwatchlogs:
        region: us-east-1
        log_group_name: MyApplicationLogs199
        log_stream_name: default
        role_arn: arn:aws:iam::DESTINATION_ACCOUNT_ID:role/DESTINATION_ROLE_NAME
        endpoint: https://logs.us-east-1.amazonaws.com

Please help us with your solution above, how do we solve our problem in any latest version of otel.

We tried to set role_arn in env as you suggested like shown below, but logs/metrics end up going to Host account and not to Destination account.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: otel-test
  labels:
    app: otel-collector
spec:
  env:
  -name: AWS_ROLE_ARN
    value: arn:aws:iam::DESTINATION_ACCOUNT_ID:role/DESTINATION_ROLE_NAME
  replicas: 1
  selector:
    matchLabels:

[axw]

@mayank-shah05-personal your Deployment spec is off. Environment variables go in the pod template, for the container. Please refer to the Kubernetes documentation for this.

[AbhishekRoy11-Private]

@pavolloffay @axw I am facing a similar issue.
I have a service account which is set to my host account.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-otel-collector-sa
  namespace: abhishek-liberty
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<Host Account ID>:role/<Host Role>

In earlier Opentelemetry version i was able to send logs Cross Account through creating Roles in Host and Customer and adding permission policy to my Host Account specifying the role_arn of customer IAM role using awsemfexporter and awscloudwatchlogs exporters.

exporters:
      debug:
        verbosity: detailed
      awscloudwatchlogs:
        region: us-east-1
        log_group_name: MyApplicationLogs
        log_stream_name: default
        role_arn: arn:aws:iam::<Customer Account ID>:role/<Customer Role>
        endpoint: https://logs.us-east-1.amazonaws.com

      awsemf:
        region: us-east-1
        endpoint: https://logs.us-east-1.amazonaws.com
        namespace: MyApp
        log_group_name: /aws/otel/aws-emf
        log_stream_name: emf-stream
        role_arn: arn:aws:iam::<Customer Account ID>:role/<Customer Role>
        resource_to_telemetry_conversion:
          enabled: true

But in newer version as the role_arn support to exporters is not working I am not able to send data cross account and my logs/metrics are polulated in my host account itself.

I tried the workaround by adding role_arn in env. But it is not working in my case.

spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      serviceAccountName: app-otel-collector-sa
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.132.0
          args: ["--config=/conf/config.yaml"]
          env:
            - name: AWS_ROLE_ARN
              value: arn:aws:iam::<Customer Account ID>:role/<Customer Role>
            - name: AWS_REGION
              value: us-east-1
          ports:
            - containerPort: 4317
            - containerPort: 4318
          volumeMounts:
            - name: config-volume
              mountPath: /conf
      volumes:
        - name: config-volume
          configMap:
            name: otel-collector-config

Can there be any solution to this? Or we need to stick to older version of OpenTelemetry until role_arn is support is back in exporters.

[phazeight]

bump here, currently trying to figure out how to send from some aws account that is a child, to a monitoring/parent account trying to figure out if there is a way to do so with profile attribute, makes most sense to me to use role_arn type of thing but apparently it's been removed?

[mayank-shah05-personal]

Hi AWSCloudWatch exporter Owners, Can someone please look into this Bug and try to fix this issue. We are blocked because of this because older versions have security issues which we cannot put into the running code.

[mayank-shah05-personal]

Hi AWSCloudWatch exporter Owners/@andrew Wilkins/@pavolloffay , Can someone please look into this Bug and try to fix this issue.
We are from IBM software group and looking for this fix so that we can plan our release.

[axw]

@mayank-shah05-personal there are currently no code owners - see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awscloudwatchlogsexporter. I don't use this component, and don't have time to look into it.