Feature/d4c poc

.github/CODEOWNERS

@@ -206,6 +206,7 @@
 /packages/claroty_xdome @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/cloud_asset_inventory @elastic/cloud-services
 /packages/cloud_defend @elastic/cloud-services
+/packages/cloud_defend_otel @elastic/cloud-services
 /packages/cloud_security_posture @elastic/cloud-services
 /packages/cloudflare @elastic/security-service-integrations @elastic/sit-crest-contractors
 /packages/cloudflare_logpush @elastic/security-service-integrations @elastic/sit-crest-contractors

packages/cloud_defend_otel/DEVELOPMENT.md

@@ -0,0 +1,115 @@
+# Local Development & Testing Guide
+
+This guide covers how to manually test the `cloud_defend_otel` integration locally against a running Elastic Stack.
+
+## Prerequisites
+
+- Go (version from `.go-version` in the repo root)
+- Docker and Docker Compose v2
+- `jq` and `yq`
+
+Install `elastic-package` from the repo root:
+
+```bash
+cd /path/to/elastic/integrations
+go build -o build/elastic-package github.com/elastic/elastic-package
+export PATH="$PWD/build:$PATH"
+```
+
+## 1. Start the Elastic Stack
+
+From within the package directory, start Elasticsearch, Kibana, and Fleet Server:
+
+```bash
+cd packages/cloud_defend_otel
+
+elastic-package stack up -v
+```
+
+> **Note**: The package requires Kibana `^8.19.0 || ^9.3.0` and an `enterprise` subscription — both flags above are required.
+
+Check the stack is ready:
+
+```bash
+elastic-package stack status
+```
+
+Kibana will be available at **https://localhost:5601** (`elastic` / `changeme`).
+
+## 2. Install the Integration
+
+Build and upload the integration from your local branch into Fleet:
+
+```bash
+elastic-package install -v
+```
+
+This registers the package so it's available to create policies in Kibana.
+
+## 3. Enroll an Elastic Agent
+
+Go to **Kibana → Fleet → Agents → Add agent**, create an agent policy using the `cloud_defend_otel` integration, then enroll a local agent using the enrollment token shown in the UI:
+
+```bash
+docker run \
+  --network elastic-package-stack_default \
+  --env FLEET_ENROLL=1  --env FLEET_INSECURE=true \
+  --env FLEET_URL=https://fleet-server:8220 \
+  --env FLEET_ENROLLMENT_TOKEN="${FLEET_ENROLLMENT_TOKEN}" \
+  -p 4317:4317 \
+  docker.elastic.co/elastic-agent/elastic-agent:9.4.2
+```
+
+Tunnel
+
+```bash
+gcloud compute ssh --zone "us-east1-d" "biscout42-d4c-build" --tunnel-through-iap --project "elastic-security-dev" -- -NR 4317:localhost:4317  -o ServerAliveInterval=30 -o ServerAliveCountMax=3
+```
+
+### Using a custom Elastic Agent image
+
+To completely replace the agent image used by the stack (e.g. a locally built or unreleased image), set the `ELASTIC_AGENT_IMAGE_REF_OVERRIDE` environment variable before starting the stack:
+
+```bash
+export ELASTIC_AGENT_IMAGE_REF_OVERRIDE=docker.elastic.co/elastic-agent/elastic-agent:my-custom-tag
+elastic-package stack up -v
+```
+
+This overrides the agent image entirely, regardless of the `--version` flag.
+
+The integration listens for OTLP telemetry on:
+- **gRPC**: `:4317`
+- **HTTP**: `:4318`
+
+The default dataset is `cloud_defend.logs`; set `data_stream.dataset` in the policy to `cloud_defend.file` for the file dataset.
+
+## 4. Run System Tests (automated end-to-end)
+
+`elastic-package test system` handles agent enrollment automatically via Docker, sends test data through the OTLP endpoints, and asserts that documents land in Elasticsearch:
+
+```bash
+elastic-package test system -v
+```
+
+Test configs are in `_dev/test/system/`. Each expects a minimum of 50 documents. Tests run sequentially (`parallel: false`).
+
+To run all test types (system, policy, pipeline, asset):
+
+```bash
+elastic-package test -v
+```
+
+## 5. View Data in Kibana
+
+After data is flowing:
+- **Discover**: query index pattern `logs-cloud_defend.*`
+- **Dashboards**: any dashboards bundled with the integration appear under Kibana → Dashboards
+
+## 6. Teardown
+
+```bash
+# Optional: dump stack logs before stopping (useful for debugging failures)
+elastic-package stack dump -v --output ../../build/elastic-stack-dump/cloud_defend_otel
+
+elastic-package stack down -v
+```

packages/cloud_defend_otel/LICENSE.txt

@@ -0,0 +1,93 @@
+Elastic License 2.0
+
+URL: https://www.elastic.co/licensing/elastic-license
+
+## Acceptance
+
+By using the software, you agree to all of the terms and conditions below.
+
+## Copyright License
+
+The licensor grants you a non-exclusive, royalty-free, worldwide,
+non-sublicensable, non-transferable license to use, copy, distribute, make
+available, and prepare derivative works of the software, in each case subject to
+the limitations and conditions below.
+
+## Limitations
+
+You may not provide the software to third parties as a hosted or managed
+service, where the service provides users with access to any substantial set of
+the features or functionality of the software.
+
+You may not move, change, disable, or circumvent the license key functionality
+in the software, and you may not remove or obscure any functionality in the
+software that is protected by the license key.
+
+You may not alter, remove, or obscure any licensing, copyright, or other notices
+of the licensor in the software. Any use of the licensor’s trademarks is subject
+to applicable law.
+
+## Patents
+
+The licensor grants you a license, under any patent claims the licensor can
+license, or becomes able to license, to make, have made, use, sell, offer for
+sale, import and have imported the software, in each case subject to the
+limitations and conditions in this license. This license does not cover any
+patent claims that you cause to be infringed by modifications or additions to
+the software. If you or your company make any written claim that the software
+infringes or contributes to infringement of any patent, your patent license for
+the software granted under these terms ends immediately. If your company makes
+such a claim, your patent license ends immediately for work on behalf of your
+company.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software from you
+also gets a copy of these terms.
+
+If you modify the software, you must include in any modified copies of the
+software prominent notices stating that you have modified the software.
+
+## No Other Rights
+
+These terms do not imply any licenses other than those expressly granted in
+these terms.
+
+## Termination
+
+If you use the software in violation of these terms, such use is not licensed,
+and your licenses will automatically terminate. If the licensor provides you
+with a notice of your violation, and you cease all violation of this license no
+later than 30 days after you receive that notice, your licenses will be
+reinstated retroactively. However, if you violate these terms after such
+reinstatement, any additional violation of these terms will cause your licenses
+to terminate automatically and permanently.
+
+## No Liability
+
+*As far as the law allows, the software comes as is, without any warranty or
+condition, and the licensor will not be liable to you for any damages arising
+out of these terms or the use or nature of the software, under any kind of
+legal claim.*
+
+## Definitions
+
+The **licensor** is the entity offering these terms, and the **software** is the
+software the licensor makes available under these terms, including any portion
+of it.
+
+**you** refers to the individual or entity agreeing to these terms.
+
+**your company** is any legal entity, sole proprietorship, or other kind of
+organization that you work for, plus all organizations that have control over,
+are under the control of, or are under common control with that
+organization. **control** means ownership of substantially all the assets of an
+entity, or the power to direct its management and policies by vote, contract, or
+otherwise. Control can be direct or indirect.
+
+**your licenses** are all the licenses granted to you for the software under
+these terms.
+
+**use** means anything you do with the software requiring one of your licenses.
+
+**trademark** means trademarks, service marks, and similar rights.

packages/cloud_defend_otel/_dev/build/docs/README.md

@@ -0,0 +1,43 @@
+{{- generatedHeader }}
+# Defend for Containers (OpenTelemetry)
+
+## Overview
+
+The Defend for Containers (OpenTelemetry) input package enables Elastic Agent to receive Defend for Containers telemetry using the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/) over gRPC and HTTP. The data is delivered as OTLP logs and routed into an Elasticsearch dataset.
+
+The package uses the upstream [OTLP Receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver) from the OpenTelemetry Collector.
+
+## How it works
+
+This package configures the OTLP receiver in the EDOT (Elastic Distribution of OpenTelemetry) Collector managed by Elastic Agent. Data flows as follows:
+
+1. An OTLP-compatible source sends logs to the receiver over gRPC or HTTP.
+2. The EDOT Collector inside Elastic Agent receives and processes the telemetry.
+3. Elastic Agent forwards the data to Elasticsearch for indexing and analysis.
+
+The receiver listens on two endpoints:
+
+- **gRPC** on `localhost:4317` (default)
+- **HTTP** on `localhost:4318` (default)
+
+## Configuration
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| gRPC Endpoint | Address and port to listen on for OTLP gRPC connections | `localhost:4317` |
+| HTTP Endpoint | Address and port to listen on for OTLP HTTP connections | `localhost:4318` |
+| Data Stream Dataset | Dataset to write data to | `cloud_defend.logs` |
+
+Each configured instance of this input writes to a single dataset, selected with the `Data Stream Dataset` variable. To route Defend for Containers telemetry into both supported datasets — `cloud_defend.logs` and `cloud_defend.file` — configure two instances of the input, one per dataset.
+
+For the complete list of receiver settings, refer to the [OTLP Receiver documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver).
+
+## Reference
+
+### Inputs used
+{{ inputDocs }}
+
+### Further reading
+
+- [OTLP Receiver Documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver)
+- [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/)

packages/cloud_defend_otel/_dev/deploy/docker/Dockerfile

@@ -0,0 +1,8 @@
+FROM golang:1.26-alpine AS builder
+WORKDIR /app
+COPY generator/ .
+RUN go mod download && go build -o /generator .
+
+FROM alpine:3.21
+COPY --from=builder /generator /usr/local/bin/generator
+ENTRYPOINT ["/usr/local/bin/generator"]

packages/cloud_defend_otel/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,4 @@
+version: '2.3'
+services:
+  cloud_defend_otel:
+    build: .

packages/cloud_defend_otel/_dev/deploy/docker/generator/go.mod

@@ -0,0 +1,31 @@
+module generator
+
+go 1.26
+
+require (
+	go.opentelemetry.io/otel v1.43.0
+	go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.19.0
+	go.opentelemetry.io/otel/log v0.19.0
+	go.opentelemetry.io/otel/sdk v1.43.0
+	go.opentelemetry.io/otel/sdk/log v0.19.0
+)
+
+require (
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/metric v1.43.0 // indirect
+	go.opentelemetry.io/otel/trace v1.43.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.10.0 // indirect
+	golang.org/x/net v0.52.0 // indirect
+	golang.org/x/sys v0.42.0 // indirect
+	golang.org/x/text v0.35.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
+	google.golang.org/grpc v1.80.0 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
+)

packages/cloud_defend_otel/_dev/deploy/docker/generator/go.sum

@@ -0,0 +1,63 @@
+github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
+github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
+go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.19.0 h1:Dn8rkudDzY6KV9dr/D/bTUuWgqDf9xe0rr4G2elrn0Y=
+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.19.0/go.mod h1:gMk9F0xDgyN9M/3Ed5Y1wKcx/9mlU91NXY2SNq7RQuU=
+go.opentelemetry.io/otel/log v0.19.0 h1:KUZs/GOsw79TBBMfDWsXS+KZ4g2Ckzksd1ymzsIEbo4=
+go.opentelemetry.io/otel/log v0.19.0/go.mod h1:5DQYeGmxVIr4n0/BcJvF4upsraHjg6vudJJpnkL6Ipk=
+go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
+go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
+go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg=
+go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
+go.opentelemetry.io/otel/sdk/log v0.19.0 h1:scYVLqT22D2gqXItnWiocLUKGH9yvkkeql5dBDiXyko=
+go.opentelemetry.io/otel/sdk/log v0.19.0/go.mod h1:vFBowwXGLlW9AvpuF7bMgnNI95LiW10szrOdvzBHlAg=
+go.opentelemetry.io/otel/sdk/log/logtest v0.19.0 h1:BEbF7ZBB6qQloV/Ub1+3NQoOUnVtcGkU3XX4Ws3GQfk=
+go.opentelemetry.io/otel/sdk/log/logtest v0.19.0/go.mod h1:Lua81/3yM0wOmoHTokLj9y9ADeA02v1naRrVrkAZuKk=
+go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw=
+go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
+go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
+go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
+go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=
+go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=
+golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
+golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
+golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
+golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
+golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 h1:VPWxll4HlMw1Vs/qXtN7BvhZqsS9cdAittCNvVENElA=
+google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9/go.mod h1:7QBABkRtR8z+TEnmXTqIqwJLlzrZKVfAUm7tY3yGv0M=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 h1:m8qni9SQFH0tJc1X0vmnpw/0t+AImlSvp30sEupozUg=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
+google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM=
+google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
+google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
+google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=