Skip to content

[Azure frontdoor] Add Health Probe dataset #20016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
herrBez wants to merge 12 commits into
base: main
Choose a base branch
from
Open

[Azure frontdoor] Add Health Probe dataset #20016

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
160ba03
Bump the version
herrBez Jul 7, 2026
43c1e78
Add health probe dataset
herrBez Jul 7, 2026
8d58b09
Make sure to deal with lowercase n/a
herrBez Jul 7, 2026
b8898a1
Update PR number
herrBez Jul 7, 2026
ca61077
Replace latinism with plain english
herrBez Jul 7, 2026
9669a03
Apply suggestions from code review
herrBez Jul 8, 2026
3147929
Remove event.ingested from test-common-config.yml
herrBez Jul 8, 2026
1fd94fc
Add if condition to geoip processors as per review
herrBez Jul 8, 2026
c0439c5
Update README
herrBez Jul 8, 2026
fdcc472
Update README.md
herrBez Jul 8, 2026
a15ee51
Add support for IP:PORT format in originIP. Add test-case
herrBez Jul 8, 2026
409da34
Fix automatic check complaints
herrBez Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions packages/azure_frontdoor/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,17 @@ The Azure Frontdoor logs integration retrieves the following types of log data f

- **Access Logs**: Logs categorized as `FrontDoorAccessLog`.
- **Web Application Firewall (WAF) Logs**: Logs categorized as `FrontDoorWebApplicationFirewallLog`.
- **Health Probe Logs**: Logs categorized as `FrontDoorHealthProbeLog`.

Currently, the integration does not support **Activity Logs** or **Health Probe logs**.
Currently, the integration does not support **Activity Logs**.

## Data streams

This integration collects two types of data streams:
This integration collects three types of data streams:

- access log
- waf logs
- health probe logs

@vera-review-bot vera-review-bot Bot Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: high path: packages/azure_frontdoor/_dev/build/docs/README.md:24

The README now lists three data streams but the sentence above still says the integration collects "two types". Update the count to "three".

Details

This PR adds 'health probe logs' to the data streams list under the '## Data streams' section, so the list now contains three entries (access, waf, health probe). The introductory sentence immediately above the list still reads 'This integration collects two types of data streams:', which is now factually incorrect. Because the built docs/README.md is generated from this template, the same stale wording ships in the rendered README.

Recommendation:

Update the count word in the '## Data streams' section:

## Data streams

This integration collects three types of data streams:

- access log
- waf logs
- health probe logs

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.


## Requirements

Expand Down Expand Up @@ -141,3 +143,7 @@ Users can also use this in case of a Hybrid Cloud model, where one may define th
## WAF Logs

{{fields "waf"}}

## Health Probe Logs

{{fields "health_probe"}}
5 changes: 5 additions & 0 deletions packages/azure_frontdoor/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.4.0"
changes:
- description: Add Health Probe dataset
type: enhancement
link: https://github.com/elastic/integrations/pull/20016
- version: "2.3.0"
changes:
- description: Add client secret authentication support for Azure Event Hub with RBAC.
Expand Down
4 changes: 4 additions & 0 deletions packages/azure_frontdoor/data_stream/health_probe/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
fields:
tags:
- preserve_original_event
- azure-frontdoor-health-probe
3 changes: 3 additions & 0 deletions packages/azure_frontdoor/data_stream/health_probe/_dev/test/pipeline/test-health-probe.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"time":"2026-07-07T09:31:41Z","resourceId":"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD","category":"FrontDoorHealthProbeLog","operationName":"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write","properties":{"healthProbeId":"259ffc3865154b8d8ca2c82692b60cf1","pop":"BL","httpVerb":"HEAD","result":"OriginError","httpStatusCode":"404","probeUrl":"http://test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io:80/","originName":"test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io","originIP":"216.160.83.56","totalLatencyMilliseconds":"142","connectionLatencyMilliseconds":"71","DNSLatencyMicroseconds":"2000"}}
{"time":"2026-07-07T10:00:00Z","resourceId":"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD","category":"FrontDoorHealthProbeLog","operationName":"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write","properties":{"healthProbeId":"abc123def456","pop":"AMS","httpVerb":"GET","result":"Success","httpStatusCode":"200","probeUrl":"https://backend.example.com/health","originName":"backend.example.com","originIP":"216.160.83.56:443","totalLatencyMilliseconds":"25","connectionLatencyMilliseconds":"10","DNSLatencyMicroseconds":"n/a"}}
{"time":"2026-07-07T10:00:00Z","resourceId":"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD","category":"FrontDoorHealthProbeLog","operationName":"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write","properties":{"healthProbeId":"abc123def456","pop":"AMS","httpVerb":"GET","result":"Success","httpStatusCode":"200","probeUrl":"https://backend.example.com/health","originName":"backend.example.com","originIP":"[2001:db8::1]:8080","totalLatencyMilliseconds":"25","connectionLatencyMilliseconds":"10","DNSLatencyMicroseconds":"n/a"}}
239 changes: 239 additions & 0 deletions ...frontdoor/data_stream/health_probe/_dev/test/pipeline/test-health-probe.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
{
"expected": [
{
"@timestamp": "2026-07-07T09:31:41.000Z",
"azure": {
"frontdoor": {
"category": "FrontDoorHealthProbeLog",
"health_probe": {
"connection_latency_milliseconds": 71,
"dns_latency_microseconds": 2000,
"health_probe_id": "259ffc3865154b8d8ca2c82692b60cf1",
"origin_name": "test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io",
"pop": "BL",
"result": "OriginError",
"total_latency_milliseconds": 142
},
"operation_name": "Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write",
"resource_id": "/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD"
}
},
"cloud": {
"provider": "azure"
},
"destination": {
"address": "216.160.83.56",
"as": {
"number": 209
},
"geo": {
"city_name": "Milton",
"continent_name": "North America",
"country_iso_code": "US",
"country_name": "United States",
"location": {
"lat": 47.2513,
"lon": -122.3149
},
"region_iso_code": "US-WA",
"region_name": "Washington"
},
"ip": "216.160.83.56"
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"network"
],
"original": "{\"time\":\"2026-07-07T09:31:41Z\",\"resourceId\":\"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD\",\"category\":\"FrontDoorHealthProbeLog\",\"operationName\":\"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write\",\"properties\":{\"healthProbeId\":\"259ffc3865154b8d8ca2c82692b60cf1\",\"pop\":\"BL\",\"httpVerb\":\"HEAD\",\"result\":\"OriginError\",\"httpStatusCode\":\"404\",\"probeUrl\":\"http://test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io:80/\",\"originName\":\"test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io\",\"originIP\":\"216.160.83.56\",\"totalLatencyMilliseconds\":\"142\",\"connectionLatencyMilliseconds\":\"71\",\"DNSLatencyMicroseconds\":\"2000\"}}",
"type": [
"connection"
]
},
"http": {
"request": {
"method": "HEAD"
},
"response": {
"status_code": 404
}
},
"related": {
"ip": [
"216.160.83.56"
]
},
"tags": [
"preserve_original_event",
"azure-frontdoor-health-probe"
],
"url": {
"domain": "test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io",
"full": "http://test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io:80/",
"original": "http://test.internal.orangemushroom-b8de5ef7.westus2.azurecontainerapps.io:80/",
"path": "/",
"port": 80,
"scheme": "http"
}
},
{
"@timestamp": "2026-07-07T10:00:00.000Z",
"azure": {
"frontdoor": {
"category": "FrontDoorHealthProbeLog",
"health_probe": {
"connection_latency_milliseconds": 10,
"health_probe_id": "abc123def456",
"origin_name": "backend.example.com",
"pop": "AMS",
"result": "Success",
"total_latency_milliseconds": 25
},
"operation_name": "Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write",
"resource_id": "/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD"
}
},
"cloud": {
"provider": "azure"
},
"destination": {
"address": "216.160.83.56:443",
"as": {
"number": 209
},
"geo": {
"city_name": "Milton",
"continent_name": "North America",
"country_iso_code": "US",
"country_name": "United States",
"location": {
"lat": 47.2513,
"lon": -122.3149
},
"region_iso_code": "US-WA",
"region_name": "Washington"
},
"ip": "216.160.83.56",
"port": 443
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"network"
],
"original": "{\"time\":\"2026-07-07T10:00:00Z\",\"resourceId\":\"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD\",\"category\":\"FrontDoorHealthProbeLog\",\"operationName\":\"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write\",\"properties\":{\"healthProbeId\":\"abc123def456\",\"pop\":\"AMS\",\"httpVerb\":\"GET\",\"result\":\"Success\",\"httpStatusCode\":\"200\",\"probeUrl\":\"https://backend.example.com/health\",\"originName\":\"backend.example.com\",\"originIP\":\"216.160.83.56:443\",\"totalLatencyMilliseconds\":\"25\",\"connectionLatencyMilliseconds\":\"10\",\"DNSLatencyMicroseconds\":\"n/a\"}}",
"type": [
"connection"
]
},
"http": {
"request": {
"method": "GET"
},
"response": {
"status_code": 200
}
},
"related": {
"ip": [
"216.160.83.56"
]
},
"tags": [
"preserve_original_event",
"azure-frontdoor-health-probe"
],
"url": {
"domain": "backend.example.com",
"full": "https://backend.example.com/health",
"original": "https://backend.example.com/health",
"path": "/health",
"scheme": "https"
}
},
{
"@timestamp": "2026-07-07T10:00:00.000Z",
"azure": {
"frontdoor": {
"category": "FrontDoorHealthProbeLog",
"health_probe": {
"connection_latency_milliseconds": 10,
"health_probe_id": "abc123def456",
"origin_name": "backend.example.com",
"pop": "AMS",
"result": "Success",
"total_latency_milliseconds": 25
},
"operation_name": "Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write",
"resource_id": "/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD"
}
},
"cloud": {
"provider": "azure"
},
"destination": {
"address": "[2001:db8::1]:8080",
"as": {
"number": 65551,
"organization": {
"name": "Documentation ASN"
}
},
"geo": {
"city_name": "Greenwich",
"continent_name": "Europe",
"country_iso_code": "GB",
"country_name": "United Kingdom",
"location": {
"lat": 51.47687,
"lon": -4.1E-4
},
"region_iso_code": "GB-ENG",
"region_name": "England"
},
"ip": "2001:db8::1",
"port": 8080
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"network"
],
"original": "{\"time\":\"2026-07-07T10:00:00Z\",\"resourceId\":\"/SUBSCRIPTIONS/49D1B571-1CBE-402D-B523-AFEE3C19B64E/RESOURCEGROUPS/WAF-RG/PROVIDERS/MICROSOFT.CDN/PROFILES/TESTCLOUD\",\"category\":\"FrontDoorHealthProbeLog\",\"operationName\":\"Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write\",\"properties\":{\"healthProbeId\":\"abc123def456\",\"pop\":\"AMS\",\"httpVerb\":\"GET\",\"result\":\"Success\",\"httpStatusCode\":\"200\",\"probeUrl\":\"https://backend.example.com/health\",\"originName\":\"backend.example.com\",\"originIP\":\"[2001:db8::1]:8080\",\"totalLatencyMilliseconds\":\"25\",\"connectionLatencyMilliseconds\":\"10\",\"DNSLatencyMicroseconds\":\"n/a\"}}",
"type": [
"connection"
]
},
"http": {
"request": {
"method": "GET"
},
"response": {
"status_code": 200
}
},
"related": {
"ip": [
"2001:db8::1"
]
},
"tags": [
"preserve_original_event",
"azure-frontdoor-health-probe"
],
"url": {
"domain": "backend.example.com",
"full": "https://backend.example.com/health",
"original": "https://backend.example.com/health",
"path": "/health",
"scheme": "https"
}
}
]
}
59 changes: 59 additions & 0 deletions packages/azure_frontdoor/data_stream/health_probe/agent/stream/azure-eventhub.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
{{#if auth_type}}
auth_type: {{auth_type}}
{{/if}}
{{#if connection_string}}
connection_string: {{connection_string}}
{{/if}}
{{#if eventhub_namespace}}
eventhub_namespace: {{eventhub_namespace}}
{{/if}}
{{#if tenant_id}}
tenant_id: {{tenant_id}}
{{/if}}
{{#if client_id}}
client_id: {{client_id}}
{{/if}}
{{#if client_secret}}
client_secret: {{client_secret}}
{{/if}}
{{#contains "client_secret" auth_type}}
{{#if authority_host}}
authority_host: {{authority_host}}
{{/if}}
{{/contains}}
{{#if storage_account_container }}
storage_account_container: {{storage_account_container}}
{{else}}
{{#if eventhub}}
storage_account_container: frontdoor-health-probe-{{eventhub}}
{{/if}}
{{/if}}
{{#if eventhub}}
eventhub: {{eventhub}}
{{/if}}
{{#if consumer_group}}
consumer_group: {{consumer_group}}
{{/if}}
{{#if storage_account}}
storage_account: {{storage_account}}
{{/if}}
{{#if storage_account_key}}
storage_account_key: {{storage_account_key}}
{{/if}}
{{#if resource_manager_endpoint}}
resource_manager_endpoint: {{resource_manager_endpoint}}
{{/if}}
tags:
{{#if preserve_original_event}}
- preserve_original_event
{{/if}}
{{#each tags as |tag|}}
- {{tag}}
{{/each}}
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
{{#if processors}}
processors:
{{processors}}
{{/if}}
Loading
Loading