diff --git a/packages/awsfirehose/_dev/build/docs/README.md b/packages/awsfirehose/_dev/build/docs/README.md index 2ec567eafe1..12c1d49b8d5 100644 --- a/packages/awsfirehose/_dev/build/docs/README.md +++ b/packages/awsfirehose/_dev/build/docs/README.md @@ -50,8 +50,8 @@ This is a current limitation in Firehose, which we are working with AWS to resol ## Instructions 1. Install the relevant integrations in Kibana - In order to make the most of your data, install AWS integrations to load index templates, ingest pipelines, and - dashboards into Kibana. In Kibana, navigate to **Management** > **Integrations** in the sidebar. + In order to make the most out of your data, first install the **AWS integration** to load index templates, ingest pipelines, and + dashboards into Kibana. For this, in Kibana, navigate to **Management** > **Integrations** in the sidebar. Find the **AWS** integration by searching or browsing the catalog. ![AWS integration](../img/aws.png) @@ -69,7 +69,7 @@ This is a current limitation in Firehose, which we are working with AWS to resol **Choose source and destination** - Unless you are streaming data from Kinesis Data Streams, set source to Direct PUT (see Setup guide for more details on data sources). + Unless you are streaming data from Kinesis Data Streams, set source to **Direct PUT** (see Setup guide for more details on data sources). Set destination to **Elastic**. @@ -81,44 +81,40 @@ This is a current limitation in Firehose, which we are working with AWS to resol **Destination settings** - 1. Set **Elastic endpoint URL** to point to your Elasticsearch cluster running in Elastic Cloud. - This endpoint can be found in the Elastic Cloud console. An example is https://my-deployment-28u274.es.eu-west-1.aws.found.io. + 1. Set **Elastic endpoint URL** to point to your **Elasticsearch** cluster running in Elastic Cloud. + This endpoint can be found from the [Elastic Cloud console](https://cloud.elastic.co/). - 2. **API key** should be a Base64 encoded Elastic API key, which can be created in Kibana by following the - instructions under API Keys. If you are using an API key with “Restricted privileges”, be sure to review the Indices - privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this - delivery stream. By default, logs will be stored in `logs-awsfirehose-default` index and metrics will be stored in - `metrics-aws.cloudwatch-default` index. Therefore, Elastic highly recommends giving `logs-awsfirehose-default` and - `metrics-aws.cloudwatch-default` indices with "write" privilege. + 2. **API key** is the base64 encoded Elastic API key. + This can be created in Kibana by following the instructions under API Keys. + If you are using an API key with Restricted privileges, be sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream. + By default, logs will be stored in `logs-awsfirehose-default` data stream and metrics will be stored in `metrics-aws.cloudwatch-default` data stream. + Therefore, Elastic highly recommends giving `logs-awsfirehose-default` and `metrics-aws.cloudwatch-default` indices with "write" privilege. - 3. We recommend leaving **Content encoding** set to **GZIP** for improved network efficiency. + 3. We recommend setting **Content encoding** to **GZIP** to reduce the data transfer costs. - 4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error. - A duration of 60-300s should be suitable for most use cases. + 4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error. + A duration between 60 and 300 seconds should be suitable for most use cases. - 5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length` - setting (typically 100MB) when the buffer is uncompressed. + 5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length` setting (typically 100MB) when the buffer is uncompressed. 6. The default **Buffer interval** of `60s` is recommended to ensure data freshness in Elastic. 7. **Parameters** - 1. Elastic recommends only setting the `es_datastream_name` parameter when ingesting logs that are not supported - by this Firehose integration. If this parameter is not specified, data is sent to the `logs-awsfirehose-default` - index by default and the routing rules defined in this integration will be applied automatically. - Please make sure the index specified with this `es_datastream_name` parameter has the proper permission given by - the API key. - ![Firehose Destination Settings](../img/destination-settings.png) - - 2. The **include_cw_extracted_fields** parameter is optional and can be set when using a CloudWatch logs subscription - filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the - subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly - increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when - the extracted fields are required for specific filtering and/or aggregation. - - 3. The **include_event_original** field is optional and should only be used for debugging purposes. When set to `true`, each - log record will contain an additional field named `event.original`, which contains the raw (unprocessed) log message. - This parameter will increase the data volume in Elasticsearch and should be used with care. + 1. `es_datastream_name`: Accepts a data stream name of your choice. + You can configure this when you ingest data formats that are not officially supported and managed through your customized data streams. + Also, you may use this to separate your ingested data from default indexes. + By default, logs will be stored in indexes of `logs-awsfirehose-default` (index template `logs-awsfirehose`) data stream and metrics will be stored in indexes of `metrics-aws.cloudwatch-default` (index template `metrics-aws.cloudwatch`) data stream. + When configuring, please make sure configured **API key** has required permissions to ingest data into the configured data stream. + + 2. `include_cw_extracted_fields`: Optional parameter that can be set when using a CloudWatch logs subscription filter as the Firehose data source. + When set to true, extracted fields generated by the filter pattern in the subscription filter will be collected. + Setting this parameter can add many fields into each record and may significantly increase data volume in Elasticsearch. + As such, use of this parameter should be carefully considered and used only when the extracted fields are required for specific filtering and/or aggregation. + + 8. **Backup settings** It is recommended to configure S3 backup for failed records. These backups can be used to restore lost data caused by unforeseen service outages. + + ![Firehose Destination Settings](../img/destination-settings.png) 3. Send data to the Firehose delivery stream 1. logs diff --git a/packages/awsfirehose/changelog.yml b/packages/awsfirehose/changelog.yml index 8f764dd6c7d..c34324714a6 100644 --- a/packages/awsfirehose/changelog.yml +++ b/packages/awsfirehose/changelog.yml @@ -1,7 +1,12 @@ # newer versions go on top +- version: "1.5.1" + changes: + - description: Improve documentation. + type: enhancement + link: https://github.com/elastic/integrations/pull/12732 - version: "1.5.0" changes: - - description: Add support for Kibana `9.0.0` + - description: Add support for Kibana `9.0.0` type: enhancement link: https://github.com/elastic/integrations/pull/12641 - version: "1.4.0" diff --git a/packages/awsfirehose/docs/README.md b/packages/awsfirehose/docs/README.md index 61c86bed7fe..25920613133 100644 --- a/packages/awsfirehose/docs/README.md +++ b/packages/awsfirehose/docs/README.md @@ -50,8 +50,8 @@ This is a current limitation in Firehose, which we are working with AWS to resol ## Instructions 1. Install the relevant integrations in Kibana - In order to make the most of your data, install AWS integrations to load index templates, ingest pipelines, and - dashboards into Kibana. In Kibana, navigate to **Management** > **Integrations** in the sidebar. + In order to make the most out of your data, first install the **AWS integration** to load index templates, ingest pipelines, and + dashboards into Kibana. For this, in Kibana, navigate to **Management** > **Integrations** in the sidebar. Find the **AWS** integration by searching or browsing the catalog. ![AWS integration](../img/aws.png) @@ -69,7 +69,7 @@ This is a current limitation in Firehose, which we are working with AWS to resol **Choose source and destination** - Unless you are streaming data from Kinesis Data Streams, set source to Direct PUT (see Setup guide for more details on data sources). + Unless you are streaming data from Kinesis Data Streams, set source to **Direct PUT** (see Setup guide for more details on data sources). Set destination to **Elastic**. @@ -81,44 +81,40 @@ This is a current limitation in Firehose, which we are working with AWS to resol **Destination settings** - 1. Set **Elastic endpoint URL** to point to your Elasticsearch cluster running in Elastic Cloud. - This endpoint can be found in the Elastic Cloud console. An example is https://my-deployment-28u274.es.eu-west-1.aws.found.io. + 1. Set **Elastic endpoint URL** to point to your **Elasticsearch** cluster running in Elastic Cloud. + This endpoint can be found from the [Elastic Cloud console](https://cloud.elastic.co/). - 2. **API key** should be a Base64 encoded Elastic API key, which can be created in Kibana by following the - instructions under API Keys. If you are using an API key with “Restricted privileges”, be sure to review the Indices - privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this - delivery stream. By default, logs will be stored in `logs-awsfirehose-default` index and metrics will be stored in - `metrics-aws.cloudwatch-default` index. Therefore, Elastic highly recommends giving `logs-awsfirehose-default` and - `metrics-aws.cloudwatch-default` indices with "write" privilege. + 2. **API key** is the base64 encoded Elastic API key. + This can be created in Kibana by following the instructions under API Keys. + If you are using an API key with Restricted privileges, be sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream. + By default, logs will be stored in `logs-awsfirehose-default` data stream and metrics will be stored in `metrics-aws.cloudwatch-default` data stream. + Therefore, Elastic highly recommends giving `logs-awsfirehose-default` and `metrics-aws.cloudwatch-default` indices with "write" privilege. - 3. We recommend leaving **Content encoding** set to **GZIP** for improved network efficiency. + 3. We recommend setting **Content encoding** to **GZIP** to reduce the data transfer costs. - 4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error. - A duration of 60-300s should be suitable for most use cases. + 4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error. + A duration between 60 and 300 seconds should be suitable for most use cases. - 5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length` - setting (typically 100MB) when the buffer is uncompressed. + 5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length` setting (typically 100MB) when the buffer is uncompressed. 6. The default **Buffer interval** of `60s` is recommended to ensure data freshness in Elastic. 7. **Parameters** - 1. Elastic recommends only setting the `es_datastream_name` parameter when ingesting logs that are not supported - by this Firehose integration. If this parameter is not specified, data is sent to the `logs-awsfirehose-default` - index by default and the routing rules defined in this integration will be applied automatically. - Please make sure the index specified with this `es_datastream_name` parameter has the proper permission given by - the API key. - ![Firehose Destination Settings](../img/destination-settings.png) - - 2. The **include_cw_extracted_fields** parameter is optional and can be set when using a CloudWatch logs subscription - filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the - subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly - increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when - the extracted fields are required for specific filtering and/or aggregation. - - 3. The **include_event_original** field is optional and should only be used for debugging purposes. When set to `true`, each - log record will contain an additional field named `event.original`, which contains the raw (unprocessed) log message. - This parameter will increase the data volume in Elasticsearch and should be used with care. + 1. `es_datastream_name`: Accepts a data stream name of your choice. + You can configure this when you ingest data formats that are not officially supported and managed through your customized data streams. + Also, you may use this to separate your ingested data from default indexes. + By default, logs will be stored in indexes of `logs-awsfirehose-default` (index template `logs-awsfirehose`) data stream and metrics will be stored in indexes of `metrics-aws.cloudwatch-default` (index template `metrics-aws.cloudwatch`) data stream. + When configuring, please make sure configured **API key** has required permissions to ingest data into the configured data stream. + + 2. `include_cw_extracted_fields`: Optional parameter that can be set when using a CloudWatch logs subscription filter as the Firehose data source. + When set to true, extracted fields generated by the filter pattern in the subscription filter will be collected. + Setting this parameter can add many fields into each record and may significantly increase data volume in Elasticsearch. + As such, use of this parameter should be carefully considered and used only when the extracted fields are required for specific filtering and/or aggregation. + + 8. **Backup settings** It is recommended to configure S3 backup for failed records. These backups can be used to restore lost data caused by unforeseen service outages. + + ![Firehose Destination Settings](../img/destination-settings.png) 3. Send data to the Firehose delivery stream 1. logs diff --git a/packages/awsfirehose/img/aws-firehose.png b/packages/awsfirehose/img/aws-firehose.png index 66171a6c6db..609e2980aa2 100644 Binary files a/packages/awsfirehose/img/aws-firehose.png and b/packages/awsfirehose/img/aws-firehose.png differ diff --git a/packages/awsfirehose/img/aws.png b/packages/awsfirehose/img/aws.png index be98056181d..2cc5d4e22f5 100644 Binary files a/packages/awsfirehose/img/aws.png and b/packages/awsfirehose/img/aws.png differ diff --git a/packages/awsfirehose/img/destination-settings.png b/packages/awsfirehose/img/destination-settings.png index 1df6cff92cc..37865ddbec2 100644 Binary files a/packages/awsfirehose/img/destination-settings.png and b/packages/awsfirehose/img/destination-settings.png differ diff --git a/packages/awsfirehose/img/install-assets.png b/packages/awsfirehose/img/install-assets.png index af67e0a0003..142187753ce 100644 Binary files a/packages/awsfirehose/img/install-assets.png and b/packages/awsfirehose/img/install-assets.png differ diff --git a/packages/awsfirehose/img/source-destination.png b/packages/awsfirehose/img/source-destination.png index e52705d91c8..9a5a9d3cf6d 100644 Binary files a/packages/awsfirehose/img/source-destination.png and b/packages/awsfirehose/img/source-destination.png differ diff --git a/packages/awsfirehose/manifest.yml b/packages/awsfirehose/manifest.yml index 0010793cb81..baa2d824db9 100644 --- a/packages/awsfirehose/manifest.yml +++ b/packages/awsfirehose/manifest.yml @@ -1,7 +1,7 @@ format_version: "3.3.0" name: awsfirehose title: Amazon Data Firehose -version: 1.5.0 +version: 1.5.1 description: Stream logs and metrics from Amazon Data Firehose into Elastic Cloud. type: integration categories: