From a0fde58210729e62b19d0ff3e71c9a3b463701c2 Mon Sep 17 00:00:00 2001 From: Mike Birnstiehl <114418652+mdbirnstiehl@users.noreply.github.com> Date: Fri, 18 Oct 2024 15:49:48 -0500 Subject: [PATCH 1/2] Add docs for new central log source setting (#4384) * update docs for new centra log source setting * update ML page (cherry picked from commit 0fb7cdd02d8af45110a4c8ed90504c2607a792df) # Conflicts: # docs/en/serverless/logging/view-and-monitor-logs.mdx --- .../en/observability/categorize-logs.asciidoc | 33 ++++--- .../configure-logs-sources.asciidoc | 37 +++----- docs/en/observability/explore-logs.asciidoc | 4 +- .../inspect-log-anomalies.asciidoc | 2 +- .../logging/view-and-monitor-logs.mdx | 90 +++++++++++++++++++ 5 files changed, 122 insertions(+), 44 deletions(-) create mode 100644 docs/en/serverless/logging/view-and-monitor-logs.mdx diff --git a/docs/en/observability/categorize-logs.asciidoc b/docs/en/observability/categorize-logs.asciidoc index 9e13aef5b8..4162f3d128 100644 --- a/docs/en/observability/categorize-logs.asciidoc +++ b/docs/en/observability/categorize-logs.asciidoc @@ -6,8 +6,8 @@ log messages are the same or very similar, so classifying them can reduce millions of log lines into just a few categories. Within the {logs-app}, the *Categories* page enables you to identify patterns in -your log events quickly. Instead of manually identifying similar logs, the logs -categorization view lists log events that have been grouped based on their +your log events quickly. Instead of manually identifying similar logs, the logs +categorization view lists log events that have been grouped based on their messages and formats so that you can take action quicker. NOTE: This feature makes use of {ml} {anomaly-jobs}. To set up jobs, you must @@ -25,47 +25,44 @@ more details, refer to {ml-docs}/setup.html[Set up {ml-features}]. Create a {ml} job to categorize log messages automatically. {ml-cap} observes the static parts of the message, clusters similar messages, classifies them into -message categories, and detects unusually high message counts in the categories. - -[role="screenshot"] -image::images/log-create-categorization-job.jpg[Configure log categorization job] +message categories, and detects unusually high message counts in the categories. // lint ignore ml -1. Select *Categories*, and you are prompted to use {ml} to create +1. Select *Categories*, and you are prompted to use {ml} to create log rate categorizations. -2. Choose a time range for the {ml} analysis. By default, the {ml} job analyzes +2. Choose a time range for the {ml} analysis. By default, the {ml} job analyzes log messages no older than four weeks and continues indefinitely. -3. Add the indices that contain the logs you want to examine. -4. Click *Create ML job*. The job is created, and it starts to run. It takes a few - minutes for the {ml} robots to collect the necessary data. After the job +3. Add the indices that contain the logs you want to examine. By default, Machine Learning analyzes messages in all log indices that match the patterns set in the *logs source* advanced setting. Update this setting by going to *Management* → *Advanced Settings* and searching for _logs source_. +4. Click *Create ML job*. The job is created, and it starts to run. It takes a few + minutes for the {ml} robots to collect the necessary data. After the job processed the data, you can view the results. [discrete] [[analyze-log-categories]] == Analyze log categories -The *Categories* page lists all the log categories from the selected indices. -You can filter the categories by indices. The screenshot below shows the +The *Categories* page lists all the log categories from the selected indices. +You can filter the categories by indices. The screenshot below shows the categories from the `elastic.agent` log. [role="screenshot"] image::images/log-categories.jpg[Log categories] -The category row contains the following information: +The category row contains the following information: * message count: shows how many messages belong to the given category. * trend: indicates how the occurrence of the messages changes in time. -* category name: it is the name of the category and is derived from the message +* category name: it is the name of the category and is derived from the message text. * datasets: the name of the datasets where the categories are present. * maximum anomaly score: the highest anomaly score in the category. -To view a log message under a particular category, click -the arrow at the end of the row. To further examine a message, it +To view a log message under a particular category, click +the arrow at the end of the row. To further examine a message, it can be viewed in the corresponding log event on the *Stream* page or displayed in its context. [role="screenshot"] image::images/log-opened.png[Opened log category] For more information about categorization, go to -{ml-docs}/ml-configuring-categories.html[Detecting anomalous categories of data]. \ No newline at end of file +{ml-docs}/ml-configuring-categories.html[Detecting anomalous categories of data]. \ No newline at end of file diff --git a/docs/en/observability/configure-logs-sources.asciidoc b/docs/en/observability/configure-logs-sources.asciidoc index 0e3a712fad..84ca27ac39 100644 --- a/docs/en/observability/configure-logs-sources.asciidoc +++ b/docs/en/observability/configure-logs-sources.asciidoc @@ -4,9 +4,8 @@ Specify the source configuration for logs in the {kibana-ref}/logs-ui-settings-kb.html[{logs-app} settings] in the {kibana-ref}/settings.html[{kib} configuration file]. -By default, the configuration uses the `filebeat-*` index pattern to query the data. -The configuration also defines field settings for things like timestamps -and container names, and the default columns displayed in the logs stream. +By default, the configuration uses the index patterns stored in the {kib} log sources advanced setting to query the data. +The configuration also defines the default columns displayed in the logs stream. If your logs have custom index patterns, use non-default field settings, or contain parsed fields that you want to expose as individual columns, you can override the @@ -20,32 +19,22 @@ default configuration settings. + . Click *Settings*. + -|=== +|=== -| *Name* | Name of the source configuration. +| *Name* | Name of the source configuration. -| *{ipm-cap}* | {kib} index patterns or index name patterns in the {es} indices -to read log data from. - -Each log source now integrates with {kib} index patterns which support creating and -querying {kibana-ref}/managing-data-views.html[runtime fields]. You can continue -to use log sources configured to use an index name pattern, such as `filebeat-*`, -instead of a {kib} index pattern. However, some features like those depending on -runtime fields may not be available. +| *{kib} log sources advanced setting* | Use index patterns stored in the {kib} *log sources* advanced setting, which provides a centralized place to store and query log index patterns. +Update this setting by going to *Stack Management* → *Advanced Settings* and searching for _logs sources_. -Instead of entering an index pattern name, -click *Use {kib} index patterns* and select the `filebeat-*` log index pattern. - -| *{data-source-cap}* | This is a new configuration option that can be used -instead of index pattern. The Logs UI can now integrate with {data-sources} to +| *{data-source-cap} (deprecated)* | The Logs UI integrates with {data-sources} to configure the used indices by clicking *Use {data-sources}*. -| *Fields* | Configuring fields input has been deprecated. You should adjust your indexing using the -<>, which use the {ecs-ref}/index.html[Elastic Common Schema (ECS) specification]. +| *Log indices (deprecated)* | {kib} index patterns or index name patterns in the {es} indices +to read log data from. | *Log columns* | Columns that are displayed in the logs *Stream* page. -|=== +|=== + . When you have completed your changes, click *Apply*. @@ -63,16 +52,16 @@ with other data source configurations. By default, the *Stream* page within the {logs-app} displays the following columns. -|=== +|=== -| *Timestamp* | The timestamp of the log entry from the `timestamp` field. +| *Timestamp* | The timestamp of the log entry from the `timestamp` field. | *Message* | The message extracted from the document. The content of this field depends on the type of log message. If no special log message type is detected, the {ecs-ref}/ecs-base.html[Elastic Common Schema (ECS)] base field, `message`, is used. -|=== +|=== 1. To add a new column to the logs stream, select *Settings > Add column*. 2. In the list of available fields, select the field you want to add. diff --git a/docs/en/observability/explore-logs.asciidoc b/docs/en/observability/explore-logs.asciidoc index a8522e3e8a..f2ffdaa7cb 100644 --- a/docs/en/observability/explore-logs.asciidoc +++ b/docs/en/observability/explore-logs.asciidoc @@ -22,7 +22,9 @@ Viewing data in Logs Explorer requires `read` privileges for *Discover* and *Int [[find-your-logs]] == Find your logs -By default, Logs Explorer shows all of your logs. +By default, Logs Explorer shows all of your logs, according to the index patterns set in the *logs source* advanced setting. +Update this setting by going to *Stack Management* → *Advanced Settings* and searching for __. + If you need to focus on logs from a specific integration, select the integration from the logs menu: [role="screenshot"] diff --git a/docs/en/observability/inspect-log-anomalies.asciidoc b/docs/en/observability/inspect-log-anomalies.asciidoc index 8790f716bc..985b586086 100644 --- a/docs/en/observability/inspect-log-anomalies.asciidoc +++ b/docs/en/observability/inspect-log-anomalies.asciidoc @@ -35,7 +35,7 @@ Create a {ml} job to detect anomalous log entry rates automatically. 1. Select *Anomalies*, and you'll be prompted to create a {ml} job which will carry out the log rate analysis. 2. Choose a time range for the {ml} analysis. -3. Add the Indices that contain the logs you want to analyze. +3. Add the indices that contain the logs you want to examine. By default, Machine Learning analyzes messages in all log indices that match the patterns set in the *logs source* advanced setting. Update this setting by going to *Management* → *Advanced Settings* and searching for _logs source_. 4. Click *Create {ml-init} job*. 5. You're now ready to explore your log partitions. diff --git a/docs/en/serverless/logging/view-and-monitor-logs.mdx b/docs/en/serverless/logging/view-and-monitor-logs.mdx new file mode 100644 index 0000000000..a1e1e68c4a --- /dev/null +++ b/docs/en/serverless/logging/view-and-monitor-logs.mdx @@ -0,0 +1,90 @@ +--- +slug: /serverless/observability/discover-and-explore-logs +title: Explore logs +description: Visualize and analyze logs. +tags: [ 'serverless', 'observability', 'how-to' ] +--- + +

+ +With **Logs Explorer**, based on Discover, you can quickly search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. +You can also customize and save your searches and place them on a dashboard. +Instead of having to log into different servers, change directories, and view individual files, all your logs are available in a single view. + +Go to Logs Explorer by opening **Discover** from the navigation menu, and selecting the **Logs Explorer** tab. + +![Screen capture of the Logs Explorer](../images/log-explorer.png) + +## Required ((kib)) privileges + +Viewing data in Logs Explorer requires `read` privileges for **Discover** and **Integrations**. +For more on assigning Kibana privileges, refer to the [((kib)) privileges](((kibana-ref))/kibana-privileges.html) docs. + +## Find your logs + +By default, Logs Explorer shows all of your logs according to the index patterns set in the **logs source** advanced setting. +Update this setting by going to *Management* → *Advanced Settings* and searching for _logs source_. + +If you need to focus on logs from a specific integrations, select the integration from the logs menu: + + + +Once you have the logs you want to focus on displayed, you can drill down further to find the information you need. +For more on filtering your data in Logs Explorer, refer to Filter logs in Logs Explorer. + +## Review log data in the documents table + +The documents table in Logs Explorer functions similarly to the table in Discover. +You can add fields, order table columns, sort fields, and update the row height in the same way you would in Discover. + +Refer to the [Discover](((kibana-ref))/discover.html) documentation for more information on updating the table. + +### Analyze data with smart fields + +Smart fields are dynamic fields that provide valuable insight on where your log documents come from, what information they contain, and how you can interact with them. +The following sections detail the smart fields available in Logs Explorer. + +#### Resource smart field + +The resource smart field shows where your logs are coming from by displaying fields like `service.name`, `container.name`, `orchestrator.namespace`, `host.name`, and `cloud.instance.id`. +Use this information to see where issues are coming from and if issues are coming from the same source. + +#### Content smart field + +The content smart field shows your logs' `log.level` and `message` fields. +If neither of these fields are available, the content smart field will show the `error.message` or `event.original` field. +Use this information to see your log content and inspect issues. + +#### Actions smart field + +The actions smart field provides access to additional information about your logs. + +**Expand:** () Open the log details to get an in-depth look at an individual log file. + +**Degraded document indicator:** () Shows if any of the document's fields were ignored when it was indexed. +Ignored fields could indicate malformed fields or other issues with your document. Use this information to investigate and determine why fields are being ignored. + +**Stacktrace indicator:** () Shows if the document contains stack traces. +This indicator makes it easier to navigate through your documents and know if they contain additional information in the form of stack traces. + +## View log details + +Click the expand icon () in the **Actions** column to get an in-depth look at an individual log file. + +These details provide immediate feedback and context for what's happening and where it's happening for each log. +From here, you can quickly debug errors and investigate the services where errors have occurred. + +The following actions help you filter and focus on specific fields in the log details: + +* **Filter for value ():** Show logs that contain the specific field value. +* **Filter out value ():** Show logs that do _not_ contain the specific field value. +* **Filter for field present ():** Show logs that contain the specific field. +* **Toggle column in table ():** Add or remove a column for the field to the main Logs Explorer table. + +## View log quality issues + +From the log details of a document with ignored fields, as shown by the degraded document indicator (()), expand the **Quality issues** section to see the name and value of the fields that were ignored. +Select **Data set details** to open the **Data Set Quality** page. Here you can monitor your data sets and investigate any issues. + +The **Data Set Details** page is also accessible from **Project settings** → **Management** → **Data Set Quality**. +Refer to Monitor data sets for more information. \ No newline at end of file From f61f5f07ad18a71901215fd434722d9f12837e30 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Fri, 18 Oct 2024 20:50:52 +0000 Subject: [PATCH 2/2] Delete docs/en/serverless directory --- .../logging/view-and-monitor-logs.mdx | 90 ------------------- 1 file changed, 90 deletions(-) delete mode 100644 docs/en/serverless/logging/view-and-monitor-logs.mdx diff --git a/docs/en/serverless/logging/view-and-monitor-logs.mdx b/docs/en/serverless/logging/view-and-monitor-logs.mdx deleted file mode 100644 index a1e1e68c4a..0000000000 --- a/docs/en/serverless/logging/view-and-monitor-logs.mdx +++ /dev/null @@ -1,90 +0,0 @@ ---- -slug: /serverless/observability/discover-and-explore-logs -title: Explore logs -description: Visualize and analyze logs. -tags: [ 'serverless', 'observability', 'how-to' ] ---- - -

- -With **Logs Explorer**, based on Discover, you can quickly search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. -You can also customize and save your searches and place them on a dashboard. -Instead of having to log into different servers, change directories, and view individual files, all your logs are available in a single view. - -Go to Logs Explorer by opening **Discover** from the navigation menu, and selecting the **Logs Explorer** tab. - -![Screen capture of the Logs Explorer](../images/log-explorer.png) - -## Required ((kib)) privileges - -Viewing data in Logs Explorer requires `read` privileges for **Discover** and **Integrations**. -For more on assigning Kibana privileges, refer to the [((kib)) privileges](((kibana-ref))/kibana-privileges.html) docs. - -## Find your logs - -By default, Logs Explorer shows all of your logs according to the index patterns set in the **logs source** advanced setting. -Update this setting by going to *Management* → *Advanced Settings* and searching for _logs source_. - -If you need to focus on logs from a specific integrations, select the integration from the logs menu: - - - -Once you have the logs you want to focus on displayed, you can drill down further to find the information you need. -For more on filtering your data in Logs Explorer, refer to Filter logs in Logs Explorer. - -## Review log data in the documents table - -The documents table in Logs Explorer functions similarly to the table in Discover. -You can add fields, order table columns, sort fields, and update the row height in the same way you would in Discover. - -Refer to the [Discover](((kibana-ref))/discover.html) documentation for more information on updating the table. - -### Analyze data with smart fields - -Smart fields are dynamic fields that provide valuable insight on where your log documents come from, what information they contain, and how you can interact with them. -The following sections detail the smart fields available in Logs Explorer. - -#### Resource smart field - -The resource smart field shows where your logs are coming from by displaying fields like `service.name`, `container.name`, `orchestrator.namespace`, `host.name`, and `cloud.instance.id`. -Use this information to see where issues are coming from and if issues are coming from the same source. - -#### Content smart field - -The content smart field shows your logs' `log.level` and `message` fields. -If neither of these fields are available, the content smart field will show the `error.message` or `event.original` field. -Use this information to see your log content and inspect issues. - -#### Actions smart field - -The actions smart field provides access to additional information about your logs. - -**Expand:** () Open the log details to get an in-depth look at an individual log file. - -**Degraded document indicator:** () Shows if any of the document's fields were ignored when it was indexed. -Ignored fields could indicate malformed fields or other issues with your document. Use this information to investigate and determine why fields are being ignored. - -**Stacktrace indicator:** () Shows if the document contains stack traces. -This indicator makes it easier to navigate through your documents and know if they contain additional information in the form of stack traces. - -## View log details - -Click the expand icon () in the **Actions** column to get an in-depth look at an individual log file. - -These details provide immediate feedback and context for what's happening and where it's happening for each log. -From here, you can quickly debug errors and investigate the services where errors have occurred. - -The following actions help you filter and focus on specific fields in the log details: - -* **Filter for value ():** Show logs that contain the specific field value. -* **Filter out value ():** Show logs that do _not_ contain the specific field value. -* **Filter for field present ():** Show logs that contain the specific field. -* **Toggle column in table ():** Add or remove a column for the field to the main Logs Explorer table. - -## View log quality issues - -From the log details of a document with ignored fields, as shown by the degraded document indicator (()), expand the **Quality issues** section to see the name and value of the fields that were ignored. -Select **Data set details** to open the **Data Set Quality** page. Here you can monitor your data sets and investigate any issues. - -The **Data Set Details** page is also accessible from **Project settings** → **Management** → **Data Set Quality**. -Refer to Monitor data sets for more information. \ No newline at end of file