Merge pull request #876 from elementary-data/prod

Prod

Author: elongl

docs/_snippets/column-metrics.mdx

@@ -1,5 +1,7 @@
+**Default monitors by type:**
+
 | Data quality metric  | Column Type |
-|----------------------| ----------- |
+|----------------------|-------------|
 | `null_count`         | any         |
 | `null_percent`       | any         |
 | `min_length`         | string      |
@@ -13,4 +15,9 @@
 | `zero_percent`       | numeric     |
 | `standard_deviation` | numeric     |
 | `variance`           | numeric     |
+
+**Opt in monitors by type:**
+
+| Data quality metric  | Column Type |
+|----------------------|-------------|
 | `sum`                | numeric     |

docs/_snippets/faq/question-tests-configuration-priorities.mdx

@@ -0,0 +1,20 @@
+<Accordion title="How test configurations are prioritized?">
+
+The configuration of Elementary is dbt native and follows the same priorities of `dbt configuration`.
+The more granular and specific configuration overrides the less granular one. 
+
+Elementary searches and prioritizes configuration in the following order:
+
+For models:
+1. Test arguments.
+2. Model configuration.
+3. Global vars in `dbt_project.yml`.
+
+For sources:
+1. Test arguments.
+2. Table configuration.
+3. Source configuration.
+4. Global vars in `dbt_project.yml`.
+
+
+</Accordion>

docs/cloud/general/security-and-privacy.mdx

@@ -0,0 +1,53 @@
+---
+title: "Security and Privacy"
+sidebarTitle: "Security and privacy"
+---
+
+## Security principals
+
+Elementary Cloud is designed with the core principle of least privilege.
+Our cloud service does not require permissions to access the customer data.
+Therefor, we instruct our customers to create a dedicated role for Elementary with `read only` access only to the Elementary schema in your data warehouse.
+
+As long as you follow the onboarding process instructions, it will be impossible for Elementary Cloud to read data from your warehouse that does not reside in the Elementary schema.
+This ensures that Elementary cloud will not mistakenly access your data, and minimizes the risk in case of a data breach.
+Our product and architecture are always evolving, but our commitment to secure design always remains.
+
+
+## How it works
+
+1. You install the Elementary dbt package in your dbt project and configure it to write to it's own schema, the Elementary schema.
+2. The package writes test results, run results, logs and metadata to the Elementary schema.
+3. The cloud service only requires `read access` to the Elementary schema, not to schemas where your sensitive data is stored.
+4. The cloud service connects to sync the Elementary schema using an **encrypted connection** and a **static IP address** that you will need to add to your allowlist. 
+
+<Frame>
+    <img
+    src="https://res.cloudinary.com/diuctyblm/image/upload/v1682954617/security-architecture_mdmh8i.png"
+    alt="Elementary cloud security"
+    />
+</Frame>
+
+
+## What is stored in the Elementary schema?
+
+The Elementary schema stores only metadata, aggregated metrics and logs.
+You can find the details of the tables [here](/guides/modules-overview/dbt-package). 
+
+The only exception to that is the `test_results_samples` which can be disabled. This is a feature that shows a sample of a few raw failed rows for failed tests, to help them triage and understand the problem. 
+To avoid this sampling, set the var `test_sample_rows_count: 0` in your `dbt_project.yml` (default is 5 sample rows).
+
+
+## Secrets and data protection
+
+- **Tokens and credentials** - For customer secrets (tokens and credentials) we use [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html). Secrets Manager uses envelope encryption with AWS KMS keys and data keys to protect each secret value. Whenever the secret value in a secret changes, Secrets Manager generates a new data key to protect it. The data key is encrypted under a KMS key and stored in the metadata of the secret. [See this link for more details](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html).
+- **Customer data (Elementary schema replica)** - The synced customer data is encrypted at rest using server-side encryption (AES-256).
+
+## Compliance 
+
+[Contact us](mailto:legal@elementary-data.com) for auditing reports and penetration testing results. 
+
+## Have more questions?
+
+We would be happy to answer!
+Reach out to us on [email](mailto:legal@elementary-data.com) or [Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-1b9vogqmq-y~IRhc2396CbHNBXLsrXcA).

docs/guides/add-elementary-tests.mdx

@@ -2,8 +2,7 @@
 title: "Add anomaly detection tests"
 ---
 
-After you [install the dbt package](/quickstart#install-the-dbt-package), you can add Elementary data anomaly detection
-tests.
+After you [install the dbt package](/quickstart#install-the-dbt-package), you can add Elementary data anomaly detection tests.
 
 ## Data anomaly detection dbt tests
 
@@ -27,13 +26,13 @@ alt="Demo"
 Monitors the row count of your table over time per time bucket (if configured without `timestamp_column`, will count table total rows).
 
 Upon running the test, your data is split into time buckets (daily by default, configurable with the `time bucket`
-field), and then we compute the row count per bucket for the last `days_back` days (by default 14).
+field), and then we compute the row count per bucket for the last [`days_back`](/guides/anomaly-detection-configuration/days-back) days (by default 14).
 
 The test then compares the row count of buckets within the detection period (last 2 days by default, controlled by the
 `backfill_days` var), and compares it with the row count of the previous time buckets.
 If there were any anomalies during the detection period, the test will fail.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 
 <CodeGroup>
@@ -96,7 +95,7 @@ The test then compares the freshness of buckets within the detection period (las
 `backfill_days` var), and compares it with the freshness of the previous time buckets.
 If there were any anomalies during the detection period, the test will fail.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 <CodeGroup>
 
@@ -150,7 +149,7 @@ timestamp ("now") and the most recent event timestamp.
 - If both an `event_timestamp_column` and an `update_timestamp_column` are provided, the test will measure over time
 the difference between these two columns.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 <CodeGroup>
 
@@ -199,7 +198,7 @@ It is best to configure it on low-cardinality fields.
 The test counts rows grouped by given columns/expressions, and can be configured using the `dimensions`
 and `where_expression` keys.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 <CodeGroup>
 
@@ -265,7 +264,7 @@ Executes column level monitors and anomaly detection on all the columns of the t
 are [detailed here](/guides/data-anomaly-detection#tests-and-monitors-types) and can be configured using
 the `all_columns_anomalies` key.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 <CodeGroup>
 
@@ -323,10 +322,13 @@ Executes column level monitors and anomaly detection. Specific monitors
 are [detailed here](/guides/data-anomaly-detection#tests-and-monitors-types) and can be configured using
 the `column_anomalies` key.
 
-For advanced configuration of Elementary anomaly tests, please click [here](/guides/add-elementary-tests#advanced-configuration-for-your-elementary-tests)
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
 
 <CodeGroup>
 
+
+For advanced configuration of Elementary anomaly tests, refer to [tests configuration](/guides/elementary-tests-configuration).
+
 ```yml Models
 version: 2
 
@@ -563,28 +565,35 @@ sources:
 
 ## Configure your elementary anomaly detection tests
 
-The elementary anomaly detection tests described above can work out-of-the-box with default configuration. However,
-we support additional configuration that can be used to customize their behavior, depending on your needs.
-Read all about data anomaly detection tests configuration [here](/guides/elementary-tests-configuration).
+<Tip>If your data set has a timestamp column that represents the creation time of a field, it is highly recommended configuring it as a `timestamp_column`.</Tip>
+
+To support different types of data sets, the tests have configuration that can be used to customize their behavior.
+Read more about [data anomaly detection tests configuration here](/guides/elementary-tests-configuration).
 
-We recommend adding a tag to the tests so you could execute these in a dedicated run using the selection
-parameter `--select tag:elementary`.
-If you wish to only be warned on anomalies, configure the severity of the tests to warn.
+We recommend adding a tag to the tests so you could execute these in a dedicated run using the selection parameter `--select tag:elementary`.
+If you wish to only be warned on anomalies, configure the `severity` of the tests to `warn`.
 
 
 ## What happens on each test?
 
-Upon running a test, your data is split into time buckets based on the `time_bucket` field and is limited by
-the `days_back` var. The test then compares a certain metric (e.g. row count) of the buckets that are within the detection
-period (`backfill_days`) to the row count of all the previous time buckets within the `days_back` period.
+Upon running a test, your data is split into time buckets based on the [`time_bucket`](/guides/anomaly-detection-configuration/time-bucket) field and is limited by
+the [`days_back`](/guides/anomaly-detection-configuration/days-back) var. The test then compares a certain metric (e.g. row count) of the buckets that are within the detection
+period ([`backfill_days`](/guides/anomaly-detection-configuration/backfill-days)) to the row count of all the previous time buckets within the [`days_back`](/guides/anomaly-detection-configuration/days-back) period.
 If there were any anomalies in the detection period, the test will fail.
-On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical
-metrics.
-At the end of the `dbt test` run, all results and collected metrics are merged into the elementary models.
+On each test elementary package executes the relevant monitors, and searches for anomalies by comparing to historical metrics.
+
+<Tooltip tip="Anomaly detection tests core concepts">
+<img
+src="https://github.com/elementary-data/assets-hosting/master/anomaly_detection/elementary-anomaly-detection-core-concepts.png"
+alt="Elementary anomaly detection tests core concepts"
+/>
+</Tooltip>
+
+To learn more, refer to [core concepts](/guides/how-anomaly-detection-works).
+
 
 ## What does it mean when a test fails?
 
-When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer
-to [anomaly detection](/guides/data-anomaly-detection#anomaly-detection).
+When a test fail, it means that an anomaly was detected on this metric and dataset. To learn more, refer to [core concepts](/guides/how-anomaly-detection-works) and [anomaly detection](/guides/data-anomaly-detection).
 
 

docs/guides/anomaly-detection-configuration/anomaly-direction.mdx

@@ -0,0 +1,45 @@
+---
+title: "anomaly_direction"
+sidebarTitle: "anomaly_direction"
+---
+
+`anomaly_direction: both | spike | drop`
+
+By default, data points are compared to the expected range and check if these are below or above it.
+For some data monitors, you might only want to flag anomalies if they are above the range and not under it, and vice versa.
+For example - when monitoring for freshness, we only want to detect data delays and not data that is “early”. 
+The anomaly_direction configuration is used to configure the direction of the expected range, and can be set to both, spike or drop.
+
+
+- _Default: `both`_
+- _Supported values: `both`, `spike`, `drop`_
+- _Relevant tests: All anomaly detection tests_
+- _Configuration level: test_
+
+<Frame caption="anomaly_direction change impact">
+    <img
+    src="https://res.cloudinary.com/diuctyblm/image/upload/v1681301375/Anomaly%20detection%20tests/anomaly_direction_r1sdl9.png"
+    alt="anomaly_direction change impact"
+    />
+</Frame>
+
+<RequestExample>
+
+```yaml test
+models:
+  - name: this_is_a_model
+    tests:
+    
+      - elementary.volume_anomalies:
+          anomaly_direction: drop
+          
+      - elementary.all_columns_anomalies:
+          column_anomalies:
+            - null_count
+            - missing_count
+            - zero_count
+          anomaly_direction: spike
+          
+```
+
+</RequestExample>

docs/guides/anomaly-detection-configuration/anomaly-sensitivity.mdx

@@ -0,0 +1,40 @@
+---
+title: "anomaly_sensitivity"
+sidebarTitle: "anomaly_sensitivity"
+---
+
+`anomaly_sensitivity: [int]`
+
+Configuration to define how the expected range is calculated. 
+A sensitivity of 3 means that the expected range is within 3 standard deviations from the average of the training set. 
+Smaller sensitivity means this range will be reduced and more values will be potentially flagged as anomalies. 
+Larger values will have the opposite effect and will reduce the number of anomalies as the expected range will be larger.
+
+
+- _Default: 3_
+- _Relevant tests: All anomaly detection tests_
+- _Configuration level: var, test config_
+
+<Frame caption="anomaly_sensitivity change impact">
+    <img
+    src="https://res.cloudinary.com/diuctyblm/image/upload/v1683997014/anomaly_sensitivity_n9dpof.png"
+    alt="anomaly_sensitivity change impact"
+    />
+</Frame>
+
+<RequestExample>
+
+```yaml dbt_project.yml
+vars:
+  anomaly_sensitivity: 3
+```
+
+```yaml test
+models:
+  - name: this_is_a_model
+    tests:
+      - elementary.volume_anomalies:
+          anomaly_sensitivity: 3
+```
+
+</RequestExample>

docs/guides/anomaly-detection-configuration/backfill-days.mdx

@@ -0,0 +1,56 @@
+---
+title: "backfill_days"
+sidebarTitle: "backfill_days"
+---
+
+`backfill_days: [int]`
+
+Configuration to define the detection period. 
+If the backfill_days are set to 2, only data points in the last 2 days will be included in the detection period and could be flagged anomalous. 
+If backfill_days is set to 7 days, the detection period will be 7 days long.
+
+For incremental models, this is also the period for re-calculating metrics. 
+If metrics for buckets in the backfill days were already calculated, Elementary will overwrite them. The reason behind it is to monitor recent backfills of data, if there were any.
+This configuration should be changed according to your data delays.
+
+- _Default: 2_
+- _Relevant tests: Anomaly detection tests with `timestamp_column`_
+- _Configuration level: test, var_
+
+<Frame caption="backfill_days change impact">
+    <img
+    src="https://res.cloudinary.com/diuctyblm/image/upload/v1681301376/Anomaly%20detection%20tests/backfill_days_mdjmon.png"
+    alt="backfill_days change impact"
+    />
+</Frame>
+
+
+<RequestExample>
+
+```yaml dbt_project.yml
+vars:
+  backfill_days: 2
+```
+
+```yaml test
+models:
+  - name: this_is_a_model
+    tests:
+      - elementary.volume_anomalies:
+          backfill_days: 7
+```
+
+</RequestExample>
+
+
+
+#### How it works?
+
+The `backfill_days` param only works for tests that have `timestamp_column` configuration.
+
+It works differently according to the table materialization:
+
+- **Regular tables and views** - `backfill_days` defines the detection period.
+- **Incremental models and sources** - `backfill_days` defines the detection period, and the period for which metrics will be re-calculated.
+
+