Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
1b92b14
Added @kochalex to contributors 🎉
Maayan-s Jun 6, 2023
24024ab
add invocations_job_identification
NoyaArie Jun 7, 2023
8c322e3
for each job identification collect the relevant invocations
NoyaArie Jun 7, 2023
54efdb0
add invocations_job_identification to tests json
NoyaArie Jun 8, 2023
684c129
update json
NoyaArie Jun 8, 2023
167c7f6
fotmated
NoyaArie Jun 8, 2023
ec52fa8
Merge pull request #925 from elementary-data/ele-1042-distinct-job-na…
NoyaArie Jun 8, 2023
df8d339
Updated dbt package requirement.
elongl Jun 8, 2023
cbd41cb
Merge pull request #927 from elementary-data/ele-1055-handle-case-in-…
elongl Jun 8, 2023
7dd517d
Merge pull request #922 from elementary-data/Maayan-s-patch-2
elongl Jun 12, 2023
cdec6b4
Changed the implementation of 'test_conn'.
elongl Jun 13, 2023
aeb75c6
Merge pull request #935 from elementary-data/ele-1087-change-implemen…
elongl Jun 13, 2023
2462192
Updated dbt package requirement revision.
elongl Jun 13, 2023
bfffc5f
Merge pull request #936 from elementary-data/ele-1091-saas-support-ex…
elongl Jun 13, 2023
7692aa6
ELE-971 change tracking to support SaaS (#938)
IDoneShaveIt Jun 14, 2023
f878c86
ELE-1108 config does not support update-bucket-website
IDoneShaveIt Jun 15, 2023
d33a475
Merge pull request #942 from elementary-data/ele-1108-config-doesnt-s…
IDoneShaveIt Jun 15, 2023
28f8a9a
ELE-1117 tracking improvements
IDoneShaveIt Jun 16, 2023
4910465
Merge pull request #944 from elementary-data/ele-1117-tracking-improv…
IDoneShaveIt Jun 18, 2023
10e88ab
TrackingAPI
IDoneShaveIt Jun 19, 2023
4c23794
Merge pull request #948 from elementary-data/ele-1127-tracking-client…
IDoneShaveIt Jun 19, 2023
a146f39
ELE-764 support making dbt columns view a table on sync
IDoneShaveIt Jun 19, 2023
7f3bda5
Merge pull request #949 from elementary-data/ele-764-sync-columns-view
IDoneShaveIt Jun 20, 2023
8a22d94
on-run-end hooks
Maayan-s May 16, 2023
566a119
on-run-end hooks
Maayan-s May 16, 2023
e128a0c
on-run-end hooks
Maayan-s May 16, 2023
c8ae9e3
on-run-end hooks
Maayan-s May 16, 2023
5f87b95
on-run-end hooks
Maayan-s May 16, 2023
f1ff72e
on-run-end hooks
Maayan-s May 16, 2023
9462106
Fixed typos.
elongl May 17, 2023
b6097ba
Fixed incorrect test argument name.
elongl May 18, 2023
e0cff97
Update signup.mdx
Maayan-s May 21, 2023
56c646c
Update connect-data-warehouse.mdx
Maayan-s May 21, 2023
04c25c9
Update mint.json
Maayan-s May 21, 2023
326114d
Update introduction.mdx
Maayan-s May 21, 2023
4b8dafd
Update elementary-in-production.mdx
Maayan-s May 21, 2023
91450e5
on-run-end hooks
Maayan-s May 21, 2023
4ce6fa1
new typeform
Maayan-s May 24, 2023
613663f
jobs info
Maayan-s May 30, 2023
9a48a4b
jobs info
Maayan-s May 30, 2023
3651673
jobs info
Maayan-s May 30, 2023
15f392e
jobs info
Maayan-s May 30, 2023
3e6119a
jobs info
Maayan-s May 30, 2023
1a73096
jobs info
Maayan-s May 30, 2023
993cea9
jobs info
Maayan-s May 30, 2023
7192caa
jobs info
Maayan-s May 30, 2023
3e9cb8d
tests docs changes
Maayan-s May 29, 2023
3f26e65
test config at all levels
Maayan-s May 29, 2023
645dc3b
test config at all levels
Maayan-s May 29, 2023
7d51aa2
removed `table_anomalies`
Maayan-s May 29, 2023
7b34e59
docs min training set size
Maayan-s May 29, 2023
fadd5f9
Update collect-job-data.mdx
Maayan-s Jun 1, 2023
d409db8
jobs info
Maayan-s Jun 1, 2023
134b028
jobs info
Maayan-s Jun 1, 2023
c24d888
jobs info
Maayan-s Jun 1, 2023
1150725
jobs info
Maayan-s Jun 1, 2023
077304b
jobs info
Maayan-s Jun 1, 2023
40d3bfb
jobs info
Maayan-s Jun 1, 2023
b17df3f
jobs info
Maayan-s Jun 1, 2023
5087841
jobs info
Maayan-s Jun 1, 2023
9cb909f
jobs info
Maayan-s Jun 1, 2023
b67c534
jobs info
Maayan-s Jun 1, 2023
cfa9e87
jobs info
Maayan-s Jun 1, 2023
87257e8
jobs info
Maayan-s Jun 1, 2023
ba2d0a6
jobs info
Maayan-s Jun 1, 2023
1a169ba
jobs info
Maayan-s Jun 1, 2023
a9a1c0d
jobs info
Maayan-s Jun 1, 2023
57f8580
jobs info
Maayan-s Jun 1, 2023
589f44c
tests config formating
Maayan-s Jun 1, 2023
c3d260d
tests config formating
Maayan-s Jun 1, 2023
cc4e3ca
tests config formating
Maayan-s Jun 1, 2023
afb40da
tests config formating
Maayan-s Jun 1, 2023
311e52a
tests config formating
Maayan-s Jun 1, 2023
9719dd1
tests config formating
Maayan-s Jun 1, 2023
3313b5b
tests config formating
Maayan-s Jun 1, 2023
bfd92e5
tests config formating
Maayan-s Jun 3, 2023
56d9fe7
tests config formating
Maayan-s Jun 4, 2023
ae75edf
Remove hidden new lines in code block
hahnbeelee Jun 5, 2023
d4a4b45
Update event-freshness-anomalies.mdx
Maayan-s Jun 5, 2023
77bfe91
tests config formating
Maayan-s Jun 5, 2023
036e7ce
tests config formating
Maayan-s Jun 5, 2023
02a2c6a
Update freshness-anomalies.mdx
Maayan-s Jun 5, 2023
fb16a95
Update freshness-anomalies.mdx
Maayan-s Jun 5, 2023
0665d48
Update column-anomalies.mdx
Maayan-s Jun 5, 2023
8f892b8
tests config formating
Maayan-s Jun 5, 2023
e6399cd
Update all-columns-anomalies.mdx
Maayan-s Jun 6, 2023
2943f95
Update all-columns-anomalies.mdx
Maayan-s Jun 6, 2023
168e490
tests config formating
Maayan-s Jun 7, 2023
60c64da
cloud docs changes
Maayan-s Jun 8, 2023
097c648
cloud docs changes
Maayan-s Jun 8, 2023
5238e59
cloud docs changes
Maayan-s Jun 8, 2023
982760a
cloud docs changes
Maayan-s Jun 8, 2023
1db49d0
cloud docs changes
Maayan-s Jun 8, 2023
b0a5a6f
cloud docs changes
Maayan-s Jun 10, 2023
c5c576f
Update manage-team.mdx
Maayan-s Jun 11, 2023
9092428
new release notes
Maayan-s Jun 11, 2023
99243fb
Update add-elementary-tests.mdx
Maayan-s Jun 13, 2023
621b7aa
Update add-elementary-tests.mdx
Maayan-s Jun 13, 2023
e5d9279
Update add-elementary-tests.mdx
Maayan-s Jun 13, 2023
6560566
Update add-elementary-tests.mdx
Maayan-s Jun 13, 2023
d87259b
Update add-elementary-tests.mdx
Maayan-s Jun 13, 2023
7cffee9
Update where-expression.mdx
Maayan-s Jun 13, 2023
0e0925b
Update how-anomaly-detection-works.mdx
Jun 14, 2023
5bc8f8d
new release notes
Maayan-s Jun 14, 2023
9866ec3
new release notes
Maayan-s Jun 14, 2023
747a4b1
new release notes
Maayan-s Jun 14, 2023
af4fa96
new release notes
Maayan-s Jun 14, 2023
9254fbc
new release notes
Maayan-s Jun 14, 2023
d6fc692
new release notes
Maayan-s Jun 14, 2023
588ed4c
Update send-report-summary.mdx
Maayan-s Jun 14, 2023
87897ac
new release notes
Maayan-s Jun 14, 2023
573501c
Update send-slack-alerts.mdx
Maayan-s Jun 14, 2023
a76c76c
Update send-slack-alerts.mdx
Maayan-s Jun 14, 2023
3d4d312
Update create-profile.mdx
Maayan-s Jun 15, 2023
120b671
Update redshift-profile.mdx
Maayan-s Jun 15, 2023
b8aec4a
Update security-and-privacy.mdx
Maayan-s Jun 15, 2023
17f2de8
Update security-and-privacy.mdx
Maayan-s Jun 15, 2023
826fc60
Update security-and-privacy.mdx
Maayan-s Jun 15, 2023
b07e9e0
Update security-and-privacy.mdx
Maayan-s Jun 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ Check out the [contributions guide](https://docs.elementary-data.com/general/con
<a href="https://github.com/dimoschi"><img src="https://github.com/u/13113025?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/manulpatel"><img src="https://github.com/u/77568048?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/arilmav"><img src="https://github.com/u/60354578?v=4" width="50" height="50" alt=""/></a>
<a href="https://github.com/kochalex"><img src="https://github.com/u/3217653?v=4" width="50" height="50" alt=""/></a>

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
Expand Down
22 changes: 12 additions & 10 deletions docs/_snippets/faq/question-tests-configuration-priorities.mdx
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
<Accordion title="How test configurations are prioritized?">

The configuration of Elementary is dbt native and follows the same priorities of `dbt configuration`.
The more granular and specific configuration overrides the less granular one.
The configuration of Elementary is dbt native and follows the same priorities and inheritance.
The more granular and specific configuration overrides the less granular one.

Elementary searches and prioritizes configuration in the following order:

For models:
**For models tests:**
1. Test arguments.
2. Model configuration.
3. Global vars in `dbt_project.yml`.
2. Tests path configuration under `tests` key in `dbt_project.yml`.
3. Model configuration.
4. Path configuration under `models` key in `dbt_project.yml`.
5. Global vars in `dbt_project.yml`.

For sources:
**For sources tests:**
1. Test arguments.
2. Table configuration.
3. Source configuration.
4. Global vars in `dbt_project.yml`.

2. Tests path configuration under `tests` key in `dbt_project.yml`.
3. Table configuration.
4. Source configuration.
5. Global vars in `dbt_project.yml`.

</Accordion>
1 change: 1 addition & 0 deletions docs/_snippets/profiles/redshift-profile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ elementary:
user: [username]
password: [password]

port: 5439
dbname: [database name]
schema: [schema name]_elementary
threads: 4
Expand Down
6 changes: 5 additions & 1 deletion docs/cloud/general/security-and-privacy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,13 @@ To avoid this sampling, set the var `test_sample_rows_count: 0` in your `dbt_pro

## Compliance

<Check>
**SOC 2 certification:** Elementary is currently in the process of obtaining SOC2 and ISO27001 compliance.
</Check>

[Contact us](mailto:legal@elementary-data.com) for auditing reports and penetration testing results.

## Have more questions?

We would be happy to answer!
Reach out to us on [email](mailto:legal@elementary-data.com) or [Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-1b9vogqmq-y~IRhc2396CbHNBXLsrXcA).
Reach out to us on [email](mailto:legal@elementary-data.com) or [Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-1b9vogqmq-y~IRhc2396CbHNBXLsrXcA).
17 changes: 3 additions & 14 deletions docs/cloud/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ title: "Introduction"

Elementary Cloud is the easiest and fastest way to get the most out of Elementary.

<Card title="Signup for details" href="https://www.elementary-data.com/cloud-beta">
_The service is currently in private beta_
<Card title="Signup to Elementary Cloud" href="https://t2taztilhde.typeform.com/to/ObfMbxB5?utm_source=docs&utm_medium=cloud_docs&utm_content=v1">
</Card>

##
Expand Down Expand Up @@ -39,15 +38,5 @@ alt="Elementary Managed high level flow"
1. [Install the Elementary dbt package in your project](/cloud/onboarding/quickstart-dbt-package).
2. [Signup and setup integrations](/cloud/onboarding/signup).

<Card title="Signup for details" href="https://www.elementary-data.com/cloud-beta">
_The service is currently in private beta_
</Card>

## Security and privacy

Elementary cloud requires access only to the Elementary schema and the tables in it.
The data in the schema in full is stored in the client's data warehouse.

We secure Elementary cloud infrastructure with the highest standards.
You can delete your account at any time, and all your configuration and reports will be deleted immediately and permanently from Elementary servers.
For details, refer to our [Terms of Service](https://www.elementary-data.com/terms-of-service).
<Card title="Signup to Elementary Cloud" href="https://t2taztilhde.typeform.com/to/ObfMbxB5?utm_source=docs&utm_medium=cloud_docs&utm_content=v1">
</Card>
13 changes: 3 additions & 10 deletions docs/cloud/manage-team.mdx
Original file line number Diff line number Diff line change
@@ -1,23 +1,16 @@
---
title: "Quickstart: Invite and remove users"
sidebarTitle: "Team settings"
title: "Invite and remove users"
sidebarTitle: "5. Invite users"
---

### Invite users

After you signup, you could invite team members to join you! 🎉

On the top left buttun select `Account settings`, and you can invite users on the `Team` screen.
On the top left button select `Account settings`, and you can invite users on the `Team` screen.

Users you invite will recieve an Email saying you invited them, and will need to accept and activate their account.

<Frame>
<img src="/pics/cloud/invite_users.png" />
</Frame>


### Remove users

On the top left buttun select `Account settings`, and select the `Team` screen.

You can remove users by clicking selecting this option under the user options.
43 changes: 12 additions & 31 deletions docs/cloud/onboarding/connect-data-warehouse.mdx
Original file line number Diff line number Diff line change
@@ -1,46 +1,27 @@
---
title: "Connect your data warehouse"
sidebarTitle: "Data warehouse"
sidebarTitle: "4. Connect data warehouse"
---

You can connect Elementary to a data warehouse that has an Elementary schema (created by the [Elementary dbt package](/cloud/onboarding/quickstart-dbt-package)).

Here are the steps needed to enable the connection:
Elementary Cloud needs:
- [`profiles.yml`](/cloud/onboarding/create-profile) with connection details
- Read permissions to the Elementary schema (and not the rest of your data)
- Network access (might require to allowlist Elementary IP address)

### Authentication and IP Allowlist

Elementary needs authentication details, permissions to read the Elementary schema (and not the rest of your data), and network access enabled by adding the cloud IPs to your data warehouse allowlist.

Here are the guides on how to configure these on each supported data warehouse:

- Bigquery
- Snowflake
- Redshift
- Databricks
- Postgres

Elementary IP for allowlist: `3.126.156.226`

### Create a `profiles.yml` file

You will need to provide the connection and authentication details by uploading a YML file with a connection profile named `elementary`.
The profile needs to point at the database and schema name where your elementary tables are.

The easiest way to generate the profile is to run the following command within the dbt project where you deployed the elementary dbt package (works in dbt cloud as well):
### Connect Elementary cloud

```shell
dbt run-operation elementary.generate_elementary_cli_profile
```
On the `Account settings` under `Integrations`, press `Connect` on the "Connect Your data warehouse" section.

Save the output to a YML file, update the missing details, and you are ready.
Provide an environment name, select a data warehouse type, and upload the `profiles.yml` file with the `elementary` profile.

Here are the formats of profile for each supported data warehouse:

<Snippet file="profiles/all-profiles.mdx" />
### Allowlist Elementary IP

Elementary IP for allowlist: `3.126.156.226`

### Connect Elementary cloud

On the `Account settings` under `Integrations`, press `Connect` on the "Connect Your data warehouse" section.
### Need help with onboarding?

Provide an env name, select a data warehouse type, and upload the `profiles.yml` file with the `elementary` profile.
We can provide [support on Slack](https://join.slack.com/t/elementary-community/shared_invite/zt-1b9vogqmq-y~IRhc2396CbHNBXLsrXcA) or hop on an [onboarding call](https://savvycal.com/MaayanSa/df29881c).
133 changes: 133 additions & 0 deletions docs/cloud/onboarding/create-profile.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
---
title: "Create `profiles.yml` file"
sidebarTitle: "2. Create profiles.yml"
---

You will need to provide Elementary cloud a `profiles.yml` file with a connection profile named `elementary`.

- The profile needs to point at the database and schema name where your elementary tables are.
- The provided credentials need to have read permissions to the elementary schema.

The easiest way to generate the profile is:
1. Run the following command in the dbt project where elementary dbt package is deployed (works in dbt cloud as well):

```shell
dbt run-operation elementary.generate_elementary_cli_profile
```

2. Copy and save the output to a `profiles.yml` file, update the missing details, and you are ready.

### Permissions and security

**Elementary cloud doesn't need permissions to your sensitive data.**

It is recommended to create a read only user for the elementary schema only, and provide it to Elementary Cloud in the profile.
For more details, refer to [security and privacy](/cloud/security-and-privacy).

### `profiles.yml` examples

Here is the format of `profiles.yml` for each supported data warehouse:

<CodeGroup>

```yml Snowflake
## SNOWFLAKE ##
## Configure the database and schema of elementary models.

elementary:
outputs:
default:
type: snowflake
account: [account id]

## User/password auth ##
user: [username]
password: [password]

port: 5439
role: [user role]
database: [database name]
warehouse: [warehouse name]
schema: [schema name]_elementary
threads: 4

```

```yml BigQuery
## BIGQUERY ##
## Configure the database and schema of elementary models.

elementary:
outputs:
default:
type: bigquery

## Service account auth ##
method: service-account
keyfile: empty

project: [project id]
dataset: [dataset name] # elementary dataset, usually [dataset name]_elementary
threads: 4
location: [dataset location]
priority: interactive
```

```yml Redshift
## REDSHIFT ##
## Configure the database and schema of elementary models.

elementary:
outputs:
default:
type: redshift
host: [hostname, like hostname.region.redshift.amazonaws.com]

## User/password auth ##
user: [username]
password: [password]

dbname: [database name]
schema: [schema name] # elementary schema, usually [schema name]_elementary
threads: 4
```

```yml Databricks
## DATABRICKS ##
## Configure the database and schema of elementary models.

elementary:
outputs:
default:
type: databricks
host: [hostname, like <ID>.cloud.databricks.com]
http_path: [like /sql/1.0/endpoints/<ID>]
schema: [schema name] # elementary schema, usually [schema name]_elementary
token: [token]
threads: [number of threads like 8]
```

```yml Postgres
## POSTGRES ##
## Configure the database and schema of elementary models.

elementary:
outputs:
default:
type: postgres
host: [hostname]
user: [username]
password: [password]
port: [port]
dbname: [database name]
schema: [schema name] # elementary schema, usually [schema name]_elementary
threads: [1 or more]

```

</CodeGroup>

### What's next?

1. [Singup to Elementary cloud](/cloud/sonboarding/signup).
2. [Connect your Elementary schema to Elementary cloud](/cloud/onboarding/connect-data-warehouse).
9 changes: 5 additions & 4 deletions docs/cloud/onboarding/quickstart-dbt-package.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Quickstart: Install Elementary dbt package"
sidebarTitle: "Install dbt package"
title: "Install Elementary dbt package"
sidebarTitle: "1. Install dbt package"
---

<Snippet file="quickstart-package-install.mdx" />
Expand Down Expand Up @@ -52,5 +52,6 @@ If you see data in these models you completed the package deployment (Congrats!

### What's next?

1. [Singup to Elementary cloud](/cloud/saas-onboarding/signup).
2. [Connect your Elementary schema to Elementary cloud](/cloud/saas-onboarding/connect-data-warehouse).
1. [Create a connection profile](/cloud/onboarding/create-profile).
2. [Singup to Elementary cloud](/cloud/sonboarding/signup).
3. [Connect your Elementary schema to Elementary cloud](/cloud/onboarding/connect-data-warehouse).
6 changes: 3 additions & 3 deletions docs/cloud/onboarding/signup.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Quickstart: Signup and connect"
sidebarTitle: "Signup and connect"
title: "Signup and login"
sidebarTitle: "3. Signup and login"
---

### Signup to Elementary cloud
Expand Down Expand Up @@ -28,4 +28,4 @@ After you connect a data warehouse with an Elementary schema in it, you can star

### What's next?

[Connect your Elementary schema to Elementary cloud](/cloud/saas-onboarding/connect-data-warehouse).
[Connect your Elementary schema to Elementary cloud](/cloud/onboarding/connect-data-warehouse).
Loading