Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions providers/google/docs/connections/gcp_looker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,42 +17,42 @@

.. _howto/connection:gcp_looker:

Google Cloud Platform Looker Connection
=======================================
Google Cloud Platform Data Studio (Looker) Connection
======================================================

Communication between Airflow and Looker is done via `Looker API <https://docs.looker.com/reference/api-and-integration/api-reference/v4.0>`_.
To facilitate the API communication Looker operators use `Looker SDK <https://pypi.org/project/looker-sdk/>`_ as an API client.
Before calling API, Looker SDK needs to authenticate itself using your Looker API credentials.
Communication between Airflow and Data Studio (Looker) is done via `Looker API <https://docs.looker.com/reference/api-and-integration/api-reference/v4.0>`_.
To facilitate the API communication, Data Studio operators use `Looker SDK <https://pypi.org/project/looker-sdk/>`_ as an API client.
Before calling API, Looker SDK needs to authenticate itself using your Data Studio API credentials.

* Obtain your Looker API credentials using instructions in the `Looker API authentication documentation <https://docs.looker.com/reference/api-and-integration/api-auth#authentication_with_an_sdk>`_.
* Obtain your Data Studio API credentials using instructions in the `Looker API authentication documentation <https://docs.looker.com/reference/api-and-integration/api-auth#authentication_with_an_sdk>`_.

* Obtain your Looker API path and port as described in the `Looker API documentation <https://docs.looker.com/reference/api-and-integration/api-getting-started#looker_api_path_and_port>`_.
* Obtain your Data Studio API path and port as described in the `Looker API documentation <https://docs.looker.com/reference/api-and-integration/api-getting-started#looker_api_path_and_port>`_.

* Setup a Looker connection in Airflow.
* Setup a Data Studio connection in Airflow.

The ``HTTP`` connection type provides connection to Looker API.
The ``HTTP`` connection type provides connection to Data Studio API.

The :class:`~airflow.providers.google.cloud.hooks.looker.LookerHook` uses this connection to run
API requests on a Looker instance issued by :class:`~airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator` and :class:`~airflow.providers.google.cloud.sensors.looker.LookerCheckPdtBuildSensor`.
API requests on a Data Studio instance issued by :class:`~airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator` and :class:`~airflow.providers.google.cloud.sensors.looker.LookerCheckPdtBuildSensor`.


Configuring the Connection
--------------------------

Host (required)
Base URL for Looker API. Do not include /api/* in the URL.
Base URL for Data Studio API. Do not include /api/* in the URL.

Login (required)
Looker API client id.
Data Studio API client id.

Password (required)
Looker API client secret.
Data Studio API client secret.

Port (optional)
Port for Looker API. If hosted on GCP, don't specify the port leaving just the host.
Port for Data Studio API. If hosted on GCP, don't specify the port leaving just the host.

Extra (optional)
Specify the extra parameters (as json dictionary) that can be used in Looker
Specify the extra parameters (as json dictionary) that can be used in Data Studio
connection. The following parameters are supported:

* ``verify_ssl`` - Set to false only if testing locally against self-signed certs. Defaults to true if not specified.
Expand All @@ -70,7 +70,7 @@ Extra (optional)
Connection URI
--------------

A URL configuration example of a Looker connection:
A URL configuration example of a Data Studio connection:

.. code-block::

Expand Down
Binary file not shown.
204 changes: 158 additions & 46 deletions providers/google/docs/operators/cloud/dataplex.rst

Large diffs are not rendered by default.

120 changes: 92 additions & 28 deletions providers/google/docs/operators/cloud/dataproc.rst

Large diffs are not rendered by default.

24 changes: 16 additions & 8 deletions providers/google/docs/operators/cloud/looker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@
specific language governing permissions and limitations
under the License.

Google Cloud Looker Operators
===============================
Google Data Studio (Looker) Operators
======================================

Looker is a business intelligence software and big data analytics platform that
Data Studio (Looker) is a business intelligence software and big data analytics platform that
helps you explore, analyze and share real-time business analytics easily.

Looker has a Public API and associated SDK clients in different languages,
which allow programmatic access to the Looker data platform.
Data Studio (Looker) has a Public API and associated SDK clients in different languages,
which allow programmatic access to the Data Studio platform.

For more information visit `Looker API documentation <https://docs.looker.com/reference/api-and-integration>`_.

Expand All @@ -39,16 +39,20 @@ To use these operators, you must do a few things:

Detailed information is available for :doc:`Installation <apache-airflow:installation/index>`.

* Setup a Looker connection in Airflow. You can check :doc:`apache-airflow:howto/connection` and :doc:`/connections/gcp_looker`
* Setup a Data Studio connection in Airflow. You can check :doc:`apache-airflow:howto/connection` and :doc:`/connections/gcp_looker`

Start a PDT materialization job
-------------------------------

To submit a PDT materialization job to Looker you need to provide a model and view name.
To submit a PDT materialization job to Data Studio you need to provide a model and view name.

The job configuration can be submitted in synchronous (blocking) mode by using:
:class:`~airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator`.

The executable example below still imports the compatibility name
``LookerStartPdtBuildOperator``. The preferred alias for new code is
``DataStudioStartPdtBuildOperator``.

.. exampleinclude:: /../../google/tests/system/google/cloud/looker/example_looker.py
:language: python
:dedent: 4
Expand All @@ -60,11 +64,15 @@ Alternatively, the job configuration can be submitted in asynchronous mode by us
:class:`~airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator` and
:class:`~airflow.providers.google.cloud.sensors.looker.LookerCheckPdtBuildSensor`.

The executable example below still imports the compatibility name
``LookerStartPdtBuildOperator``. The preferred alias for new code is
``DataStudioStartPdtBuildOperator``.

.. exampleinclude:: /../../google/tests/system/google/cloud/looker/example_looker.py
:language: python
:dedent: 4
:start-after: [START cloud_looker_async_start_pdt_sensor]
:end-before: [END cloud_looker_async_start_pdt_sensor]

There are more arguments to provide in the jobs than the examples show.
For the complete list of arguments take a look at Looker operator arguments at :class:`airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator`
For the complete list of arguments take a look at Data Studio operator arguments at :class:`airflow.providers.google.cloud.operators.looker.LookerStartPdtBuildOperator`
36 changes: 17 additions & 19 deletions providers/google/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -310,10 +310,6 @@ integrations:
- /docs/apache-airflow-providers-google/operators/cloud/compute_ssh.rst
logo: /docs/integration-logos/Compute-Engine.png
tags: [gcp]
- integration-name: Google Data Proc
external-doc-url: https://cloud.google.com/dataproc/
logo: /docs/integration-logos/Google-Data-Proc.png
tags: [gcp]
- integration-name: Google Dataflow
external-doc-url: https://cloud.google.com/dataflow/
how-to-guide:
Expand All @@ -326,7 +322,7 @@ integrations:
- /docs/apache-airflow-providers-google/operators/cloud/datafusion.rst
logo: /docs/integration-logos/Google-Data-Fusion.png
tags: [gcp]
- integration-name: Google Dataplex
- integration-name: Google Knowledge Catalog
external-doc-url: https://cloud.google.com/dataplex/
how-to-guide:
- /docs/apache-airflow-providers-google/operators/cloud/dataplex.rst
Expand All @@ -342,7 +338,7 @@ integrations:
how-to-guide:
- /docs/apache-airflow-providers-google/operators/cloud/dataproc_metastore.rst
tags: [gcp]
- integration-name: Google Dataproc
- integration-name: Google Managed Service for Apache Spark
external-doc-url: https://cloud.google.com/dataproc/
how-to-guide:
- /docs/apache-airflow-providers-google/operators/cloud/dataproc.rst
Expand Down Expand Up @@ -439,7 +435,7 @@ integrations:
how-to-guide:
- /docs/apache-airflow-providers-google/operators/cloud/vertex_ai.rst
tags: [gcp]
- integration-name: Google Looker
- integration-name: Google Data Studio
external-doc-url: https://cloud.google.com/looker
logo: /docs/integration-logos/Cloud-Looker.png
how-to-guide:
Expand Down Expand Up @@ -515,18 +511,20 @@ operators:
- integration-name: Google Data Fusion
python-modules:
- airflow.providers.google.cloud.operators.datafusion
- integration-name: Google Dataplex
- integration-name: Google Knowledge Catalog
python-modules:
- airflow.providers.google.cloud.operators.dataplex
- airflow.providers.google.cloud.operators.knowledge_catalog
- integration-name: Google Dataprep
python-modules:
- airflow.providers.google.cloud.operators.dataprep
- integration-name: Google Dataproc Metastore
python-modules:
- airflow.providers.google.cloud.operators.dataproc_metastore
- integration-name: Google Dataproc
- integration-name: Google Managed Service for Apache Spark
python-modules:
- airflow.providers.google.cloud.operators.dataproc
- airflow.providers.google.cloud.operators.managed_spark
- integration-name: Google Datastore
python-modules:
- airflow.providers.google.cloud.operators.datastore
Expand Down Expand Up @@ -612,7 +610,7 @@ operators:
- airflow.providers.google.cloud.operators.vertex_ai.generative_model
- airflow.providers.google.cloud.operators.vertex_ai.feature_store
- airflow.providers.google.cloud.operators.vertex_ai.ray
- integration-name: Google Looker
- integration-name: Google Data Studio
python-modules:
- airflow.providers.google.cloud.operators.looker
- integration-name: Google Cloud Dataform
Expand Down Expand Up @@ -662,10 +660,10 @@ sensors:
- integration-name: Google Dataprep
python-modules:
- airflow.providers.google.cloud.sensors.dataprep
- integration-name: Google Dataplex
- integration-name: Google Knowledge Catalog
python-modules:
- airflow.providers.google.cloud.sensors.dataplex
- integration-name: Google Dataproc
- integration-name: Google Managed Service for Apache Spark
python-modules:
- airflow.providers.google.cloud.sensors.dataproc
- integration-name: Google Dataproc Metastore
Expand All @@ -692,7 +690,7 @@ sensors:
- integration-name: Google Display&Video 360
python-modules:
- airflow.providers.google.marketing_platform.sensors.display_video
- integration-name: Google Looker
- integration-name: Google Data Studio
python-modules:
- airflow.providers.google.cloud.sensors.looker
- integration-name: Google Cloud Dataform
Expand Down Expand Up @@ -778,7 +776,7 @@ hooks:
- integration-name: Google Data Fusion
python-modules:
- airflow.providers.google.cloud.hooks.datafusion
- integration-name: Google Dataplex
- integration-name: Google Knowledge Catalog
python-modules:
- airflow.providers.google.cloud.hooks.dataplex
- integration-name: Google Dataprep
Expand All @@ -787,7 +785,7 @@ hooks:
- integration-name: Google Dataproc Metastore
python-modules:
- airflow.providers.google.cloud.hooks.dataproc_metastore
- integration-name: Google Dataproc
- integration-name: Google Managed Service for Apache Spark
python-modules:
- airflow.providers.google.cloud.hooks.dataproc
- integration-name: Google Datastore
Expand Down Expand Up @@ -899,7 +897,7 @@ hooks:
- airflow.providers.google.cloud.hooks.vertex_ai.prediction_service
- airflow.providers.google.cloud.hooks.vertex_ai.feature_store
- airflow.providers.google.cloud.hooks.vertex_ai.ray
- integration-name: Google Looker
- integration-name: Google Data Studio
python-modules:
- airflow.providers.google.cloud.hooks.looker
- integration-name: Google Cloud Dataform
Expand Down Expand Up @@ -957,10 +955,10 @@ triggers:
- integration-name: Google Data Fusion
python-modules:
- airflow.providers.google.cloud.triggers.datafusion
- integration-name: Google Dataplex
- integration-name: Google Knowledge Catalog
python-modules:
- airflow.providers.google.cloud.triggers.dataplex
- integration-name: Google Dataproc
- integration-name: Google Managed Service for Apache Spark
python-modules:
- airflow.providers.google.cloud.triggers.dataproc
- integration-name: Google Cloud Storage (GCS)
Expand Down Expand Up @@ -1330,7 +1328,7 @@ connection-types:
type: ["string", "null"]
format: password
- hook-class-name: airflow.providers.google.cloud.hooks.looker.LookerHook
hook-name: "Google Looker"
hook-name: "Google Data Studio"
connection-type: gcp_looker

extra-links:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations

from typing import TypeAlias

from airflow.providers.google.cloud.operators.dataplex import (
DataplexCatalogCreateAspectTypeOperator,
DataplexCatalogCreateEntryGroupOperator,
DataplexCatalogCreateEntryOperator,
DataplexCatalogCreateEntryTypeOperator,
DataplexCatalogDeleteAspectTypeOperator,
DataplexCatalogDeleteEntryGroupOperator,
DataplexCatalogDeleteEntryOperator,
DataplexCatalogDeleteEntryTypeOperator,
DataplexCatalogGetAspectTypeOperator,
DataplexCatalogGetEntryGroupOperator,
DataplexCatalogGetEntryOperator,
DataplexCatalogGetEntryTypeOperator,
DataplexCatalogListAspectTypesOperator,
DataplexCatalogListEntriesOperator,
DataplexCatalogListEntryGroupsOperator,
DataplexCatalogListEntryTypesOperator,
DataplexCatalogLookupEntryOperator,
DataplexCatalogSearchEntriesOperator,
DataplexCatalogUpdateAspectTypeOperator,
DataplexCatalogUpdateEntryGroupOperator,
DataplexCatalogUpdateEntryOperator,
DataplexCatalogUpdateEntryTypeOperator,
DataplexCreateAssetOperator,
DataplexCreateLakeOperator,
DataplexCreateOrUpdateDataProfileScanOperator,
DataplexCreateOrUpdateDataQualityScanOperator,
DataplexCreateTaskOperator,
DataplexCreateZoneOperator,
DataplexDeleteAssetOperator,
DataplexDeleteDataProfileScanOperator,
DataplexDeleteDataQualityScanOperator,
DataplexDeleteLakeOperator,
DataplexDeleteTaskOperator,
DataplexDeleteZoneOperator,
DataplexGetDataProfileScanOperator,
DataplexGetDataProfileScanResultOperator,
DataplexGetDataQualityScanOperator,
DataplexGetDataQualityScanResultOperator,
DataplexGetTaskOperator,
DataplexListTasksOperator,
DataplexRunDataProfileScanOperator,
DataplexRunDataQualityScanOperator,
)

KnowledgeCatalogCreateTaskOperator: TypeAlias = DataplexCreateTaskOperator
KnowledgeCatalogDeleteTaskOperator: TypeAlias = DataplexDeleteTaskOperator
KnowledgeCatalogListTasksOperator: TypeAlias = DataplexListTasksOperator
KnowledgeCatalogGetTaskOperator: TypeAlias = DataplexGetTaskOperator
KnowledgeCatalogCreateLakeOperator: TypeAlias = DataplexCreateLakeOperator
KnowledgeCatalogDeleteLakeOperator: TypeAlias = DataplexDeleteLakeOperator
KnowledgeCatalogCreateOrUpdateDataQualityScanOperator: TypeAlias = (
DataplexCreateOrUpdateDataQualityScanOperator
)
KnowledgeCatalogGetDataQualityScanOperator: TypeAlias = DataplexGetDataQualityScanOperator
KnowledgeCatalogDeleteDataQualityScanOperator: TypeAlias = DataplexDeleteDataQualityScanOperator
KnowledgeCatalogRunDataQualityScanOperator: TypeAlias = DataplexRunDataQualityScanOperator
KnowledgeCatalogGetDataQualityScanResultOperator: TypeAlias = DataplexGetDataQualityScanResultOperator
KnowledgeCatalogCreateOrUpdateDataProfileScanOperator: TypeAlias = (
DataplexCreateOrUpdateDataProfileScanOperator
)
KnowledgeCatalogGetDataProfileScanOperator: TypeAlias = DataplexGetDataProfileScanOperator
KnowledgeCatalogDeleteDataProfileScanOperator: TypeAlias = DataplexDeleteDataProfileScanOperator
KnowledgeCatalogRunDataProfileScanOperator: TypeAlias = DataplexRunDataProfileScanOperator
KnowledgeCatalogGetDataProfileScanResultOperator: TypeAlias = DataplexGetDataProfileScanResultOperator
KnowledgeCatalogCreateZoneOperator: TypeAlias = DataplexCreateZoneOperator
KnowledgeCatalogDeleteZoneOperator: TypeAlias = DataplexDeleteZoneOperator
KnowledgeCatalogCreateAssetOperator: TypeAlias = DataplexCreateAssetOperator
KnowledgeCatalogDeleteAssetOperator: TypeAlias = DataplexDeleteAssetOperator
KnowledgeCatalogCreateEntryGroupOperator: TypeAlias = DataplexCatalogCreateEntryGroupOperator
KnowledgeCatalogGetEntryGroupOperator: TypeAlias = DataplexCatalogGetEntryGroupOperator
KnowledgeCatalogDeleteEntryGroupOperator: TypeAlias = DataplexCatalogDeleteEntryGroupOperator
KnowledgeCatalogListEntryGroupsOperator: TypeAlias = DataplexCatalogListEntryGroupsOperator
KnowledgeCatalogUpdateEntryGroupOperator: TypeAlias = DataplexCatalogUpdateEntryGroupOperator
KnowledgeCatalogCreateEntryTypeOperator: TypeAlias = DataplexCatalogCreateEntryTypeOperator
KnowledgeCatalogGetEntryTypeOperator: TypeAlias = DataplexCatalogGetEntryTypeOperator
KnowledgeCatalogDeleteEntryTypeOperator: TypeAlias = DataplexCatalogDeleteEntryTypeOperator
KnowledgeCatalogListEntryTypesOperator: TypeAlias = DataplexCatalogListEntryTypesOperator
KnowledgeCatalogUpdateEntryTypeOperator: TypeAlias = DataplexCatalogUpdateEntryTypeOperator
KnowledgeCatalogCreateAspectTypeOperator: TypeAlias = DataplexCatalogCreateAspectTypeOperator
KnowledgeCatalogGetAspectTypeOperator: TypeAlias = DataplexCatalogGetAspectTypeOperator
KnowledgeCatalogListAspectTypesOperator: TypeAlias = DataplexCatalogListAspectTypesOperator
KnowledgeCatalogUpdateAspectTypeOperator: TypeAlias = DataplexCatalogUpdateAspectTypeOperator
KnowledgeCatalogDeleteAspectTypeOperator: TypeAlias = DataplexCatalogDeleteAspectTypeOperator
KnowledgeCatalogCreateEntryOperator: TypeAlias = DataplexCatalogCreateEntryOperator
KnowledgeCatalogGetEntryOperator: TypeAlias = DataplexCatalogGetEntryOperator
KnowledgeCatalogListEntriesOperator: TypeAlias = DataplexCatalogListEntriesOperator
KnowledgeCatalogSearchEntriesOperator: TypeAlias = DataplexCatalogSearchEntriesOperator
KnowledgeCatalogLookupEntryOperator: TypeAlias = DataplexCatalogLookupEntryOperator
KnowledgeCatalogUpdateEntryOperator: TypeAlias = DataplexCatalogUpdateEntryOperator
KnowledgeCatalogDeleteEntryOperator: TypeAlias = DataplexCatalogDeleteEntryOperator
Loading