From be18b8c31d7ef26e41510ecb3e16dccb6afe78c0 Mon Sep 17 00:00:00 2001 From: Shahar Epstein Date: Sat, 1 Jun 2024 00:04:45 +0300 Subject: [PATCH] Update integration tests docs and add "Writing Integration Tests" guide Co-authored-by: Jarek Potiuk --- .../testing/integration_tests.rst | 139 ++++++++++++++++-- 1 file changed, 125 insertions(+), 14 deletions(-) diff --git a/contributing-docs/testing/integration_tests.rst b/contributing-docs/testing/integration_tests.rst index 2c55d1f857891..80cf169d909e3 100644 --- a/contributing-docs/testing/integration_tests.rst +++ b/contributing-docs/testing/integration_tests.rst @@ -18,9 +18,11 @@ Airflow Integration Tests ========================= -Some of the tests in Airflow are integration tests. These tests require ``airflow`` Docker -image and extra images with integrations (such as ``celery``, ``mongodb``, etc.). -The integration tests are all stored in the ``tests/integration`` folder. +Integration tests in Airflow check the interactions between Airflow components and external services +that could run as separate Docker containers, without connecting to an external API on the internet. +These tests require ``airflow`` Docker image and extra images with integrations (such as ``celery``, ``mongodb``, etc.). +The integration tests are all stored in the ``tests/integration`` folder, and similarly to the unit tests they all run +using `pytest `_, but they are skipped by default unless ``--integration`` flag is passed to pytest. **The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).** @@ -28,18 +30,20 @@ Enabling Integrations --------------------- Airflow integration tests cannot be run in the local virtualenv. They can only run in the Breeze -environment with enabled integrations and in the CI. See `CI <../../dev/breeze/doc/ci/README.md>`_ for +environment and in the CI, with their respective integrations enabled. See `CI <../../dev/breeze/doc/ci/README.md>`_ for details about Airflow CI. -When you are in the Breeze environment, by default, all integrations are disabled. This enables only true unit tests -to be executed in Breeze. You can enable the integration by passing the ``--integration `` -switch when starting Breeze. You can specify multiple integrations by repeating the ``--integration`` switch -or using the ``--integration all-testable`` switch that enables all testable integrations and -``--integration all`` switch that enables all integrations. +When you initiate a Breeze environment, by default, all integrations are disabled. This enables only unit tests +to be executed in Breeze. You can enable an integration by passing the ``--integration `` +switch when starting Breeze, either with ``breeze shell`` or with ``breeze start-airflow``. As there's no need to simulate +a full setup of Airflow during integration tests, using ``breeze shell`` (or simply ``breeze``) to run them is +sufficient. You can specify multiple integrations by repeating the ``--integration`` switch, or by using the ``--integration all-testable`` switch +that enables all testable integrations. You may use ``--integration all`` switch to enable all integrations that +includes also non-testable integrations such as openlineage. NOTE: Every integration requires a separate container with the corresponding integration image. These containers take precious resources on your PC, mainly the memory. The started integrations are not stopped -until you stop the Breeze environment with the ``stop`` command and started with the ``start`` command. +until you stop the Breeze environment with the ``breeze down`` command. The following integrations are available: @@ -77,18 +81,20 @@ The following integrations are available: .. END AUTO-GENERATED INTEGRATION LIST' -To start the ``mongo`` integration only, enter: +To start a shell with ``mongo`` integration enabled, enter: .. code-block:: bash breeze --integration mongo -To start ``mongo`` and ``cassandra`` integrations, enter: +You could add multiple ``--integration`` options as the types of the integrations that you want to enable. +For example, to start a shell with both ``mongo`` and ``cassandra`` integrations enabled, enter: .. code-block:: bash breeze --integration mongo --integration cassandra + To start all testable integrations, enter: .. code-block:: bash @@ -99,7 +105,7 @@ To start all integrations, enter: .. code-block:: bash - breeze --integration all-testable + breeze --integration all Note that Kerberos is a special kind of integration. Some tests run differently when Kerberos integration is enabled (they retrieve and use a Kerberos authentication token) and differently when the @@ -109,7 +115,7 @@ for the CI system should run all tests with the Kerberos integration enabled to Running Integration Tests ------------------------- -All tests using an integration are marked with a custom pytest marker ``pytest.mark.integration``. +All integration tests are marked with a custom pytest marker ``pytest.mark.integration``. The marker has a single parameter - the name of integration. Example of the ``celery`` integration test: @@ -172,6 +178,111 @@ Runs all mongo DB tests: breeze testing integration-tests --db-reset --integration mongo +Writing Integration Tests +------------------------- +Before creating the integration tests, you'd like to make the integration itself (i.e., the service) available for use. +For that, you'll first need to create a Docker Compose YAML file under ``scripts/ci/docker-compose``, named +``integration-.yml``. The file should define one service for the integration, and another one +for the Airflow instance that depends on it. It is recommended to stick to the following guidelines: + + +1. Name the ``services::::container_name`` as the service's name and give it an appropriate description under +``services::::labels:breeze.description``, so it would be easier to detect it in Docker for debugging +purposes. + +2. Use an official stable release of the service with a pinned version. When there are number of possibilities for an +image, you should probably pick the latest version that is supported by Airflow. + +3. Set the ``services::::restart`` to "on-failure". + +4. For integrations that require persisting data (for example, databases), define a volume at ``volumes::`` +and mount the volume to the data path on the container by listing it under ``services:::volumes`` +(see example). + +5. Check what ports should be exposed to use the service - carefully validate that these ports are not in use by other +integrations (consult the community what to do if such case happens). To avoid conflicts with host's ports, it is a +good practice to prefix the corresponding host port with a number (usually 2), parametrize it and to list the parameter +under ``# Initialise base variables`` section in ``dev/breeze/src/airflow_breeze/global_constants.py``. + +6. In some cases you might need to change the entrypoint of the service's container, for example, by setting +``stdin_open: true``. + +7. In the Airflow service definition, ensure that it depends on the integration's service (``depands_on``) and set +the env. var. ``INTEGRATION-`` to true. + +8. If you need to mount a file (for example, a configuration file), you could put it at ``scripts/ci/docker-compose`` +(or a subfolder of this path) and list it under ``services::::volumes``. + +For example, ``integration-drill.yml`` looks as follows: + + .. code-block:: yaml + + version: "3.8" + services: + drill: + container_name: drill + image: "apache/drill:1.21.1-openjdk-17" + labels: + breeze.description: "Integration required for drill operator and hook." + volumes: + - drill-db-volume:/data + - ./drill/drill-override.conf:/opt/drill/conf/drill-override.conf + restart: "on-failure" + ports: + - "${DRILL_HOST_PORT}:8047" + stdin_open: true + airflow: + depends_on: + - drill + environment: + - INTEGRATION_DRILL=true + volumes: + drill-db-volume: + + +In the example above, ``DRILL_HOST_PORT = "28047"`` has been added to ``dev/breeze/src/airflow_breeze/global_constants.py``. + +Then, you'll also need to set the host port as an env. var. for Docker commands in ``dev/breeze/src/airflow_breeze/params/shell_params.py`` +under the property ``env_variables_for_docker_commands``. +For the example above, the following statement was added: + +.. code-block:: python + + _set_var(_env, "DRILL_HOST_PORT", None, DRILL_HOST_PORT) + +The final setup for the integration would be adding a netcat to check that upon setting the integration, it is possible +to access the service in the internal port. + +For that, you'll need to add the following in ``scripts/in_container/check_environment.sh`` under "Checking backend and integrations". +The code block for ``drill`` in this file looks as follows: + +.. code-block:: bash + + if [[ ${INTEGRATION_DRILL} == "true" ]]; then + check_service "drill" "run_nc drill 8047" 50 + fi + +Then, create the integration test file under ``tests/integration`` - remember to prefix the file name with ``test_``, +and to use the ``@pytest.mark.integration`` decorator. It is recommended to define setup and teardown methods +(``setup_method`` and ``teardown_method``, respectively) - you could look at existing integration tests to learn more. + +Before pushing to GitHub, make sure to run static checks (``breeze static-checks --only-my-changes``) to apply linters +on the Python logic, as well as to update the commands images under ``dev/breeze/docs/images``. + +When writing integration tests for components that also require Kerberos, you could enforce auto-enabling the latter by +updating ``compose_file()`` method in ``airflow_breeze.params.shell_params.ShellParams``. For example, to ensure that +Kerberos is active for ``trino`` integration tests, the following code has been introduced: + +.. code-block:: python + + if "trino" in integrations and "kerberos" not in integrations: + get_console().print( + "[warning]Adding `kerberos` integration as it is implicitly needed by trino", + ) + compose_file_list.append(DOCKER_COMPOSE_DIR / "integration-kerberos.yml") + + + ----- For other kinds of tests look at `Testing document <../09_testing.rst>`__