Connection password values appearing unmasked in the "Task Instance Details" -> "Environment" field

[chriscmorgan]

Apache Airflow version

Airflow 2.5.1

What happened

Connection password values appearing in the "Task Instance Details" -> "Task Attributes" -> environment field.

We are setting environment variables for the docker_operator with values from the password field in a connection.

The values from the password field are masked in the "Rendered Template" section and in the logs but it's showing the values in the "environment" field under Task Instance Details.

What you think should happen instead

These password values should be masked like they are in the "Rendered Template" and logs.

How to reproduce

Via this DAG, can run off any image.

Create a connection called "DATABASE_CONFIG" with a password in the password field.

Run this DAg and then check its Task Instance Details.

DAG Code:

from airflow import DAG
from docker.types import Mount
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import timedelta
from airflow.models import Variable
from airflow.hooks.base_hook import BaseHook
import pendulum
import json

# Amount of times to retry job on failure
retries = 0

environment_config = {
    "DB_WRITE_PASSWORD": BaseHook.get_connection("DATABASE_CONFIG").password,
}

# Setup default args for the job
default_args = {
    "owner": "airflow",
    "start_date": pendulum.datetime(2023, 1, 1, tz="Australia/Sydney"),
    "retries": retries,
}

# Create the DAG
dag = DAG(
    "test_dag",  # DAG ID
    default_args=default_args,
    schedule_interval="* * * * *",  
    catchup=False,
)

# # Create the DAG object
with dag as dag:
    docker_task = DockerOperator(
        task_id="task",
        image="<image>",
        execution_timeout=timedelta(minutes=2),
        environment=environment_config,
        command="<command>",
        api_version="auto",
        docker_url="tcp://docker.for.mac.localhost:2375",
    )
    

Rendered Template is good:

In "Task Instance Details"

Operating System

centOS Linux and MAC

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Running on a docker via the airflow docker-compose

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[chriscmorgan]

This article seems to have a fix for it:
https://medium.com/@poatek/airflow-might-be-leaking-your-passwords-f4ff94647de7

[potiuk]

Assigned you @chriscmorgan

[chriscmorgan]

Adding this code to line 1593 of views.py does resolve it:

 task = copy.copy(dag.get_task(task_id))
 < after this line >
 ...
 from airflow.utils.log.secrets_masker import get_sensitive_variables_fields

        for sensitive_key in get_sensitive_variables_fields():
            pattern = rf"((\"|\')[a-z0-9_\.]*{sensitive_key}[a-z0-9_\.]*(\"|\'|\=|:|\s)+(\"|\')?.*?(\"|\'))"
            for attr_name in dir(task):
                if not attr_name.startswith("_"):
                    attr = getattr(task, attr_name)
                    regex_matches = re.findall(pattern, str(attr), re.IGNORECASE)
                    if regex_matches:
                        for match in regex_matches:
                            case_1 = re.findall(r":", match[0], re.IGNORECASE)
                            if case_1:
                                attr = re.sub(re.escape(match[0]), f"{match[0].split(':')[0]}:*******", str(attr))
                            else:
                                attr = re.sub(re.escape(match[0]), f"{match[0].split('=')[0]}=*******'", str(attr))
                            try:
                                setattr(task, attr_name, attr)
                            except Exception as e:
                                logging.error(f"Failed to mask password. Error:{e}")

[chriscmorgan]

@potiuk, is there a guide on how I can get a local build environment up and running. I am keen to a MR for this but struggled to get it building locally.

[potiuk]

https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst (look at "New Contributor" section).

[niegowic]

I can see same problem on bash_attribute BashOperator on Task Instance Details - this issue was there in 2.4.0 I was hoping 2.5.2 will fix that but this bug is still there Secrets are properly masked in logs but not in Task Instance Details page: