update watchtower version in amazon provider

[alexInhert]

Description

there is limitation to version 2

airflow/airflow/providers/amazon/provider.yaml

Line 48 in 809d95e

# watchtower 3 has been released end Jan and introduced breaking change across the board that might

Use case/motivation

using up to date version of the library

Related issues

didnt find

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Would you like to attempt it @alexInhert ? I marked it as good first issue so maybe someone will pick it up, but if you need it badly then the most certain way is to implement it yourself,

[potiuk]

Or maybe @ferruzzi or @o-nikolas - this is already somehow planned by the MWAA team :) ?

[o-nikolas]

@potiuk, We have no immediate plans, our needs are covered by at least 2.0.1, but I think upgrading is still a good idea! I can assist @alexInhert if they get stuck, since I've done version bumps for Watchtower in the past.

But note: This change will need a careful check of Airflow logs, since v3 of Watcher can now include much more information in the logs than previous (due to lacking json deserialization support many things were previously just logged as null but which now are logged in full)

[cBiscuitSurprise]

I would like to give this issue a try.

We have the option to override the default-json serializer to whatever we want. I would propose that we override this to preserve the 2.x behavior with a config option to allow opt-in to the new repr format using logging.REMOTE_LOG_KWARGS (a new config). This way we can upgrade to 3.x, but keep the old serializer.

def json_serialize_legacy(o):
    """
    The legacy `watchtower@2.0.1` json serializer function that serialized
    datetime objects as ISO format andall other non-JSON-serializable to `null`
    """
    if isinstance(o, (date, datetime)):
        return o.isoformat()

...

class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
...
    def __init__(self, base_log_folder: str, log_group_arn: str, filename_template: str | None = None):
        ...

        remote_log_options: Dict[str, Any] = conf.get("logging", "REMOTE_LOG_KWARGS", {})
        self.opt_in_new_json_serialize = remote_log_options.get("opt_in_new_json_serialize", False)

...

    def set_context(self, ti):
        super().set_context(ti)
        self.handler = watchtower.CloudWatchLogHandler(
            ...
            json_serialize_default=None if self.opt_in_new_json_serialize else json_serialize_legacy,
        )

This is my first interaction with this project, so I understand if it's not desirable to add config like this, just brainstorming. Let me know what you guys think.

[cBiscuitSurprise]

Is this class deprecated? Would that make this issue a non-issue (and I should find another issue)?

airflow/airflow/utils/log/__init__.py

Lines 23 to 28 in 6618c5f

	__deprecated_classes = {
	"cloudwatch_task_handler": {
	"CloudwatchTaskHandler": (
	"airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
	),
	},