Skip to content

Validate dbt trigger_reason field to be less than 255 characters#38896

Merged
josh-fell merged 4 commits intoapache:mainfrom
boraberke:fix/dbt-validate-trigger-reason
May 6, 2024
Merged

Validate dbt trigger_reason field to be less than 255 characters#38896
josh-fell merged 4 commits intoapache:mainfrom
boraberke:fix/dbt-validate-trigger-reason

Conversation

@boraberke
Copy link
Copy Markdown
Contributor

Validate and truncate trigger_reason field if it is longer than the limit of 255 characters.

closes: #34676

@boraberke
Copy link
Copy Markdown
Contributor Author

Hey @josh-fell, would greatly appreciate your thoughts and suggestions on this!

@boraberke boraberke force-pushed the fix/dbt-validate-trigger-reason branch from 9885b54 to 284da2a Compare April 13, 2024 20:24
@boraberke boraberke requested a review from dirrao April 13, 2024 20:40
@boraberke
Copy link
Copy Markdown
Contributor Author

Hi @dirrao, did you have a chance to look at this PR again? Would appreciate your comments!

Copy link
Copy Markdown
Contributor

@josh-fell josh-fell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work here!

I do think though that this logic should live in the DbtCloudHook instead. Moving the logic to the hook would cover both scenarios of folks using the operator and the DbtCloudHook.trigger_job_run() method explicitly (as part of a TaskFlow function for example).

@boraberke boraberke force-pushed the fix/dbt-validate-trigger-reason branch from 284da2a to 89943cf Compare May 5, 2024 11:29
@boraberke
Copy link
Copy Markdown
Contributor Author

Nice work here!

I do think though that this logic should live in the DbtCloudHook instead. Moving the logic to the hook would cover both scenarios of folks using the operator and the DbtCloudHook.trigger_job_run() method explicitly (as part of a TaskFlow function for example).

Thanks for your review @josh-fell, I have updated accordingly!

I have a question about testing. I have only implemented a test for the DbtCloudHook, do you think we need to add a test to cover DbtCloudRunJobOperator trigger reason explicitly? I think the test for DbtCloudHook covers all the possible cases including operator and DbtCloudHook.trigger_job_run().

@boraberke boraberke requested a review from josh-fell May 5, 2024 11:37
@josh-fell
Copy link
Copy Markdown
Contributor

Nice work here!
I do think though that this logic should live in the DbtCloudHook instead. Moving the logic to the hook would cover both scenarios of folks using the operator and the DbtCloudHook.trigger_job_run() method explicitly (as part of a TaskFlow function for example).

Thanks for your review @josh-fell, I have updated accordingly!

I have a question about testing. I have only implemented a test for the DbtCloudHook, do you think we need to add a test to cover DbtCloudRunJobOperator trigger reason explicitly? I think the test for DbtCloudHook covers all the possible cases including operator and DbtCloudHook.trigger_job_run().

Agreed. Adding the test to DbtCloudHook is perfectly fine! There is another test which confirms passing the trigger_reason value from the operator to the hook already which is good enough.

@josh-fell josh-fell merged commit c528090 into apache:main May 6, 2024
pateash pushed a commit to pateash/airflow that referenced this pull request May 13, 2024
* Validate dbt `trigger_reason` field to be less than 255 characters

* Rename function to better reflect the behavior

* Move validation of cause parameter to DbtCloudHook

* Add stacklevel to dbt `cause` warning
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

'cause' key need to be limited to 255 characters to avoid failures due to trigger_reason field

3 participants