Skip to content

feat: Add new documentation facet to all OL events#52343

Merged
mobuchowski merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-doc-facet
Jul 3, 2025
Merged

feat: Add new documentation facet to all OL events#52343
mobuchowski merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-doc-facet

Conversation

@kacpermuda
Copy link
Copy Markdown
Collaborator

@kacpermuda kacpermuda commented Jun 27, 2025

ContentType was added to OL documentation job facet in #3748. OL client requirement was bumped in #52075 to include that change.
Now I'm adding a documentation facet to all OL events, with this new content type field.

We're checking all the fields mentioned in this doc: doc, doc_md etc.

This also introduce hierarchy, as we can only include one doc in the facet.
The order for task is the following:

  1. doc
  2. doc_md
  3. doc_json
  4. doc_yaml
  5. doc_rst

For DAG it's:

  1. doc_md
  2. description

Also for tasks if task doc is not present, we're including DAG docs (if present).

Most of the lines is from testing it properly in system tests.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kacpermuda kacpermuda requested a review from mobuchowski as a code owner June 27, 2025 14:41
@kacpermuda kacpermuda force-pushed the feat-ol-doc-facet branch 2 times, most recently from ff0b151 to ab813e2 Compare June 27, 2025 15:45
@potiuk potiuk force-pushed the feat-ol-doc-facet branch from ab813e2 to 4a05014 Compare June 28, 2025 18:53
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Jun 28, 2025

Rebasing to check compatibility after fixing it in main.

@kacpermuda kacpermuda force-pushed the feat-ol-doc-facet branch 4 times, most recently from 5d06a9b to 68a00e2 Compare July 1, 2025 09:22
@mobuchowski
Copy link
Copy Markdown
Contributor

@kacpermuda just one question - is there any practical limit on the size of the documentation? Only thing I'm worried about here is the potential impact on event size.

@kacpermuda
Copy link
Copy Markdown
Collaborator Author

Yeah, good point, I don't think there is one. We can set some limit in the provider, I'll add some variable for that, wdyt should be the max size of such docs?

I'm not sure if we should allow users to configure this limit, maybe instead we'll introduce some event size limits that can be configured (globally or per transport), if such need occurs.

@mobuchowski
Copy link
Copy Markdown
Contributor

For now, maybe hardcode the length to something like 64kb and truncate the field if it gets larger?

@kacpermuda kacpermuda force-pushed the feat-ol-doc-facet branch from 68a00e2 to 48a28a7 Compare July 3, 2025 12:08
@kacpermuda
Copy link
Copy Markdown
Collaborator Author

Added _truncate_string_to_byte_size function and applied it to doc included in OL event. Thanks for the input

@kacpermuda kacpermuda force-pushed the feat-ol-doc-facet branch from 48a28a7 to 6a9f527 Compare July 3, 2025 12:33
@mobuchowski
Copy link
Copy Markdown
Contributor

Thanks!

@mobuchowski mobuchowski merged commit edf86df into apache:main Jul 3, 2025
75 checks passed
@kacpermuda kacpermuda deleted the feat-ol-doc-facet branch July 3, 2025 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants