Enhance pre-commit hook to automate migrations ref doc and revision metadata#21601
Merged
dstandish merged 3 commits intoMar 1, 2022
Merged
Conversation
dstandish
requested review from
ashb,
kaxil,
mik-laj and
potiuk
as code owners
February 15, 2022 22:29
dstandish
commented
Feb 15, 2022
dstandish
force-pushed
the
use-yaml-config-for-migration-documentation
branch
from
February 16, 2022 05:23
c061b7f to
6c3e57a
Compare
dstandish
marked this pull request as draft
February 16, 2022 22:56
Contributor
Author
|
converted to draft for now while i make some updates to make this a more comprehensive and robust solution |
dstandish
force-pushed
the
use-yaml-config-for-migration-documentation
branch
2 times, most recently
from
February 17, 2022 00:04
09431ec to
f2d5d61
Compare
dstandish
marked this pull request as ready for review
February 17, 2022 00:05
jedcunningham
left a comment
Member
There was a problem hiding this comment.
We don't need both a precommit and static check, right, as precommits are also run in static check?
dstandish
force-pushed
the
use-yaml-config-for-migration-documentation
branch
from
February 17, 2022 19:27
d68a17f to
317d11f
Compare
dstandish
force-pushed
the
use-yaml-config-for-migration-documentation
branch
2 times, most recently
from
February 17, 2022 22:07
47fe197 to
8fbe74e
Compare
potiuk
approved these changes
Feb 26, 2022
* document airflow version in each alembic migration module and use this to autogen the doc * update each migration module to have the same description used in migration ref (so it can be used in autogen) See PR apache#21596 for reference.
dstandish
force-pushed
the
use-yaml-config-for-migration-documentation
branch
from
February 26, 2022 19:25
8fbe74e to
3ef995c
Compare
Member
|
Needs rebase. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now the revision table in
migrations-ref.rstis fully automated.When a new migration file is committed, the pre-commit hook will insert the (expected release) airflow version (if it wasn't added manually). Then we'll walk the revisions, update the docs table,
and update the version-revision map constant (for now located in db.py).I had to make it a breeze hook since it depends on airflow itself.
Since we store the revision map in db.py, the hook will have to run every time db.py is modified. If that proves to be annoying, we can move the constant elsewhere.How you can quickly verify this works locally
Test 1 - change version: Open
airflow/migrations/versions/c306b5b5ae4a_switch_xcom_table_to_use_run_id.pyand change the airflow version. Observe thatmap constant andmigrations ref are updated.Test 2 - new migration: Copy
airflow/migrations/versions/c306b5b5ae4a_switch_xcom_table_to_use_run_id.pyand nameairflow/migrations/versions/c306b5b5ae4a_switch_xcom_table_to_use_run_id_2.py. Make it a new revisionc306b5b5ae4a2with down_revision=c306b5b5ae4aand remove line withairflow_versionand try to commit. Observe that airflow version is added, doc updated,constant updated.