Add instructions on manually fixing MySQL Charset problems#25938
Conversation
5b367c5 to
672df33
Compare
24be4ec to
fbe3d24
Compare
Depending on the way how and when you MySQL DB was created, it might have problems with the right CHARACTER SET and COLLATION used. This PR attempts to describe a process that users can follow to fix such a problem. Also a chapter was added to recommend taking a backup before the migration. Based on discussions and user input from apache#25866, apache#24526 Closes: apache#24526
fbe3d24 to
616bf80
Compare
| Wrong Encoding in MySQL database | ||
| ................................ | ||
|
|
||
| If you are using old Airflow 1.10 as a database created initially either manually or with previous version of MySQL, | ||
| depending on the original character set of your database, you might have problems with migrating to a newer | ||
| version of Airflow and your migration might fail with strange errors ("key size too big", "missing indexes" etc). | ||
| The next chapter describes how to fix the problem manually. |
There was a problem hiding this comment.
I think this one might even be an item in https://airflow.apache.org/docs/apache-airflow/stable/upgrading-from-1-10/index.html and we can link from here to there.
this feels more like a step users were suppose to do when they migrated from 1 to 2?
There was a problem hiding this comment.
I think it is not really something that is 1.10 specific. The way we treated encoding for IDs changed over time - at some point of time it was recommended, and at some point of time it turned into automated/mandatory and the way it works is that I think depends more on when you do certain steps rather than Airflow 1.10 vs Airflow 2.
It's hard to wrap your head around the possible number of scenarios users might have here especialy when you add the fact that people could have (and did) migrates from MySQL 5.7 to MySQL 8 somwhere bettween 1.10 and 2.4.0 which basicallly could have changed the default encoding (from latin1/swedish to utf8) or they could have a completely different encoding set in general.
Then - depending which version you were at when you did the migration, you could end up with pretty much random set of intermixed encodings in your columns. That's why I ended up with "if you have a problem - you need to end up in this state and here are useful hints that can bring you to the right state" rather than trying to fix it automatically. Also see the answer I gave to @itayB in #25866 (reply in thread) why we do not try to automate the fix :)
What we have currently is that encoding is set to uf8mb3 automatically for those ID fields if you do not override it. So once you fix it, it should not happen in the future.
Small rant though... With MySQL 9 or 10 this might return and hit us back again. But then we should likely only enable Mysql 9, 10 (if we decide to) when we test the migration scenarios. The problem with successor of MySQL 8 is that utf8 (which is not default encoding - utf8mb4 is) is an alias to ut8mb3. But documentation of MySQL mentions that in the future versions the utf8 alias will point to utf8mb4 and .... utf8mb3 will be gone. Officially utf8mb3 is deprecated now. I have no idea how we are going to handle this, because this opens up a whole host of various scenarios again.
But I decided to not to worry about this. I sincerely think we should drop MySQL at some point of time, and maybe lack of support for whatever new version of MySQL comes (if it comes) might be a good way to handle dropping MySQL altogether. I even think that instead of supporting MySQL 9 (or whatever) we should develop a tool that will help people to migrate to Postgres from MySQL.
Let's see how much less of MySQL we will have in the next survey.
There was a problem hiding this comment.
Good choice (of Postgres). No I have not used any myself but I saw people reporting successes using simple export/import scripts. Actually migrating out of MySQL to postgres is such a popular thing that there are PLENTY tools out there (and actually I would love to hear more concrete success stories from people like you :).
I personally think that the best approach is to create the database using Airlfow db cli and only migrate the data via export/import through .csv files (after earlier cleanup of historcal data to decrease the size of the data to migrate and speed it up). Migrating structure is prone to errors but the data in both databases should be exactly the same.
Of course assuming the export from MySQL will use the right encoding - which is not at all given. Unfortunately MySQL complicates encodings in so many strange ways that it's very likely this is not gonna work first time. For example when you use mysql client to connect to mysql server, the default encoding used by the client is NOT the server encoding but your machine encoding - so you can get weird encoding issues if encoding on your "terminal' machine is different than the one in the server (now you might slowly understand my hatred for encoding approach mess in MySQL).
Airflow has suprisingly little number of tables, so this can be done entirely by writing hand-written scripts, there is no need to use specialized tools for that. The only thing that you have to remember is to do the import in the right sequence due to foreign keys and likely speed up the import by temporarily disabling the foreign keys while importing and re-enabling them after.
There was a problem hiding this comment.
thanks @potiuk ! I will try to migrate as you suggested and I will share the results ;)
There was a problem hiding this comment.
I'd love to approve blog post "How I migrated our Airflow from MySQL to PostgresSQL and you can too" in our Medium publication :D
There was a problem hiding this comment.
In progress (it will take me a few days) - stay tuned ;)
There was a problem hiding this comment.
took more than a few days, but for the closure: https://medium.com/apache-airflow/migrating-airflow-backend-database-from-mysql-to-postgres-25eebed59f8d
Depending on the way how and when you MySQL DB was created, it might
have problems with the right CHARACTER SET and COLLATION used.
This PR attempts to describe a process that users can follow to
fix such a problem.
Also a chapter was added to recommend taking a backup before
the migration.
Based on discussions and user input from #25866, #24526
Closes: #24526
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.