ARM64 support in docker images

[andormarkus]

Description

ARM64 support in docker images

Use case / motivation

In the recent years ARM based cpu popularity hugely increased. The arm based cpus are offering better price to performance ratio and laptops outstanding performance. It's getting popular to run ARM nodes in your Kubernetes clusterss.
If you want to run airflow on arm node in kubernetes the docker image wont start it wailt fails with standard_init_linux.go:219: exec user process caused: exec format error.

ARM based cpu in the database world like AWS Graviton or in the consumer world like Apple M1.

Related Issues

In #14796 @potiuk confirmed there is no office support for ARM64 support in docker images

Update: ARM is supported for development now. The support for released imges is coming.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Absolutely

[potiuk]

Been on my radar for quite some time. We were mostly waiting for getting the docker into stable version for MacOS. And yeah I think this is the next 'big' thing for the images once we get full PIP21 compatibility and python 3.9 support

[uranusjr]

The biggest obstacle is likely not Airflow itself (which would likely run on ARM just fine), but the many dependencies it has. Either upstreams need to publish ARM wheels (although my understanding is the most difficult-to-build ones already do), or Airflow needs to develop infrastructures to build the images.

[potiuk]

The biggest obstacle is likely not Airflow itself (which would likely run on ARM just fine), but the many dependencies it has. Either upstreams need to publish ARM wheels (although my understanding is the most difficult-to-build ones already do), or Airflow needs to develop infrastructures to build the images.

Correct. We tried it before and we had limited success previously. The biggest problems were pyarrow/numpy and related especially for the older versions. But with the recent dependency bumps for pyarrow i think this problem might be gone already.

[potiuk]

Airflow needs to develop infrastructures to build the images.

Correct. We have credits from GCP that we will be able to use to run Runners in the ARM infrastructure. We are now running out of S3 credits (trying to get some more) and I am working with Ash, Bin and Claudio on switching to GCP. I think infra won't be a problem :)

[potiuk]

Just checked pyarrow 4.0.0 has aarch wheels. We are using 3.0 now so maybe it's not yet solved.

[andormarkus]

We could install and run numpy on AWS ARM without issue, however installing pyarrow 2.0.0 and 3.0.0 was not possible without serious workarounds. Pyarrow 4.0.0 support aarch64.

@potiuk Can airflow move to pyarrow 4.0.0 ? Should I create a separate ticket issue for it?

[potiuk]

This ain't as easy as that. The pyarrow limitation comes from apache-beam currently. And there will likely be other dependencies with no ARM support - with all the transitive ones we have 500 (!) dependencies for Airflow: https://github.com/apache/airflow/blob/constraints-master/constraints-3.8.txt.

The basic 'no-providers" dependencies are much smaller - just 142 of those :) https://github.com/apache/airflow/blob/constraints-master/constraints-no-providers-3.8.txt. So those might be much easier to get working.

For ARM we might have to add support to exclude some of the providers (like apache-beam) to make it works but that would require implementing more complex CI build/test path, specifically for the ARM image. We will enventually do that, but I think this will come only after we have full GCP support for our CI builds, release all remaining providers for PyPI 21 and add support for Python 3.9. All those are a bit more higher priority.

Also we might want also to exclude some packages for Python 3.9 #15515 - for now we have thrift-sasl which comes from Hive/Kerberos for example (but they are working on support). So likely I am going to develop some exclusion mechanism for our CI build + tests that we might want to use.

However if someone would like to take the current Dockerfile/constraints/extras and try it work for ARM - to see what else we are still missing - happy to hear some experiences from that.

[FyZzyss]

I managed to run airflow 2.0 in docker on the apple m1 with python 3.8.2.
This was helped by upgrading docker to version 3.3.2 with qemu support and adding more resources to docker.

[potiuk]

Ah cool. Any special considerations?

[FyZzyss]

Ah cool. Any special considerations?

There were no more problems.