Add Union type casting support to `arrow-cast`

[friendlymatthew]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Arrow's cast kernel currently doesn't support casting Union types to other types. This blocks useful operations like comparing Union columns with scalar values in query engines like Datafusion (see apache/datafusion#18825)

I propose the following behavior:

When casting Union(Int32, Utf8) to Int32:

• extract values from the int32 variant
• non-matching variants (Utf8) become Null

There's also an intermediary cast that could occur, for instance when casting Union(Int32, Utf8) to Int64:

• extract values from the int32 variant
• cast them if needed (e.g., Int32 -> int64)
• non-matching variants (Utf8) become null

This implementation is quite greedy: it finds the matching variant by exact type match first, then by cast-compatibility. (Users can determine precedence by a Union variant's type_id).

Motivation:

• Related Coerce union pydantic/datafusion#49
• Related Extend comparison support for Union arrays against an opaque array #8881

[friendlymatthew]

Curious what you think @alamb, this might be too crazy but this seems to work really well in our pydantic fork

[alamb]

I think this sounds like a specialized cast kernel -- the arrow-rs cast kernel is pretty dumb and just does what it is told. I think the logic you describe seems reasonable but I can imagine other semantics as well

Maybe a new kernel like union_cast 🤔 (that DataFusion could potentially call) ?