[SPARK-47736][SQL] Add support for AbstractArrayType#45891
[SPARK-47736][SQL] Add support for AbstractArrayType#45891mihailomilosevic2001 wants to merge 15 commits into
Conversation
|
@cloud-fan Do you think adding AbstractArrayType could be a good idea as a general and not only for StringTypeCollated? Otherwise this PR should be ready for review. |
| * Use ArrayType(StringTypeCollated) for expressions supporting different type of | ||
| * collations in ArrayTypes | ||
| */ | ||
| case class AbstractArrayType(elementType: StringTypeCollated) extends AbstractDataType { |
There was a problem hiding this comment.
can we make it general to take AbstractDataType? I don't think we should limit it to string type.
There was a problem hiding this comment.
Done, I also realised, should I move the StringTypeCollated to the right package? It is now in catalyst.expressions instead of types as all other types.
# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
|
@cloud-fan Did some refactoring with some renaming, so I re-requested approve |
|
thanks, merging to master! |
| import org.apache.spark.sql.types.{AbstractDataType, DataType, StringType} | ||
|
|
||
| /** | ||
| * StringTypeCollated is an abstract class for StringType with collation support. |
There was a problem hiding this comment.
missed a docstring update here
There was a problem hiding this comment.
Noted, will include the change with some of the next PRs.
What changes were proposed in this pull request?
Addition of abstract arraytype which accepts StringTypeCollated as elementType. Changes in this PR #45693 work for ArrayJoin, but will not work in general for other functions. This PR introduces a change to give an interface for all functions. Merge only after #45693.
Why are the changes needed?
This is needed in order to enable functions to use collated arrays.
Does this PR introduce any user-facing change?
Yes, collation functions will work.
How was this patch tested?
Test for array_join added to
CollationSuiteWas this patch authored or co-authored using generative AI tooling?
No.