Minor: Introduce utils::hash for StructArray

[jayzhan211]

Which issue does this PR close?

Closes #.
We will need this for #7893

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[waynexia]

Looks good to me, thanks @jayzhan211

[Weijun-H]

Why do you create a new vec for each iteration?

[jayzhan211]

create each values_hashes for each column. And combine those hashed element in column to one

[Dandandan]

The values_hashes can be created upfront, outside of the loop.

[Dandandan]

This seems not correct, as i here to a column in the struct and hashes_buffer to a hash of one value?

[jayzhan211]

@Dandandan. Not fully got your point. I do consider each column as one value (iterate each row in one column with get one final hash hashes_buffer[i]).

i is the column index in the struct array and is also the hashes_buffer index.
For example, column1, null, column3, column4. We iterate 0, 2, and 3. And hash each of them to hashes_buffer[0], hashes_buffer[2], and hashes_buffer[3] respectively. A column-wise hashes.

[jayzhan211]

I think I mistake how the null buffer works in StructArray. It seems to be row-wise, but I consider it as column-wise...

[jayzhan211]

It looks pretty much better now :)

[Dandandan]

This doesn't yet seem correct, valid_indices contains the rows that are valid, not the columns.

[jayzhan211]

I hash each row now, so we iterate the valid row and hash each column at that row to one hash value.

[Dandandan]

We should be able to use hashes_buffer here directly?

[Dandandan]

If we do that, we should be able to remove the code related to valid_indices, as this already takes care of nulls.

[jayzhan211]

I dont think so.

values_hashes: [9258723240401068087, 9258723240401068087, 8502738074356456021, 8502738074356456021, 4222447303697976283, 9753707356376286577]
hashes_buffer: [9258723240401091360, 9258723240401091360, 0, 8502738074356479294, 0, 0]

We can see that create_hashes does not consider if the row is valid or not. Therefore, we need to iterate valid_indices once .

[Dandandan]

This should work?

Suggested change

	hash_struct_array(array, random_state, hashes_buffer)?;
	create_hashes(array.columns(), random_state, hashes_buffer)?;

[jayzhan211]

If we consider nulls, this doesn't work

[alamb]

@Dandandan do you think this PR is ready to merge?

[jayzhan211]

@alamb @Dandandan Any update on this?

[alamb]

Thank you @jayzhan211 @waynexia and @Dandandan

[jayzhan211]

I found the current implementation failed this

    #[test]
    // Tests actual values of hashes, which are different if forcing collisions
    #[cfg(not(feature = "force_hash_collisions"))]
    fn create_hashes_for_struct_arrays_more_column_than_row() {
        let struct_array = StructArray::from(
            vec![
                (
                    Arc::new(Field::new("bool", DataType::Boolean, false)),
                    Arc::new(BooleanArray::from(vec![
                        false, false
                    ])) as ArrayRef,
                ),
                (
                    Arc::new(Field::new("i32-1", DataType::Int32, false)),
                    Arc::new(Int32Array::from(vec![10, 10])) as ArrayRef,
                ),
                (
                    Arc::new(Field::new("i32-2", DataType::Int32, false)),
                    Arc::new(Int32Array::from(vec![10, 10])) as ArrayRef,
                ),
                (
                    Arc::new(Field::new("i32-3", DataType::Int32, false)),
                    Arc::new(Int32Array::from(vec![10, 10])) as ArrayRef,
                ),
            ],
        );

        assert!(struct_array.is_valid(0));
        assert!(struct_array.is_valid(1));

        let array = Arc::new(struct_array) as ArrayRef;

        let random_state = RandomState::with_seeds(0, 0, 0, 0);
        let mut hashes = vec![0; array.len()];
        create_hashes(&[array], &random_state, &mut hashes).unwrap();
        assert_eq!(hashes[0], hashes[1]);
    }

Let me checkout the reason