Python UDF rasters cannot set NODATA, and Python and the JVM disagree about it

[jiayuasu]

Describe the bug

A raster returned from a Python UDF cannot carry a NODATA value that differs from its input, and Python and the JVM report different NODATA for the same band.

GridSampleDimensionSerializer.write() writes a nodata double "for interoperability with Python RasterType", but read() discards it and rebuilds the sample dimension from the Kryo Category[] blob instead:

double offset = input.readDouble();
double scale = input.readDouble();
input.readDouble(); // noDataValue is included in categories, so we just skip it
input.readInt();
Category[] categories = kryo.readObject(input, Category[].class);
return new GridSampleDimension(description, categories, offset, scale);

InDbSedonaRaster.with_bands() replays those blobs from the source raster, so the value Python declares in bands_meta never reaches the JVM. Two consequences:

1. Inherited NODATA silently corrupts derived rasters. Narrowing keeps _category_blobs[:n_bands], so a mask carved out of a scene inherits band 1's NODATA. Given a 4-band raster with NODATA 0 on band 1 and a UDF thresholding it to a 0/1 mask, every 0 in the mask is treated as NODATA:

@udf(returnType=RasterType())
def mask(raster):
    return raster.with_bands((raster.as_numpy()[0] < 6).astype(np.float64))

# on a 12-pixel raster:
#   RS_Count(out, 1, true)  -> 6    (zeros skipped as NODATA)
#   RS_Count(out, 1, false) -> 12
#   RS_BandNoDataValue(out, 1) -> 0.0

There is no way to correct this from inside the UDF; RS_SetBandNoDataValue afterwards is the only workaround.

2. Python and the JVM disagree. Widening replicates the last source band's category blob while bands_meta records NaN. Widening a 4-band raster whose band 4 has NODATA -1 to 8 bands:

python  raster.bands_meta -> [nan, nan, nan, -1.0, nan, nan, nan, nan]
sql     RS_BandNoDataValue(out, 5..8) -> -1.0

Expected behavior

The nodata double on the wire should be authoritative, and with_bands() should accept an explicit NODATA — parity with RS_MapAlgebra's noDataValue argument.

Proposed fix

Have read() reconcile the two via the existing RasterUtils.createSampleDimensionWithNoDataValue(), which early-returns when the values already agree. The JVM writer emits RasterUtils.getNoDataValue(sampleDimension) — derived from the categories — so JVM-to-JVM round trips reconcile to a no-op and are unaffected. This mirrors the reconcileColorModel() that #2956 added for the colour model, for the same class of Python-mutated-raster mismatch.

Then add with_bands(new_data, nodata=...) on the Python side, and make widened bands report the value they will actually receive rather than NaN.

Sedona version

master (1.9.1-SNAPSHOT). Introduced with the Python raster serializer in #2956, so not present in any release yet.

Environment

Reproduced with Spark 3.4 / Scala 2.12, -Dgeotools, pyspark 3.4.4.