ARROW-11299: [Python] Fix invalid-offsetof warnings

[cyb70289]

Use unique_ptr to hold FunctionOptions derived classes instances to fix
invalid-offsetof python build warnings.

~/arrow/python/build/temp.linux-x86_64-3.6/_compute.cpp:26034:146:
warning: offsetof within non-standard-layout type ‘__pyx_obj_7pyarrow_8_compute__CastOptions’ is undefined [-Winvalid-offsetof]
x_type_7pyarrow_8_compute__CastOptions.tp_weaklistoffset = offsetof(struct __pyx_obj_7pyarrow_8_compute__CastOptions, __pyx_base.__pyx_base.weakref);

[github-actions]

https://issues.apache.org/jira/browse/ARROW-11299

[cyb70289]

All debug build failed due to this change. FunctionOptions is supposed to be polymorphic, but looks not necessary.
Would like to hear comments before moving on.
@bkietz @pitrou

[bkietz]

@cyb70289 arrow::dataset::Expression::Call::options requires that virtual destructor since a shared_ptr<FunctionOptions> is used to store subclasses of FunctionOptions. Removing this virtual destructor results in a resource leak since the derived classes' (implicit) destructors are not being called, leaking data members such as arrow::compute::SetLookupOptions::value_set which are not destroyed.

[bkietz]

I think the best way to resolve this issue is to avoid direct instances of derived classes of FunctionOptions in Cython, since Cython inappropriately applies offsetof. For example, note that the warning is emitted for MinMaxOptions but not for SetLookupOptions (where the options are held by unique_ptr rather than being a direct member).

I'd recommend the following approach: rewrite _compute.pyx::FunctionOptions to hold a shared_ptr[CFunctionOptions], and let the cdef classes acquire a non owning pointer to the appropriate FunctionOptions subclass. For example:

_compute.pxd

cdef class FunctionOptions(_Weakrefable):
    cdef:
        shared_ptr[CFunctionOptions] options

    cdef const CFunctionOptions* get_options(self) except NULL

_compute.pyx

cdef class _CastOptions(FunctionOptions):
    cdef:
        const CCastOptions* cast_options

    __slots__ = ()  # avoid mistakingly creating attributes

    cdef const CFunctionOptions* get_options(self) except NULL:
        return self.cast_options

[cyb70289]

@cyb70289 arrow::dataset::Expression::Call::options requires that virtual destructor since a shared_ptr<FunctionOptions> is used to store subclasses of FunctionOptions. Removing this virtual destructor results in a resource leak since the derived classes' (implicit) destructors are not being called, leaking data members such as arrow::compute::SetLookupOptions::value_set which are not destroyed.

Thanks @bkietz , I missed this point.
Will update according to your comment.

[bkietz]

LGTM, thanks!