Please allow to use external packages for all dependencies (ex. zstd, flatbuffers, etc) to facilitate packaging

[yurivict]

Here: https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L218

Based on Repology, zstd packages are available for about every OS: https://repology.org/metapackage/zstd/versions

[wesm]

hi @yurivict what are you asking for exactly? If the user does not have zstd installed, our build system builds the package from source. We should ensure that different kinds of system-level installed packages are properly detected in https://github.com/apache/arrow/blob/master/cpp/cmake_modules/FindZSTD.cmake, though

[yurivict]

There should be a switch to use only external packages, ex. USE_EXTERNAL_PACKAGES cmake variable.

My use case: I would like to create a FreeBSD port for arrow. On FreeBSD (as in most systems) package builder doesn't allow ad-hoc file downloads started from the project build process. This is for security reasons. So there should a central switch that uses only external packages.

If the user does not have zstd installed, our build system builds the package from source.

Users work on some OSes, which nowadays all have packaging systems (maybe except Windows). Packages should be used. And users shouldn't be building individual softwares themselves, they should use pre-built packages provided by their systems. You are trying to cover up the lack of dependency packages when the right way is to have them created.

[wesm]

Can you please open a JIRA issue? There is also a mailing list thread about improvements to the build system to make packaging for Linux distributions easier (the same concerns apply to FreeBSD)

https://lists.apache.org/thread.html/56e3e5e44901a4ff4f62adc93bb0d34d4c4eac48c993146172e7684e@%3Cdev.arrow.apache.org%3E

You are trying to cover up the lack of dependency packages when the right way is to have them created.

I think this statement is a little bit judgmental -- we are absolutely not trying to "cover up" anything. Since Arrow is still a relatively new project, we have been concerned with providing a uniform development environment to all developers of the C++ project to help promote forward library development progress. In that we have been objectively successful

We don't have enough maintainers to handle support for all of the different third party versions of system libraries on different OS package managers. We have the ExternalProjects set up so that developers are not held back by old packages in their OS PM. This is where we need your help

[yurivict]

JIRA issue: https://issues.apache.org/jira/browse/ARROW-4607

[yurivict]

This is where we need your help

I am only a volunteer FreeBSD port maintainer. I identify software that needs to be ported, create ports, and later maintain them. If a particular package is not ready to be ported and requires significant efforts, I can't invest too much time into it, because then I'll become a developer on a large number of software packages. I simply don't have enough resources for this, sorry!

Basically, cmake provides all the tools to depend on the external packages. These are cmake scripts and pkg-config instructions. You should just use these tools, as documented in the cmake documentation.

[wesm]

You should just use these tools, as documented in the cmake documentation.

I agree, though I take issue with the "should" and "just" words here (see http://www.humansystemsinaction.com/lullaby-language/). There is development to do here, and testing. We will also probably need to document which distros have adequate system-level dependencies to build the project, since not all of them do.

[kou]

Thanks for opening the JIRA issue.
I'm working of this issue to improve MinGW package (and .deb packages) such as #3675. 0.13.0 or 0.14.0 will support more system libraries. You may be able to work on FreeBSD port with 0.13.0 or 0.14.0.