ARROW-4223: [Python] Support scipy.sparse integration by rok · Pull Request #4779 · apache/arrow

rok · 2019-07-02T17:53:34Z

This is to resolve ARROW-4223.

jorisvandenbossche

@rok I know the PR is still indicated as draft, so all comments I made you probably were still going to do, but while looking at the diff, thought to at once comment

python/pyarrow/tests/test_serialization.py

python/pyarrow/serialization.py

rok · 2019-07-20T21:53:17Z

I think this is ready for review.
Appveyor seems to be failing on unrelated issue.

rok · 2019-11-11T21:55:39Z

@ursabot build

rok · 2019-11-11T22:15:57Z

@pitrou I think this is now ready for another review round. I've updated #4990 too.

pitrou

LGTM, just one question.

pitrou · 2019-11-12T13:38:49Z

python/pyarrow/tests/test_sparse_tensor.py

What is the purpose of the astype() calls here?

This is a workaround for scipy bug (or lack of feature) see this issue.

But you have if dtype_str != 'f2' above.

Oh, yeah.. this probably should have failed. That's odd. I'll look into it tonight.

jorisvandenbossche

minor nit

jorisvandenbossche · 2019-11-12T16:47:17Z

python/pyarrow/tensor.pxi

Could check here that the passed object is actually a scipy.sparse.coo_matrix ?
(right now the user will probably get an attribute error if you pass a different scipy matrix type)

(and the same for the other from_scipy below)

Thank you :)

…or roundtrip test.

…s serialization.

… test.

…ensorCSR.

pitrou · 2019-11-13T09:43:40Z

The R download failure on AppVeyor is unrelated, merging.

rok · 2019-11-13T10:15:29Z

Thanks @pitrou @mrkn and @jorisvandenbossche! :)

This is to resolve [ARROW-4224](https://issues.apache.org/jira/browse/ARROW-4224). Please note this requires #4779 to be reviewed and merged first. Closes #4990 from rok/ARROW-4224 and squashes the following commits: d99d52d <Rok> Implementing review feedback. 4efdd35 <Rok> Adding type check to from_pydata_sparse. 759e0b3 <Rok> Adding from_pydata_sparse and to_pydata_sparse. Authored-by: Rok <rok@mihevc.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

kou force-pushed the master branch from e3ae1b0 to 03576af Compare July 4, 2019 05:51

kszucs changed the title ~~ARROW-4223 [Python] Support scipy.sparse integration~~ ARROW-4223: [Python] Support scipy.sparse integration Jul 4, 2019

kszucs force-pushed the ARROW-4223 branch from 78c25ae to 159b3f2 Compare July 4, 2019 07:58

jorisvandenbossche reviewed Jul 4, 2019

View reviewed changes

python/pyarrow/tests/test_serialization.py Outdated Show resolved Hide resolved

python/pyarrow/serialization.py Outdated Show resolved Hide resolved

python/pyarrow/serialization.py Outdated Show resolved Hide resolved

emkornfield force-pushed the ARROW-4223 branch from 159b3f2 to f7bb1e5 Compare July 5, 2019 09:03

rok force-pushed the ARROW-4223 branch 2 times, most recently from 5b7472b to eca7a70 Compare July 8, 2019 21:51

rok force-pushed the ARROW-4223 branch 10 times, most recently from ff60f58 to 75097f7 Compare July 20, 2019 16:50

rok marked this pull request as ready for review July 20, 2019 17:02

rok force-pushed the ARROW-4223 branch from ed7a806 to ef8a825 Compare July 20, 2019 18:28

rok force-pushed the ARROW-4223 branch from 16e8517 to 507a5d5 Compare July 21, 2019 23:37

kszucs force-pushed the master branch from 85fe336 to ed180da Compare July 22, 2019 17:04

rok force-pushed the ARROW-4223 branch 2 times, most recently from 081032b to 2ca9ed7 Compare July 22, 2019 18:12

kszucs force-pushed the master branch from ed180da to 85fe336 Compare July 22, 2019 19:29

rok force-pushed the ARROW-4223 branch 5 times, most recently from 15b973e to ebbbdf2 Compare July 28, 2019 23:44

rok force-pushed the ARROW-4223 branch 2 times, most recently from d812904 to 54ad5ad Compare November 11, 2019 19:40

pitrou approved these changes Nov 12, 2019

View reviewed changes

jorisvandenbossche reviewed Nov 12, 2019

View reviewed changes

rok and others added 16 commits November 13, 2019 09:45

Adding scipy.sparse integration.

247cdbd

Changes to GetSparseTensorMessage. Enabling comparison for SparseTens…

c54f005

…or roundtrip test.

Changes to GetSparseTensorMessage to enable SparseTensor to component…

958b354

…s serialization.

Prevent copying buffers on component serialization of a SparseTensor

56df620

Temporarily disabling test_sparse_tensor_csr_components_serialization…

b13604d

… test.

Adding from_scipy and to scipy methods to SparseTensorCOO and SparseT…

11068eb

…ensorCSR.

Re-enabling test_sparse_tensor_coo_components_serialization.

2bc2534

Applying review feedback for python tests.

db1fb5a

Enabling serialization with pydata/sparse.

f4d2e1e

Extract a common part from ReadSparseTensorPayload and ReadSparseTensor

90cbadf

Add SparseCOOIndex::Make

b06429d

Add SparseCSRIndex::Make

927bf5d

Fixing deserialization issue. Rebasing for new tensor type names.

22e864d

Implementing review feedback.

4d7d2b0

Fixing scipy->sparse_tensor tests for dtype=f2.

d5484bf

Adding type check to from_scipy.

eca3885

rok force-pushed the ARROW-4223 branch from 898b581 to eca3885 Compare November 13, 2019 08:45

pitrou closed this in cdaf988 Nov 13, 2019

rok deleted the ARROW-4223 branch November 13, 2019 10:39

evansd mentioned this pull request Feb 10, 2020

Bump pyarrow from 0.15.1 to 0.16.0 bennettoxford/openprescribing#2458

Closed

asfimport mentioned this pull request Nov 13, 2019

[Python] Support scipy.sparse integration #20803

Closed

Conversation

rok commented Jul 2, 2019

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok commented Jul 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rok commented Nov 11, 2019

Uh oh!

rok commented Nov 11, 2019

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Nov 13, 2019

Uh oh!

rok commented Nov 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rok commented Jul 20, 2019 •

edited

Loading