Skip to content

Commit e1d9c7f

Browse files
committed
ARROW-1550: [Python] Explicitly close owned file handles in ParquetWriter.close to avoid Windows flakiness
I can reproduce this failure locally, but I'm unsure why this just now started happening. The 0.7.0 release build passed (https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/build/1.0.3357/job/477b1iicmwuy51l8) and there haven't been related code changes since then. Either way it's better to close the sink explicitly Author: Wes McKinney <wes.mckinney@twosigma.com> Closes #1114 from wesm/ARROW-1550 and squashes the following commits: 863827c [Wes McKinney] Check status 7248c79 [Wes McKinney] Explicitly close owned file handles in ParquetWriter.close to avoid flakiness on Windows
1 parent 4a65fea commit e1d9c7f

2 files changed

Lines changed: 5 additions & 3 deletions

File tree

python/pyarrow/_parquet.pyx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,7 @@ cdef class ParquetWriter:
562562
cdef:
563563
unique_ptr[FileWriter] writer
564564
shared_ptr[OutputStream] sink
565+
bint own_sink
565566

566567
cdef readonly:
567568
object use_dictionary
@@ -588,8 +589,10 @@ cdef class ParquetWriter:
588589
check_status(FileOutputStream.Open(c_where,
589590
&filestream))
590591
self.sink = <shared_ptr[OutputStream]> filestream
592+
self.own_sink = True
591593
else:
592594
get_writer(where, &self.sink)
595+
self.own_sink = False
593596

594597
self.use_dictionary = use_dictionary
595598
self.compression = compression
@@ -664,6 +667,8 @@ cdef class ParquetWriter:
664667
def close(self):
665668
with nogil:
666669
check_status(self.writer.get().Close())
670+
if self.own_sink:
671+
check_status(self.sink.get().Close())
667672

668673
def write_table(self, Table table, row_group_size=None):
669674
cdef CTable* ctable = table.table

python/pyarrow/tests/test_parquet.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1202,9 +1202,6 @@ def test_write_error_deletes_incomplete_file(tmpdir):
12021202
except pa.ArrowException:
12031203
pass
12041204

1205-
# Ensure that object has been destructed; this causes test failures on
1206-
# Windows
1207-
gc.collect()
12081205
assert not os.path.exists(filename)
12091206

12101207

0 commit comments

Comments
 (0)