Bump Parquet.Net from 4.23.5 to 5.5.0

[dependabot]

Updated Parquet.Net from 4.23.5 to 5.5.0.

Release notes

Sourced from Parquet.Net's releases.

5.5.0

Improvements

• BREAKING: ParquetSerializer deserialization generic methods now constrain the type parameter to class, new() (previously new() only). This explicitly prevents using value types as deserialization targets (#​698).
• TimeSpanDataField constructor has an option to set IsAdjustedToUTC (#​650).
• IAsyncEnumerable<T> is limited to .NET 10 and above now.
• Allow reading and writing really large decimal values from Parquet files (larger than 29 significant digits) (#​689, #​697).
• Column Chunk encodings are populated according to their use rather than being hardcoded (#​628).
• Internally, compression/decompression logic has been changed to use managed external packages for Snappy and Zstandard algorithms. IronCompress dependency is now completely removed, because there are decent managed implementations available nowadays. This will also allow to upgrade compression libraries more easily in the future and optimise memory usage. There is a slight memory usage improvement (~5%).

5.4.0

Improvements

• It's now possible to serialize nested lists (list of lists, List<List<T>>) in class serializer, by @​aloneguid in #​612. Thanks @​Vannevelj.
• Support for implicit parquet lists in schema parser and serializer, by @​aloneguid and @​mukunku in #​681.
• Added IAsyncEnumerable serializer with System.Linq.AsyncEnumerable, by @​Arithmomaniac in #​674.
• All the documentation has been condensed and moved back to README file, making it easier to find and read, by @​aloneguid.

Bugs fixed

• When encoding bool values, the results are now consistent, by @​Kevin-Ross-ECC in #​643.
• ParquetRowGroupWriter.Dispose() will not throw exceptions as it conflicts with try/catch/finally ideology. You should call CompleteValidate after writing all columns instead. Thanks to @​rkarim-nnk in #​666.
• Required struct members were always annotated as optional, by @​aloneguid in #​582.

5.3.0

• new: added support for BYTE_STREAM_SPLIT encoding, by @​alex-harper in #​539.
• new: DELTA_BINARY_PACKED is supported on more data types (short,ushort,ulong,uint), by @​ee-naveen and @​wickedmachinator in #​599.
• imp: added cancellation token to ReadEntireRowGroupAsync for consistency with the rest of the API, by @​kiloOhm in #​647.
• imp: added validation to prevent writing malformed Parquet files, by @​Kuinox in #​656.
• imp: necessary fixes to get .NET 10 ready.
• fix: DataPageHeaderV2.IsCompressed is defaulting to false on read even though the spec seems to say it should be true, by @​ngbrown in #​625.
• fix: class deserialiser can handle pre-initialised dictionaries, by @​kiloOhm in #​651.
• fix: add bounds check for pageSize in RleBitpackedHybridEncoder.Decode, by @​mlptownsend in #​662.
• test: added DuckDB for integration tests (#​661 by @​aloneguid).

5.2.0

• Fixed parsing empty byte arrays in #​616 by @​Kuinox.
• Fixed ArgumentException "destination is too short" when reading a parquet file generated by DuckDB, by @​BMehlkop
  in #​636.
• Exposed FieldId in DataField by @​Hrachkata in #​632.
• Fixed class serialisation bug that didn't handle empty/null list of nullable type. By @​mrinal-thomas in #​630.
• Handle decimals with no scale defined: DECIMAL(X, 0) by @​mukunku in #​602.
• Fix dictionary serialisation issue when key is non-nullable in #​606 by @​AshishGupta1995.
• Documentation fixes.
• floor (experimental UI) is discontinued due to having massive performance issues with Avalonia. I don't think Avalonia is a suitable UI framework for low latency desktop applications.

5.1.1

Improvements

• RecyclableMemoryStreamManager exposes memory settings in ParquetOptions. by @​DrewMcArthur in #​597.

Bug fixes

• Fixed ParquetSchema equality collisions in #​596 by @​goeffthomas.
• Fixed failures when decoding LZ4Raw encoding. Big shoutout to @​mukunku for saving us from the data gremlins! See mukunku/ParquetViewer#​126
• Make isAdjustedToUTC defualt to true and allow it to be set. By @​cliedeman in #​547.

Floor

• Turns out, MacOSX binaries were having an identity crisis because they were built by a Linux build server. They just couldn't get along! But fear not, this has been fixed. Big thanks to @​daningalla and @​dmunch in #​587. It's like a tech peace treaty!

5.1.0

Improvements

• Class-reflected schemas for map complex types will generate key/value properties with "key" and "value" names, respectively. This is required in order to deserialise externally generated parquet files with dictionaries.
• Updated dependent packages.
• Class deserializer will check for type compatibility when deserialising incompatible types, which will prevent accidental data loss or narrowing. Thanks to @​dkotov in #​573.
• Write RowGroup total size and compressed size by @​justinas-marozas in #​580.
• DELTA_BYTE_ARRAY aka "Delta Strings" encoding can read byte array types in addition to string types.

Bug fixes

• Adopted to breaking change in Parquet specification (technically it's a fault of upstream, not this library's) resulting in bugs like #​576 and #​579.
• Boolean fields encoded in RLE failed to read at all. Technically RLE can be used for integer and booleans, but we only ever supported integers.

Changes

• Removed .NET 6 build target as it is out of support.

Floor

• Compiled using .NET 9 (upgrade from .NET 8).

5.0.2

New features

• Untyped serialisation supports async enumerable, thanks to @​flambert860 in #​566.

Improvements

• Serialisation of CLR (not Parquet) structs and nullable structs is now properly handled and supported, thanks to @​paulengineer.
• Microsoft.Data.Analysis related functionality was moved out to a separate nuget package - Parquet.Net.Data.Analysis. This is because it introduces quite a few dependencies which are not always needed with the slim main package.
• For Windows, run unit tests on x86 and x32 explicitly.
• Improved GHA build/release process, combining all workflows into one and simplifying it, most importantly release management.

Floor

• application slimmed down a bit, removing "File Explorer". Sticking to doing one thing and do it well - view Parquet files.
• Due to Avalonia startup times being not satisfactory when you are in the "mode" (1-2 seconds) Floor will reuse existing instance to open a file rather than starting the app again.

Announcements 🎉

There is a new, very young project I've been thinking a lot for a long time and finally started - DeltaIO. It's attempting to do what Parquet.Net did for Apache Parquet but for Delta tables. It heavily relies on this library to read delta logs and data from it. It's still very young, but if you are interested in Delta with .NET, please check it out, bookmark, start and leave feedbacks/suggestions.

5.0.1

New feature

You can deserialise "required" lists and "required" list elements, as raised by @​akaloshych84 in #​502. See nullability and lists.

Improvements

• Better error reporting in case class serializer has mismatched definition and repetition levels (as per #​502).
• Pass property attributes down to list data field, by @​agaskill in #​559.

Bug fixed

• Compression/decompression would fail on some platforms like x86 or Linux x86 with musl runtime.

Floor

• boolean columns display as checks.
• Structs display as expandable objects, with properly aligned keys.

5.0.0

Support Parquet.Net

If you find the project helpful, you can support Parquet.Net by starring it.

Breaking changes

• This is the first version without old Table/Row API, which is now completely removed. This API was one of the major headaches and source of bugs since being introduced in the very first version of this library. If you need a similar functionality, consider untyped serializer which should be stable enough (Floor utility relies on this exclusively for quite some time).
• ParquetSerializer's SerializeAsync was accepting ParquetSerializerOptions but DeserializeAsync was accepting ParquetOptions. This is now aligned for consistency so they both use ParquetSerializerOptions.

New features

• Class deserializer can optionally ignore property name casing (#​536).

Improvements

• ParquetWriter supports asynchronous dispose pattern (IAsyncDisposable), thanks to @​andagr in #​479.
• IronCompress upstream dependency updated to 1.6.0.

Bugs fixed

• Nullable Enums were not correctly unwrapped to primitive types, by @​cliedeman in #​551.
• Reverting #​537 due to it breaking binary compatibility in 4.25.0. Thanks to @​NeilMacMullen for reporting this.

4.25.0

Improvements

• File merger utility has Stream overload for non file-based operations.
• File merger utility has extra overload to choose compression codec and specify custom metadata, by @​dxdjgl in #​519.
• Timestamp logical type is supported, by @​cliedeman in #​521.
• More data types support encoding using Dictionary encoding, by @​EamonHetherton in #​531.
• Support for Roslyn nullable types, by @​ErikApption in #​537.
• internal: fix return of Decode methods to returning the actual destination length, by @​artnim in #​543.

4.24.0

New features

• Enum serialization is supported, using Enum's underlying type as a storage type.
• [ParquetIgnore] is supported in addition to [JsonIgnore] for class properties. This is useful when you want to ignore a property in Parquet serialization but not in JSON serialization. Thanks to @​rhvieira1980 in #​411.
• By popular demand, there is now a FileMerger utility which can merge multiple parquet files into a single file by either merging files or actual data together.

Improvements

• Nullable TimeSpan support in ParquetSerializer by @​cliedeman in #​409.
• DataFrame support for int16/uint16 types by @​asmirnov82 in #​469.
• Dropping build targets for .NET Core 3.1 and .NET 7.0 (STS). This should not affect anyone as .NET 6 and 8 are the LTS versions now.
• Added convenience methods to serialize/deserialize collections into a single row group in #​506 by @​piiertho.
• Serialization of interfaces and interface member properties is now supported, see #​513 thanks to @​Pragmateek.
• ParquetReader is now easier to use in LINQ expressions thanks to @​danielearwicker in #​509.
• Upgraded to latest IronCompress dependency.

Bug fixes

• Loop will read past the end of a block #​487 by @​alex-harper.
• Decimal scale condition check fixed in #​504 by @​sierzput.
• Class schema reflector was using single cache for reading and writing, which resulted in incorrect schema for writing. Thanks to @​Pragmateek in #​514.
• Incorrect definition level for null values in #​516 by @​greg0rym.

Parquet Floor

• New feature "File explorer" lists filesystem using a panel on the left, allowing you to quickly load different files in the same directory and navigate to other directories.
• Hovering over title will show full file path and load time in milliseconds.
• Right-click on a row shows context menu allowing to copy the row to clipboard in text format.
• Icon updated to use the official Parquet logo.
• You will get a notification popup if a new version of Parquet Floor is available.
• Telemetry agreement changed and made clearer to understand.

Commits viewable in compare view.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)